TunerHyperband
class that implements hyperband tuning (HB). HB
repeatedly calls SHA (TunerSuccessiveHalving) with different numbers of
starting configurations. A larger number of starting configurations
corresponds to a smaller budget allocated in the base stage. Each run of SHA
within HB is called a bracket. HB considers s_max + 1
brackets with
s_max = floor(log(r_max / r_min, eta)
. The most explorative bracket
s = s_max
constructs s_max + 1
stages and allocates the minimum budget
(r_min
) in the base stage. The minimum budget is increased in each bracket
by a factor of eta
and the number of starting configurations is computed so
that each bracket approximately spends the same budget. Use
hyperband_schedule()
to get a preview of the bracket layout.
s | 3 | 2 | 1 | 0 | ||||||||
i | n_i | r_i | n_i | r_i | n_i | r_i | n_i | r_i | ||||
0 | 8 | 1 | 6 | 2 | 4 | 4 | 8 | 4 | ||||
1 | 4 | 2 | 3 | 4 | 2 | 8 | ||||||
2 | 2 | 4 | 1 | 8 | ||||||||
3 | 1 | 8 |
s
is the bracket number, i
is stage number, n_i
is the number of
configurations and r_i
is the budget allocated to a single configuration.
The budget hyperparameter must be tagged with "budget"
in the search space.
The minimum budget (r_min
) which is allocated in the base stage of the most
explorative bracket, is set by the lower bound of the budget parameter. The
upper bound defines the maximum budget (r_max
) which which is allocated to
the candidates in the last stages.
Source
Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A (2018). “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.” Journal of Machine Learning Research, 18(185), 1-52. https://jmlr.org/papers/v18/16-558.html.
Subsample Budget
If the learner lacks a natural budget parameter, mlr3pipelines::PipeOpSubsample can be applied to use the subsampling rate as budget parameter. The resulting mlr3pipelines::GraphLearner is fitted on small proportions of the mlr3::Task in the first stage, and on the complete task in last stage.
Dictionary
This Optimizer can be instantiated via the dictionary
mlr_optimizers or with the associated sugar function opt()
:
mlr_optimizers$get("hyperband")
opt("hyperband")
Parameters
eta
numeric(1)
With every stage, the budget is increased by a factor ofeta
and only the best1 / eta
configurations are promoted to the next stage. Non-integer values are supported, buteta
is not allowed to be less or equal 1.sampler
paradox::Sampler
Object defining how the samples of the parameter space should be drawn in the base stage of each bracket. The default is uniform sampling.repetitions
integer(1)
If1
(default), optimization is stopped once all brackets are evaluated. Otherwise, optimization is stopped afterrepetitions
runs of hyperband. The bbotk::Terminator might stop the optimization before all repetitions are executed.
Archive
The mlr3tuning::ArchiveTuning holds the following additional columns that are specific to the hyperband algorithm:
bracket
(integer(1)
)
The bracket index. Counts down to 0.stage
(integer(1))
The stages of each bracket. Starts counting at 0.repetition
(integer(1))
Repetition index. Start counting at 1.
Custom Sampler
Hyperband supports custom paradox::Sampler object for initial configurations in each bracket. A custom sampler may look like this (the full example is given in the examples section):
# - beta distribution with alpha = 2 and beta = 5
# - categorical distribution with custom probabilities
sampler = SamplerJointIndep$new(list(
Sampler1DRfun$new(params[[2]], function(n) rbeta(n, 2, 5)),
Sampler1DCateg$new(params[[3]], prob = c(0.2, 0.3, 0.5))
))
Progress Bars
$optimize()
supports progress bars via the package progressr
combined with a Terminator. Simply wrap the function in
progressr::with_progress()
to enable them. We recommend to use package
progress as backend; enable with progressr::handlers("progress")
.
Parallelization
This hyperband implementation evaluates hyperparameter configurations of
equal budget across brackets in one batch. For example, all configurations
in stage 1 of bracket 3 and stage 0 of bracket 2 in one batch. To select a
parallel backend, use future::plan()
.
Logging
Hyperband uses a logger (as implemented in lgr) from package
bbotk.
Use lgr::get_logger("bbotk")
to access and control the logger.
Super classes
mlr3tuning::Tuner
-> mlr3tuning::TunerFromOptimizer
-> TunerHyperband
Examples
if(requireNamespace("xgboost")) {
library(mlr3learners)
# define hyperparameter and budget parameter
search_space = ps(
nrounds = p_int(lower = 1, upper = 16, tags = "budget"),
eta = p_dbl(lower = 0, upper = 1),
booster = p_fct(levels = c("gbtree", "gblinear", "dart"))
)
# \donttest{
# hyperparameter tuning on the pima indians diabetes data set
instance = tune(
method = "hyperband",
task = tsk("pima"),
learner = lrn("classif.xgboost", eval_metric = "logloss"),
resampling = rsmp("cv", folds = 3),
measures = msr("classif.ce"),
search_space = search_space,
term_evals = 100
)
# best performing hyperparameter configuration
instance$result
# }
}
#> Loading required namespace: xgboost
#> nrounds eta booster learner_param_vals x_domain classif.ce
#> 1: 8 0.5512812 gbtree <list[6]> <list[3]> 0.2395833