Tuner Using the Hyperband Algorithm

TunerHyperband class that implements hyperband tuning (HB). HB repeatedly calls SHA (TunerSuccessiveHalving) with different numbers of starting configurations. A larger number of starting configurations corresponds to a smaller budget allocated in the base stage. Each run of SHA within HB is called a bracket. HB considers s_max + 1 brackets with s_max = floor(log(r_max / r_min, eta). The most explorative bracket s = s_max constructs s_max + 1 stages and allocates the minimum budget (r_min) in the base stage. The minimum budget is increased in each bracket by a factor of eta and the number of starting configurations is computed so that each bracket approximately spends the same budget. Use hyperband_schedule() to get a preview of the bracket layout.

s			3			2			1			0
i		n_i	r_i		n_i	r_i		n_i	r_i		n_i	r_i
0		8	1		6	2		4	4		8	4
1		4	2		3	4		2	8
2		2	4		1	8
3		1	8

s is the bracket number, i is stage number, n_i is the number of configurations and r_i is the budget allocated to a single configuration.

The budget hyperparameter must be tagged with "budget" in the search space. The minimum budget (r_min) which is allocated in the base stage of the most explorative bracket, is set by the lower bound of the budget parameter. The upper bound defines the maximum budget (r_max) which which is allocated to the candidates in the last stages.

Source

Li L, Jamieson K, DeSalvo G, Rostamizadeh A, Talwalkar A (2018). “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization.” Journal of Machine Learning Research, 18(185), 1-52. https://jmlr.org/papers/v18/16-558.html.

Subsample Budget

If the learner lacks a natural budget parameter, mlr3pipelines::PipeOpSubsample can be applied to use the subsampling rate as budget parameter. The resulting mlr3pipelines::GraphLearner is fitted on small proportions of the mlr3::Task in the first stage, and on the complete task in last stage.

Dictionary

This Optimizer can be instantiated via the dictionary mlr_optimizers or with the associated sugar function opt():

mlr_optimizers$get("hyperband")
opt("hyperband")

Parameters

eta

numeric(1)
With every stage, the budget is increased by a factor of eta and only the best 1 / eta configurations are promoted to the next stage. Non-integer values are supported, but eta is not allowed to be less or equal 1.

sampler

paradox::Sampler
Object defining how the samples of the parameter space should be drawn in the base stage of each bracket. The default is uniform sampling.

repetitions

integer(1)
If 1 (default), optimization is stopped once all brackets are evaluated. Otherwise, optimization is stopped after repetitions runs of hyperband. The bbotk::Terminator might stop the optimization before all repetitions are executed.

Archive

The mlr3tuning::ArchiveTuning holds the following additional columns that are specific to the hyperband algorithm:

• bracket (integer(1))
  The bracket index. Counts down to 0.

• stage (integer(1))
  The stages of each bracket. Starts counting at 0.

• repetition (integer(1))
  Repetition index. Start counting at 1.

Custom Sampler

Hyperband supports custom paradox::Sampler object for initial configurations in each bracket. A custom sampler may look like this (the full example is given in the examples section):

# - beta distribution with alpha = 2 and beta = 5
# - categorical distribution with custom probabilities
sampler = SamplerJointIndep$new(list(
  Sampler1DRfun$new(params[[2]], function(n) rbeta(n, 2, 5)),
  Sampler1DCateg$new(params[[3]], prob = c(0.2, 0.3, 0.5))
))

Progress Bars

$optimize() supports progress bars via the package progressr combined with a Terminator. Simply wrap the function in progressr::with_progress() to enable them. We recommend to use package progress as backend; enable with progressr::handlers("progress").

Parallelization

This hyperband implementation evaluates hyperparameter configurations of equal budget across brackets in one batch. For example, all configurations in stage 1 of bracket 3 and stage 0 of bracket 2 in one batch. To select a parallel backend, use future::plan().

Logging

Hyperband uses a logger (as implemented in lgr) from package bbotk. Use lgr::get_logger("bbotk") to access and control the logger.

Super classes

mlr3tuning::Tuner -> mlr3tuning::TunerFromOptimizer -> TunerHyperband

Methods

Public methods

• TunerHyperband$new()

• TunerHyperband$clone()

Inherited methods

• mlr3tuning::Tuner$format()

• mlr3tuning::Tuner$print()

• mlr3tuning::TunerFromOptimizer$optimize()

---

Method new()

Creates a new instance of this R6 class.

Usage

TunerHyperband$new()

---

Method clone()

The objects of this class are cloneable with this method.

Usage

TunerHyperband$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

Examples

if(requireNamespace("xgboost")) {
  library(mlr3learners)

  # define hyperparameter and budget parameter
  search_space = ps(
    nrounds = p_int(lower = 1, upper = 16, tags = "budget"),
    eta = p_dbl(lower = 0, upper = 1),
    booster = p_fct(levels = c("gbtree", "gblinear", "dart"))
  )

  # \donttest{
  # hyperparameter tuning on the pima indians diabetes data set
  instance = tune(
    method = "hyperband",
    task = tsk("pima"),
    learner = lrn("classif.xgboost", eval_metric = "logloss"),
    resampling = rsmp("cv", folds = 3),
    measures = msr("classif.ce"),
    search_space = search_space,
    term_evals = 100
  )

  # best performing hyperparameter configuration
  instance$result
  # }
}
#> Loading required namespace: xgboost
#>    nrounds       eta booster learner_param_vals  x_domain classif.ce
#> 1:       8 0.5512812  gbtree          <list[6]> <list[3]>  0.2395833