This package provides hyperband tuning for mlr3. Various termination criteria can be set and combined. The class ‘AutoTuner’ provides a convenient way to perform nested resampling in combination with ‘mlr3’.
If you are already familiar with
mlr3tuning, then the only change compared to other tuners is to give a numeric hyperparameter a
budget tag. Afterwards, you can handle hyperband like all other tuners:
library(paradox) library(mlr3tuning) library(mlr3hyperband) # give a hyperparameter the "budget" tag params = list( ParamInt$new("nrounds", lower = 1, upper = 16, tags = "budget"), ParamDbl$new("eta", lower = 0, upper = 1), ParamFct$new("booster", levels = c("gbtree", "gblinear", "dart")) ) inst = ... # here goes the usual mlr3tuning TuningInstance constructor # initialize hyperband tuner tuner = tnr("hyperband", eta = 2L) # tune the previously defined TuningInstance tuner$optimize(inst)
For the full working example, please check out the Examples section below.
Hyperband is a budget oriented-procedure, weeding out suboptimally performing configurations early on during their training process aiming at increasing the efficiency of the tuning procedure. For this, several brackets are constructed with an associated set of configurations for each bracket. These configuration are initialized by stochastic, often uniform, sampling. Each bracket is divided into multiple stages, and configurations are evaluated for a increasing budget in each stage. Note that currently all configurations are trained completely from the beginning, so no online updates to the models are performed.
Different brackets are initialized with different number of configurations, and different budget sizes. To identify the budget for evaluating hyperband, the user has to specify explicitly which hyperparameter of the learner influences the budget by tagging a single hyperparameter in the parameter set with
"budget". An alternative approach using subsampling and pipelines is described further below.
Originally, hyperband was created with a “natural” learning parameter as the budget parameter in mind, like
nrounds of the XGBoost learner:
library(mlr3) library(mlr3hyperband) # hyperband tuner library(mlr3tuning) # tuning methods library(mlr3learners) # xgboost learner library(paradox) # search space definition set.seed(123) # Define hyperparameter and budget parameter for tuning with hyperband params = list( ParamInt$new("nrounds", lower = 1, upper = 16, tags = "budget"), ParamDbl$new("eta", lower = 0, upper = 1), ParamFct$new("booster", levels = c("gbtree", "gblinear", "dart")) ) # Initialize TuningInstance as usual # hyperband terminates on its own, so the terminator acts as a upper bound inst = TuningInstanceSingleCrit$new( task = tsk("iris"), learner = lrn("classif.xgboost"), resampling = rsmp("holdout"), measure = msr("classif.ce"), search_space = ParamSet$new(params), terminator = trm("none") # hyperband terminates on its own ) # Initialize Hyperband Tuner and tune tuner = tnr("hyperband", eta = 2L) tuner$optimize(inst) # View results inst$result
Additionally, it is also possible to use
mlr3hyperband to tune learners that do not have a natural fidelity parameter. In such a case
mlr3pipelines can be used to define data subsampling as a preprocessing step. Then, the
frac parameter of subsampling, defining the fraction of the training data to be used, can act as the budget parameter:
library(mlr3pipelines) set.seed(123) ll = po("subsample") %>>% lrn("classif.rpart") # Define extended hyperparameters with subsampling fraction as budget and hence # no learner budget is required params = list( ParamDbl$new("classif.rpart.cp", lower = 0.001, upper = 0.1), ParamInt$new("classif.rpart.minsplit", lower = 1, upper = 10), ParamDbl$new("subsample.frac", lower = 0.1, upper = 1, tags = "budget") ) # Define TuningInstance with the Graph Learner and the extended hyperparams inst = TuningInstanceSingleCrit$new( tsk("iris"), ll, rsmp("holdout"), msr("classif.ce"), ParamSet$new(params), trm("none") # hyperband terminates on its own ) # Initialize Hyperband Tuner and tune tuner = tnr("hyperband", eta = 4L) tuner$optimize(inst) # View results inst$result