I have been trying to use mlr3 to do some hyperparameter tuning for xgboost. I want to compare three different models:
1. xgboost tuned over just the alpha hyperparameter
2. xgboost tuned over alpha and lambda hyperparameters
3. xgboost tuned over alpha, lambda, and maxdepth hyperparameters.
library(mlr3proba)
library(xgboost)
library(tidyverse)
library(survival)
train_indxs = sample(seq_len(nrow(veteran)), 100)
task = as_task_surv(x = veteran, time = 'time', event = 'status')
poe = po('encode')
task = poe$train(list(task))[[1]]
xgb_learn <- lrn("surv.xgboost")
measure = msr("surv.cindex")
set.seed(103)
fivefold.cv = rsmp("cv", folds = 5)
param.list <- list( alpha = p_dbl(lower = 0.001, upper = 100),
lambda = p_dbl(lower = 0.001, upper = 100),
max_depth = p_int(lower = 2, upper = 10)
)
model.list <- list()
for(model.i in 1:length(param.list)){
param.list.subset <- param.list[1:model.i]
search_space <- do.call(ps, param.list.subset)
model.list[[model.i]] <- AutoTuner$new(
learner = xgb_learn,
resampling = fivefold.cv,
measure = measure,
search_space = search_space,
terminator = trm("evals", n_evals = 50),
tuner = tnr("random_search", batch_size = 50),
store_tuning_instance = TRUE
)
}
grid <- benchmark_grid(
task = task,
learner = model.list,
resampling = rsmp("cv", folds =3)
)
bmr <- benchmark(grid, store_models = TRUE)
I wonder how to get the model results from a benchmark result.
but I still need to extract information. How can I access the individual runs of the benchmark?
Any help is appreciated. Thanks a lot!