How to get the model results from a benchmark result?

29 Views Asked by At

I have been trying to use mlr3 to do some hyperparameter tuning for xgboost. I want to compare three different models:

1. xgboost tuned over just the alpha hyperparameter

2. xgboost tuned over alpha and lambda hyperparameters

3. xgboost tuned over alpha, lambda, and maxdepth hyperparameters.

library(mlr3proba)
library(xgboost)
library(tidyverse)
library(survival)
train_indxs = sample(seq_len(nrow(veteran)), 100)
task = as_task_surv(x = veteran, time = 'time', event = 'status')
poe = po('encode')
task = poe$train(list(task))[[1]]

xgb_learn <- lrn("surv.xgboost")
measure = msr("surv.cindex")
set.seed(103)
fivefold.cv = rsmp("cv", folds = 5)

param.list <- list(  alpha = p_dbl(lower = 0.001, upper = 100),
                     lambda = p_dbl(lower = 0.001, upper = 100),
                     max_depth = p_int(lower = 2, upper = 10)
)


model.list <- list()
for(model.i in 1:length(param.list)){
  
  param.list.subset <- param.list[1:model.i]
  search_space <- do.call(ps, param.list.subset)
  
  model.list[[model.i]] <- AutoTuner$new(
    learner = xgb_learn,
    resampling = fivefold.cv,
    measure = measure,
    search_space = search_space,
    terminator = trm("evals", n_evals = 50),
    tuner = tnr("random_search", batch_size = 50),
    store_tuning_instance = TRUE
  )
}
grid <- benchmark_grid(
  task = task,
  learner = model.list,
  resampling = rsmp("cv", folds =3)
)

bmr <- benchmark(grid, store_models = TRUE)

I wonder how to get the model results from a benchmark result.

but I still need to extract information. How can I access the individual runs of the benchmark?

Any help is appreciated. Thanks a lot!

0

There are 0 best solutions below