Issues with DataLoader Reinstantiation and Resource Cleanup in Optuna Trials

Question

Description

I am using Optuna for hyperparameter optimization in my project. However, I've encountered a problem where DataLoader instances seem to accumulate over multiple trials. Despite creating a new DataLoader instance in each trial, it appears that the old instances are not being properly released. This issue becomes particularly noticeable in a multi-GPU environment, leading to excessive memory usage.

Questions

1. How should I properly release previous DataLoader instances when instantiating a new one in each Optuna trial?
2. What are the best practices for cleaning up DataLoader-related resources at the end of each trial?
3. Are there any specific considerations for improving memory efficiency in a multi-GPU environment?

Relevant Code

import optuna
from torch.utils.data import DataLoader

def your_dataset_function():
    # Code to prepare your dataset
    pass

def objective(trial):
    dataset = your_dataset_function()
    data_loader = DataLoader(dataset, batch_size=trial.suggest_int("batch_size", 32, 128))
    # Code for the trial's objective function
    # ...
    del data_loader
    torch.cuda.empty_cache()  # Clear GPU memory cache

study = optuna.create_study()
study.optimize(objective, n_trials=100)