How do I log a model with metrics and plots in MLRun?

Question

I'm training a model using MLRun and would like to log the model using experiment tracking. What kinds of things can I log with the model? I'm specifically looking for metrics (i.e. accuracy, F1, etc.) and plots like loss over time

Answer 1

MLRun has the ability to automatically log models with metrics and plots generated and attached.

You will use something like

from sklearn import ensemble
from mlrun.frameworks.sklearn import apply_mlrun

# Get data
X_train, X_test, y_train, y_test = ...

# Pick an ideal ML model
model = ensemble.RandomForestClassifier()
    
# Wrap our model with Mlrun features, specify the test dataset for analysis and accuracy measurements
apply_mlrun(model, model_name='my_model', X_test=X_test, y_test=y_test)
    
# Train our model
model.fit(X_train, y_train)

The result is a model logged in the experiment tracking framework with metrics, code, logs, plots, etc. available per run. The MLRun auto-logger supports standard ML frameworks such as SciKit-Learn, TensorFlow (and Keras), PyTorch, XGBoost, LightGBM, and ONNX.

Alternatively, you can log something manually using the MLRun context object that is available during the run. This lets you do things like context.log_model(...), context.log_dataset(...) or context.logger.info("Something happened"). More info on the MLRun execution context can be found here.