F1 metric and LeaveOneOut validation strategy in scikit-learn

Question

I want to use GridSearchCV to find the optimal n_neighbors parameter of KNeighborsClassifier

I want to use 'f1_score' metrics AND 'leave one out' strategy. But this code

clf = GridSearchCV(KNeighborsClassifier(), {'n_neighbors': [1, 2, 3]}, cv=LeaveOneOut(), scoring='f1')
clf.fit(x_train, y_train)

leads to an error

UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no true nor predicted samples. Use `zero_division` parameter to control this behavior.

I want to compute f1 score not of each fold of cross validation (it is not possible to compute f1 score of the only one test example), but to compute f1 score based on the whole iteration set with n_neighbors = n.

Is it possible using GridSearchCV?

Answer 1

Not sure if this functionality is directly available in Scikit-Learn, but you can implement the following function to get the desired outcome.

In particular, we will make a dummy scorer which just returns the predicted class instead of computing any score using the ground-truth and the prediction. In this way we can access the predictions of each hyperparameters combination on the different examples in the LOO cv.

from sklearn.metrics import f1_score, make_scorer

def get_pred(y_true, y_predicted):
    return y_predicted

get_pred_scorer = make_scorer(get_pred)
clf = GridSearchCV(
    KNeighborsClassifier(),
    {'n_neighbors': [1, 2, 3]},
    cv=LeaveOneOut(),
    refit=False,
    scoring=get_pred_scorer
)
clf.fit(X_train, y_train)

The problem with this approach is that certain results available in the cv_results_ dictionary (and in certain attributes of GridSearchCV) won't have any meaning, but that probably is not a problem. We should just remember to put refit=False, since GridSearchCV doesn't have a way to determine the best model.

Now we can access the predictions through cv_results_ and just use f1_score to compute the metric for each hyperparams configuration.

def print_params_f1_scores(clf, y_true):
    y_preds = [] # will contain the predictions of each params combination
    results = clf.cv_results_
    params = results["params"] # all params combinations 

    for j in range(len(params)): # for each combination
        y_preds.append([])
        for i in range(clf.n_splits_): # for each split (sample in loo)
            prediction_of_j_on_i = results[f"split{i}_test_score"][j]
            y_preds[j].append(prediction_of_j_on_i)

    # show the f1-scores of each combination
    for j in range(len(y_preds)):
        score = f1_score(y_true, y_preds[j])
        print(f"KNeighborsClassifier with {params[j]} obtained f1-score of {score}")

print_params_f1_scores(clf, y_train)

The function prints the following output:

KNeighborsClassifier with {'n_neighbors': 1} obtained f1-score of 0.94
KNeighborsClassifier with {'n_neighbors': 2} obtained f1-score of 0.94
KNeighborsClassifier with {'n_neighbors': 3} obtained f1-score of 0.92