Randomized CV value error with estimator KerasClassifier

Question

Trying to use Random CV parameter array but get error message as;

ValueError: Invalid parameter model_optimizer_learning_rate for estimator KerasClassifier.
This issue can likely be resolved by setting this parameter in the KerasClassifier constructor:
`KerasClassifier(model_optimizer_learning_rate=0.01)`
Check the list of available parameters with `estimator.get_params().keys()`

code:

def create_model_v4(lr,batch_size):  
    np.random.seed(1337)
    model = Sequential()
    model.add(Dense(256,activation='relu',input_dim = X_train.shape[1]))
  ............................................................................
    model.add(Dense(32,activation='relu')) 
    model.add(Dense(1, activation='sigmoid'))

    #compile model
    optimizer = tf.keras.optimizers.Adam(learning_rate=lr)
    model.compile(optimizer = optimizer,loss = 'binary_crossentropy', metrics = ['accuracy'])
    return model

keras_estimator = KerasClassifier(build_fn=create_model_v4, verbose=1)


# define the grid search parameters
param_random = {
    'batch_size':[32, 64, 128],
    "lr":[0.01,0.1,0.001],}

kfold_splits = 3
random= RandomizedSearchCV(estimator=keras_estimator,  
                    verbose=1,
                    cv=kfold_splits,  
                    param_distributions=param_random,n_jobs=-1)
random_result = random.fit(X_train, y_train,validation_split=0.2,verbose=1) 

# Summarize results
print("Best: %f using %s" % (random_result.best_score_, random_result.best_params_))
means = random_result.cv_results_['mean_test_score']
stds = random_result.cv_results_['std_test_score']
params = random_result.cv_results_['params']

I have tried lr for learning_rate as suitable, I have tried optimizer_lr etc, but possibly I am not implementing the solutions I am finding correctly.

Answer 1

Learning rate is a parameter of the optimizer, not the model. So, In the SciKeras wrapper you need to route the parameters to the optimizer. You can do it with the prefix optimizer__ in the grid dict.

Try the following dictionary

param_random = {
    'batch_size':[32, 64, 128],
    "optimizer__learning_rate":[0.01,0.1,0.001],
    #"optimizer__lr":[0.01,0.1,0.001],
}

Not sure if you should use the lr parameter as your constructor does, or the Keras optimizer default parameter (learning_rate). Try both and choose the suitable one.

I suggest you the following resource to fine tune Keras models with the SciKeras wrapper. This case and other are addressed.

EDIT:

Working code to test the learning rate:

import numpy as np
import tensorflow as tf
from keras.models import Sequential
from keras.layers import Dense
from scikeras.wrappers import KerasClassifier
from sklearn.model_selection import RandomizedSearchCV

# Assuming X_train is your training data
# Replace this with your actual training data
num_features=256
X_train = np.random.rand(100, num_features)
y_train = (np.random.rand(100, 1)>0.5).astype(int)
def create_model_v4():
    np.random.seed(1337)
    model = Sequential()
    model.add(Dense(256, activation='relu', input_dim=X_train.shape[1]))
    model.add(Dense(32, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))

    # compile model
    optimizer = tf.keras.optimizers.Adam()
    model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])
    return model

keras_estimator = KerasClassifier(build_fn=create_model_v4, verbose=1)

# define the grid search parameters
param_random = {
    'batch_size': [32, 64, 128],
    "optimizer__learning_rate": [0.01, 0.1, 0.001],
}

kfold_splits = 3

random = RandomizedSearchCV(
    estimator=keras_estimator,
    verbose=1,
    cv=kfold_splits,
    param_distributions=param_random,
    n_jobs=-1
)
random_result = random.fit(X_train, y_train, validation_split=0.2, verbose=1)

# Summarize results
print("Best: %f using %s" % (random_result.best_score_, random_result.best_params_))

# Assuming X_test is your testing data
# Replace this with your actual testing data
X_test = np.random.rand(10, num_features)

# Now, you can use the trained model to make predictions on X_test
predictions = random_result.best_estimator_.predict(X_test)

# Print the predictions
print(predictions)

>> Best: 0.540107 using {'optimizer__learning_rate': 0.1, 'batch_size': 32}
1/1 [==============================] - 0s 63ms/step
[[0]
 [0]
 [0]
 [0]
 [0]
 [0]
 [0]
 [0]
 [0]
 [0]]

https://machinelearningmastery.com/grid-search-hyperparameters-deep-learning-models-python-keras/