AdaBoostClassifier feature_importances_ parameter returns nan

173 Views Asked by At

I have mixed data features (3 continuous, 2 binomial, and 3 ordinal categorial) and the binomial (1,0) target. I have label-encoded the ordinal categorical features (e.g. instead of very low/low/medium/high/very high there are 1,2,3,4,5 in a column). I do not have any nans in my data. So it is a binomial classification problem.

I have the following code:

#Initialize a tree (Decision Tree with max depth = 2)

dt = DecisionTreeClassifier(max_depth=2, random_state = 32)

#Initialize an AdaBoost classifier with the tree as the base estimator

ada_boost = AdaBoostClassifier(base_estimator = dt, algorithm="SAMME.R", random_state=32)


parameter_grid = {   
    'base_estimator__max_depth':[6,8,10,12,14],
    'n_estimators':[ 160, 180, 200],
    'learning_rate':[0.9, 1, 1.1]
}

grid_search3 = GridSearchCV(estimator = ada_boost,  cv = 5, param_grid = parameter_grid, scoring="f1").fit(cross_val_df.drop("left", axis=1), cross_val_df["left"])

grid_search3.best_params_
#Out: {'base_estimator__max_depth': 14, 'learning_rate': 1, 'n_estimators': 200}

# train and fit the model

abc = AdaBoostClassifier(base_estimator = DecisionTreeClassifier(max_depth=14), n_estimators = 200, learning_rate= 1, random_state = 32)
abc.fit(cross_val_df.drop("left", axis=1), cross_val_df["left"])

abc.feature_importances_
# Out: array([nan,        nan,        nan,        nan,        nan,
              nan,        nan,        nan])

with another random seed, almost the same.
Out: array([0.00236466,        nan,        nan,        nan,        nan,
              nan,        nan,        nan])

When I did the similar analysis with RandomForestClassifier(), I got the following array of feature importances:

array([0.00369934, 0.27461498, 0.01995202, 0.0162823 , 0.10218577,
       0.27397469, 0.00924151, 0.3000494 ]).

Can anybody explain to me why the feature importances are so different, and there are no feature importances with AdaBoostClassifier()?

My goal is to select 3 most important features that lead to the target y =1. so I want to optimise the f1, or recall/precision.

0

There are 0 best solutions below