I have mixed data features (3 continuous, 2 binomial, and 3 ordinal categorial) and the binomial (1,0) target. I have label-encoded the ordinal categorical features (e.g. instead of very low/low/medium/high/very high there are 1,2,3,4,5 in a column). I do not have any nans in my data. So it is a binomial classification problem.
I have the following code:
#Initialize a tree (Decision Tree with max depth = 2)
dt = DecisionTreeClassifier(max_depth=2, random_state = 32)
#Initialize an AdaBoost classifier with the tree as the base estimator
ada_boost = AdaBoostClassifier(base_estimator = dt, algorithm="SAMME.R", random_state=32)
parameter_grid = {
'base_estimator__max_depth':[6,8,10,12,14],
'n_estimators':[ 160, 180, 200],
'learning_rate':[0.9, 1, 1.1]
}
grid_search3 = GridSearchCV(estimator = ada_boost, cv = 5, param_grid = parameter_grid, scoring="f1").fit(cross_val_df.drop("left", axis=1), cross_val_df["left"])
grid_search3.best_params_
#Out: {'base_estimator__max_depth': 14, 'learning_rate': 1, 'n_estimators': 200}
# train and fit the model
abc = AdaBoostClassifier(base_estimator = DecisionTreeClassifier(max_depth=14), n_estimators = 200, learning_rate= 1, random_state = 32)
abc.fit(cross_val_df.drop("left", axis=1), cross_val_df["left"])
abc.feature_importances_
# Out: array([nan, nan, nan, nan, nan,
nan, nan, nan])
with another random seed, almost the same.
Out: array([0.00236466, nan, nan, nan, nan,
nan, nan, nan])
When I did the similar analysis with RandomForestClassifier(), I got the following array of feature importances:
array([0.00369934, 0.27461498, 0.01995202, 0.0162823 , 0.10218577,
0.27397469, 0.00924151, 0.3000494 ]).
Can anybody explain to me why the feature importances are so different, and there are no feature importances with AdaBoostClassifier()?
My goal is to select 3 most important features that lead to the target y =1. so I want to optimise the f1, or recall/precision.