What is the relationship between eps and the number of features considered by the regularization model?

Question

I observed that as I adjust the eps parameter downwards, the model considers more variables (i.e. less variables have a coef of zero). Is there an explanation for this trend?

Figure 1: eps == 0.001

Figure 2: eps == 0.0001

Answer 1

ElasticNetCV calculates a regularization path [min_alpha, max_alpha] of length n_alphas and runs cross-validation to determine the best alpha. Here, max_alpha is the value for which ElasticNet produces all-zero solution, and min_alpha equals eps * max_alpha. So, for smaller alphas we expect fewer zero coefficients as the objective Loss + min_alpha * Regularization approaches ordinary least squares.