Selection Rate in selection_rate_group_summary in fairlearn

Question

How does selection_rate_group_summary decide the cutoff value when we pass the output of a mitigated model to it in fairlearn module.

The selection rate in one of our data is really high all the time while using the function which is unnatural.

Answer 1

There is no cutoff value. selection_rate_group_summary simply collects selection rates for all groups and overall.

Example:

>>> from fairlearn.metrics import selection_rate_group_summary
>>> y = [0, 1, 1, 1, 1, 1, 1, 1]
>>> sensitive_features = ["a", "b", "a", "b", "a", "b", "a", "b"]
>>> selection_rate_group_summary(y, y, sensitive_features=sensitive_features)
{'overall': 0.875, 'by_group': {'a': 0.75, 'b': 1.0}}

Similarly, if we have lots and lots of samples and the numbers get very large, this still works (no cutoff):

>>> y = [1] * 1000000
>>> y[0] = 0
>>> sensitive_features = ["a", "b"] * 500000
>>> selection_rate_group_summary(y, y, sensitive_features=sensitive_features)
{'overall': 0.999999, 'by_group': {'a': 0.999998, 'b': 1.0}}

If you meant something else please update the question or write a message to the community on Gitter