Nested preferences inside fragment (from bottom navigation) - back stack not working

Question

Everything is working, except that I am not able to go back to the settings fragment. Why is the Fragment not called, when clicking on the back button?

Structure:

MainActivity -> SettingsFragment (inside NavController) -> Preferences Overview -> First Preference Category

SettingsFragment.java

public class SettingsFragment extends Fragment implements
    PreferenceFragmentCompat.OnPreferenceStartFragmentCallback{

private SettingsViewModel settingsViewModel;

public View onCreateView(@NonNull LayoutInflater inflater,
                         ViewGroup container, Bundle savedInstanceState) {

    settingsViewModel = ViewModelProviders.of(this).get(SettingsViewModel.class);
    final View root = inflater.inflate(R.layout.settings_activity, container, false);

    getActivity().getSupportFragmentManager()
            .beginTransaction()
            .replace(R.id.settings, new SetupSettingsFragment(), "SetupSettingsFragment")
            .addToBackStack(null)
            .commit();

    return root;
}

public static class SetupSettingsFragment extends PreferenceFragmentCompat {
    @Override
    public void onCreatePreferences(Bundle savedInstanceState, String rootKey) {
        setPreferencesFromResource(R.xml.preferences_overview, rootKey);
    }
}

@Override
public boolean onPreferenceStartFragment(PreferenceFragmentCompat caller, Preference pref) {

    final Bundle args = pref.getExtras();
    final Fragment fragment = getActivity().getSupportFragmentManager().getFragmentFactory().instantiate(
            getActivity().getClassLoader(),
            pref.getFragment());
    fragment.setArguments(args);
    fragment.setTargetFragment(caller, 0);

    int setupSettingsFragment = getFragmentManager().findFragmentByTag("SetupSettingsFragment").getId();

   getActivity().getSupportFragmentManager().beginTransaction()
            .replace(setupSettingsFragment, fragment)
            .addToBackStack(null)
            .commit();

    return true;
}

FirstPreferenceCategory.java

public class FirstPreferenceCategory extends PreferenceFragmentCompat {

@Override
public void onCreatePreferences(Bundle savedInstanceState, String rootKey) {
    setPreferencesFromResource(R.xml.PreferenceOverview, rootKey);
}}

}

PreferenceOverview.xml

<PreferenceScreen xmlns:android="http://schemas.android.com/apk/res/android"
android:key="preferenceScreenOverview"
xmlns:app="http://schemas.android.com/apk/res-auto">

    <Preference  
        app:title="FirstPreferenceCategory"
        app:fragment="com.ui.FirstPreferenceCategory"/>

Edit:

I also moved the onPreferenceStartManager and the SetupSettingsFragment to the MainActivity, but still there is something wrong with the back stack.