How to switch branches when a feature branch introduces submodule changes?

Question

My master branch uses two submodules:

[submodule "content-repo"]
    path = content-repo
    url = https://github.com/dogweather/forkful.git
[submodule "themes/ananke"]
    path = themes/ananke
    url = https://github.com/theNewDynamic/gohugo-theme-ananke.git

But I'm changing out one of these submodules in a feature branch, hextra-yet-again Because I'm switching to a new theme. This branch's .gitmodules:

[submodule "content-repo"]
    path = content-repo
    url = https://github.com/dogweather/forkful.git
[submodule "themes/hextra"]
    path = themes/hextra
    url = https://github.com/imfing/hextra.git

The "problem" is left-behind changes when I switch back to master. I get two warnings:

$ git checkout master
warning: unable to rmdir 'themes/hextra': Directory not empty
M   content-repo
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

1. So, the feature branch has updated with a newer commit of the content-repo submodule. And I guess I'm seeing that git checkout doesn't set this to where master was.

2. The themes/hextra's submodule dirs & files are not removed.

Am I doing this right? It doesn't feel right.