Exclude files staged as deleted fom `git ls-files`

Question

I use git ls-files in a script to collect all files tracked by git. Unfortunately if you rm a file (rather than git rm) it does not exist on the file system any more but it will still be listed by git ls-files

So the question: Is there a simple and efficient way to exclude files which do not exist on file system any more from either the git ls-files output itself or by filtering it afterwards (using e.g. bash)? Something like

git ls-files --existing-only

Background: I want to create a CMake dummy target, which holds all files that are part of the project directory (i.e. tracked by git). And I use something like

execute_process(
    COMMAND bash -c "cd ${CMAKE_SOURCE_DIR}; git ls-files"
    OUTPUT_VARIABLE ADDITIONAL_PROJECT_FILES
)

to generate the file list. But unfortunately rming a file and not yet staging the change will result in errors because the file can't be found by CMake any more..

Update: Before my edit I was talking about git rming a file - which will be handled correctly by git ls-files. But the problem persists: If someone removes a file (without using git for it) git ls-files will list it (and I run into trouble).

Answer 1

comm -23 <(git ls-files | sort) <(git ls-files --deleted | sort)

From comm --help:

  -2              suppress column 2 (lines unique to FILE2)
  -3              suppress column 3 (lines that appear in both files)

Answer 2

I don't think you can do this with git ls-files itself (a minor shame since it's otherwise pretty handy for these sorts of things).

Ultimately, to Git, the fact that a file doesn't exist in the work-tree is not important for the next commit. The commit uses whatever's in the index / staging-area. The main focus of git ls-files is on the index contents, hence options like --stage and --debug. However, the presence of the --others option proves that git ls-files can scan the work-tree. Having scanned the work-tree, it can then show you which files are in the work-tree but missing from the index: --others. This list can be further subtracted-away-from, using --exclude-standard and the like.

Your particular case is one where, instead of taking a list of work-tree files and subtracting away those that are in the index, you would like to take a list of index files and subtract away those that aren't in the work-tree. That is, if we define I as the set of index files and W as the set of work-tree files, we we see that git ls-files can easily compute W \ I. We'd like to have it compute I \ W, the set of files that are in the index but missing from the work-tree, so that we can remove them from git ls-files's output.

Alas, there's no such option. That leaves you with git diff-files, which can easily compute this set of files: git diff-files --name-only --diff-filter=D HEAD gets you the files that are in the index but missing from the work-tree. Use this list to remove filenames from the git ls-files output, and you have what you need.

Important sidebar

Often, files that are missing from the work-tree, but are present in the index, are in this state by mistake, and the correct thing to do is to extract them from the index to the work-tree. Also, any time you plan to do something with the set of files that are in the index right now, you should account for the fact that the copies of those files in the work-tree may differ from the copies in the index. For instance, a Git pre-commit hook might be intended to make sure that files are correctly formatted for some source language: that running them through clang-format or black would leave them unchanged, for example.

Testing for this by examining the work-tree files is fundamentally wrong, because Git isn't going to build the commit from the work-tree. What this means is that these tools should extract the entire index into a temporary (new and empty) work-tree somewhere else in the file system, and then run the formatter, or whatever tool is to be used, on that temporary work-tree. The result of this process tells you whether the commit should proceed. You then remove the temporary work-tree (i.e., clean up after yourself) and provide the appropriate "can commit" / "no, don't commit" exit status.