Is it OK to use `* -text diff` in .gitattributes?

Question

We have a large collection of files we want to put in a Git repository. Line endings include LF, CRLF, and mixed LF/CRLF. It's important in our context to keep these files exactly as-is, byte-for-byte.

Git likes to store text files with LF line endings. Normally if you have a case that must be CRLF, you should add .gitattributes entries to indicate that it should be written to the working directory as CRLF but in our case there are no easy patterns. That is, we can't just say *.foo eol=crlf or /bar/** eol=crlf.

We thought about using * binary, but then we learned that according to GitHub:

The binary setting is also an alias for -text -diff.

We had the idea that we could disable Git's line-ending conversion by writing a gitattributes containing only:

* -text diff

and get all the diff/history/merge behaviors of text files, without the line ending conversions.

Are there any downsides would should be aware of with this approach? Does it affect more than just line ending encoding?