EDIT: problem/issue/solution is explained here: https://github.com/wireservice/csvkit/issues/898 A fix involves setting the environment variable PYTHONIOENCODING
Today I learned I need to use the -e ENCODING option to deal my input data (it's ascii with some characters > 0x7f. "extended ascii") to csvcut, so for example
csvcut -v -e latin-1 -c AGE,NAME input.txt
works as expected, however
cat input.txt | csvcut -v -e latin-1 -c AGE,NAME
does not, failing with an error
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 22: invalid continuation byte
In other words, csvcut appears to be ignoring the -e option in the 2nd case.
the file input.txt is
NAME,AGE
Jimmie, 10
Ñandor, 357
Ñ is hex 0xd1
Why would the results be different thru the pipe? Is there a fix?