UnicodeEncodeError with Git Bash but not with cmd.exe

Question

I'm making a simple API call to url, saving the data as a python dictionary d and print d.

import requests

r = requests.get(url)
d = r.json()
print(d)

When I execute the script via cmd.exe on Windows 10 everything works:

> python script.py
{'pagination': {'page': 1, 'pages': 2, 'per_page': 50, 'items': 58, 'urls': {'last': ...

But why does it throw an error when I run it via Git Bash? Can you help me understand the error?

$ git --version
git version 2.33.0.windows.2

$ python script.py
Traceback (most recent call last):
  File "C:\Users\...\Projects\discogs-data\script.py", line 6, in <module>
    print(d)
  File "C:\Users\...\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2153' in position 1544: character maps to <undefined>

I assume that Python tries to decode d by using the cp1252-encoding before printing it. But why does it have to decode d in the first place and why does it work with cmd.exe but not with Git Bash?

Answer 1

I think both(Windows and Git Bash) uses different encoding for standard output. You can confirm it via sys module.

>>> import sys
>>> print(sys.stdout.encoding)

From your traceback Git Bash uses cp1252 encoding which cannot encode all character that d is pointing to. If you still want to display characters in GIT bash terminal you can encode the string with 'cp1252' encoding and set errors as replace so any non-encodable characters will be render as ?

>>> a = '\u2153' # from your error message.
>>> a.encode('cp1252', errors='replace')
b'?'
>>> a.encode('cp1252', errors='replace').decode()
'?'