How might I persuade Kombu to use simplejson instead of the json module?

Question

Is there a good way to persuade Celery+Kombu to use simplejson instead of the standard library's json module?

I'm on:

tact@tact_pub_api:/app$ python3.10 -m pip list -v | egrep -i 'celery|kombu'
celery                  5.2.3       /usr/local/lib/python3.10/site-packages pip
kombu                   5.2.4       /usr/local/lib/python3.10/site-packages pip
tact@tact_pub_api:/app$ python2.7 -m pip list -v | egrep -i 'celery|kombu'
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
celery                        3.1.26.post2 /usr/local/lib/python2.7/dist-packages pip
kombu                         3.0.37       /usr/local/lib/python2.7/dist-packages pip

...and I'm getting a traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/kombu/serialization.py", line 39, in _reraise_errors
    yield
  File "/usr/local/lib/python3.10/site-packages/kombu/serialization.py", line 210, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.10/site-packages/kombu/utils/json.py", line 68, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "/usr/local/lib/python3.10/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/local/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.10/site-packages/kombu/utils/json.py", line 58, in default
    return super().default(o)
  File "/usr/local/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

What I'm hearing is that this is caused by recent versions of Kombu using the python standard library's json module, instead of pypi's simplejson module (https://docs.celeryq.dev/projects/kombu/en/stable/changelog.html#rc1). Apparently the standard library's "json" module cannot (de)serialize bytes, but pypi's "simplejson" can. Older versions of Celery+Kombu use simplejson by default.

EG:

>>> import json
>>> s = b'abc'
>>> json.dumps(s)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/local/lib/python3.10/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.10/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.10/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

My team/project may or may not want to move to the json module's serialization eventually, but for now it seems to make the most sense to use simplejson for the sake of interoperability.

(This is part of a largish CPython 2.7 -> CPython 3.10 port; I just want things to work on 3.10 first, and then worry about updating dependencies to be modern. Right now, the client and server are on different versions of celery and kombu)

Thanks!

Answer 1

Perhaps you can monkeypatch kombu from your code, after importing it, to use simplejson instead of json. From the traceback above:

File "/usr/local/lib/python3.10/site-packages/kombu/serialization.py", line 210, in dumps
    payload = encoder(data)

If encoder is a global name, rebind kombu.encoder to a function that uses simplejson. If not, but dumps is, rebind kombu.dumps.

Or, this pair

File "/usr/local/lib/python3.10/site-packages/kombu/utils/json.py", line 68, in dumps
    return _dumps(s, cls=cls or _default_encoder,

suggests that you can set kombu.utils.json.cls to something other than None to change behavior. Check the kombu docs or read the module code.