Celery throwing error when started with --pool gevent but works fine otherwise

Question

I have a celery task like so:

from celery import Celery
from asgiref.sync import async_to_sync
from celery_app.celery_commands import some_async_command
import os

redis_connection_string = os.environ.get("REDIS_URL")
celery_app = Celery(
    'tasks', backend=f"{redis_connection_string}/0", broker=f"{redis_connection_string}/1")

@celery_app.task()
def sample_task(**kwargs):
    result = async_to_sync(some_async_command)(**kwargs)
    return result

When I use run celery for this task with : celery -A celery_app.tasks worker --loglevel=INFO -n my_app_worker --concurrency=4, it works fine. However, since the async function some_async_command is mainly I/O-bound and makes calls to the network, I was considering using gevent pool. I have installed gevent, but when I run celery -A celery_app.tasks worker --loglevel=INFO -n my_app_worker --concurrency=4 --pool gevent, it throws the error:

Usage: celery [OPTIONS] COMMAND [ARGS]...
Try 'celery --help' for help.

Error: Invalid value for '-A' / '--app': 
Unable to load celery application.
Module 'select' has no attribute 'epoll'

What am I missing?

Answer 1

Thing with gevent is that the first thing it does is perform a monkey patch to make the standard library cooperative so that they behave as coroutines.

When you use the gevent pool, the gevent module adds itself ahead of the standard library to the sys.path and replaces already-loaded modules in the list of modules. This is what allows you to run gevent without having to use gevent-specific modules.

I suspect that you use urllib3? In that case, urllib3 chooses gevent's select.py over the standard library select module which explains the error you are seeing.

It might be possible that gevent is not monkey patching as early as possible or that it is related to package versions you are using.