Celery Task received but not executing in Django project hosted on DigitalOcean

Question

I have a Django project set up with Celery, hosted on a DigitalOcean droplet running Ubuntu 22.04. I'm running into a peculiar issue where a task is received by the Celery worker but doesn't seem to execute it. Here's the task that I am trying to run (provided by django-cookiecutter):

from django.contrib.auth import get_user_model
from config import celery_app

User = get_user_model()

@celery_app.task()
def get_users_count():
    return User.objects.count()

I'm running the worker with the following command:

celery -A config.celery_app worker --loglevel=DEBUG --without-gossip --without-mingle --without-heartbeat -Ofair --pool=solo

I'm executing the task manually via django shell:

get_users_count.delay()
<AsyncResult: 821982b7-f256-462b-869c-e415da168c3f>

The worker logs show that it received the task, but not that it's executed:

[2023-10-06 23:57:25,568: INFO/MainProcess] Task myapp.users.tasks.get_users_count[821982b7-f256-462b-869c-e415da168c3f] received

The task appears in Redis:

127.0.0.1:6379> keys *
1) "_kombu.binding.celery.pidbox"
2) "celery-task-meta-821982b7-f256-462b-869c-e415da168c3f"
3) "_kombu.binding.celery"

When I query the task state and result manually, it shows that the task actually succeeded:

>>> res = AsyncResult("821982b7-f256-462b-869c-e415da168c3f")
>>> print("Task State: ", res.state)
Task State:  SUCCESS
>>> print("Task Result: ", res.result)
Task Result:  2

Despite the AsyncResult showing SUCCESS, there's no indication in the worker logs that the task was executed. What could be going wrong here? Any guidance would be greatly appreciated. The only thing that's really different from my local machine and this server is that I'm using requirepass to secure redis.

celery==5.3.1
django-celery-beat==2.5.0
Django==4.2.3

Edit:

Some more puzzling things, after logging a message it's appearing, but it still doesn't show that the task was executed:

@celery_app.task()
def get_users_count():
    """A pointless Celery task to demonstrate usage."""
    logger.info("get_users_count task executed")
    return User.objects.count()



[2023-10-07 00:55:11,000: INFO/MainProcess] mingle: searching for neighbors
[2023-10-07 00:55:12,012: INFO/MainProcess] mingle: all alone
[2023-10-07 00:55:16,573: INFO/MainProcess] Task myapp.users.tasks.get_users_count[c02f6522-2d42-4721-8600-d1c6426e67d0] received
[2023-10-07 00:55:16,577: INFO/ForkPoolWorker-2] get_users_count task executed

Edit2:

The counts are showing the right numbers:

@celery_app.task()
def get_users_count():
    """A pointless Celery task to demonstrate usage."""
    logger.info("get_users_count task executed")
    count = User.objects.count()
    logger.info(f"user count is {count}")
    return count

[2023-10-07 05:09:37,810: INFO/MainProcess] Task myapp.users.tasks.get_users_count[474736b2-d568-45f6-ae34-f512612cca2f] received
[2023-10-07 05:09:37,811: INFO/MainProcess] get_users_count task executed
[2023-10-07 05:09:37,821: INFO/MainProcess] user count is 2
[2023-10-07 05:11:01,741: INFO/MainProcess] Task myapp.users.tasks.get_users_count[61a3db97-712a-4fff-bb20-7d33e9e52aeb] received
[2023-10-07 05:11:01,742: INFO/MainProcess] get_users_count task executed
[2023-10-07 05:11:01,749: INFO/MainProcess] user count is 3

Answer 1

So everything WAS working correctly, the issue was that for whatever reason celery was not added as a logger in the production settings. I am not sure if the creators didn't include this on purpose and that we should keep it disabled.

Fix:

LOGGING = {
    "version": 1,
    "disable_existing_loggers": True,
    "formatters": {
        "verbose": {
            "format": "%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s",
        },
    },
    "handlers": {
        "console": {
            "level": "DEBUG",
            "class": "logging.StreamHandler",
            "formatter": "verbose",
        }
    },
    "root": {"level": "INFO", "handlers": ["console"]},
    "loggers": {
      "celery": {
        "level": "INFO",
        "handlers": ["console"],
        "propagate": True,
    },
    // other loggers here