Beat scheduler not working automatically but does work when manually triggered

Question

I am learning to use celery. I want to automate API triggers using beat. The follwing code works if I manually call increment_value.delay() from the shell. However, it does not automate. Also, I think Redis still gets updated at every interval. Just the task method does not get called.

config.py

broker_url = f'redis://{REDIS_IP}:6379/0'
result_backend = f'redis://{REDIS_IP}:6379/0'
task_serializer = 'json'
result_serializer = 'json'
accept_content = ['json']
timezone = 'UTC'

CELERY_BEAT_SCHEDULE = {
    'run-every-15-seconds': {
        'task': 'scheduler.increment_value',
        'schedule': timedelta(seconds=15),
    },
}

scheduler.py

app = Celery()
app.config_from_object('config')


@app.task
def increment_value():
    response = requests.post(url=f"http://{FASTAPI_IP}:8000/redis/first")

    logging.info("Task Has Been Triggered")

    return response.json()

Dockerfile

FROM python:3.8-slim

WORKDIR /app

# Copy only the necessary files
COPY ./requirements.txt /app/requirements.txt
COPY . /app

# Install dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Start Celery worker and beat in the background using a shell script
COPY ./start.sh /app/start.sh
RUN chmod +x /app/start.sh

CMD ["/app/start.sh"]

start.sh

#!/bin/sh
celery -A scheduler worker --loglevel=info &
celery -A scheduler beat --loglevel=info

Both worker and beat start when I run the application.

I should also mention that I am working on WSL2.

I would really appreciate if someone could point it out to me what I am doing wrong.

Answer 1

You are using a different configuration variable to set beat schedule. Replace CELERY_BEAT_SCHEDULE with CELERYBEAT_SCHEDULE and see if it works. Also as @Shivam suggested, you can use supervisor instead of running worker in the background for better signal handling mechanisms.

Answer 2

For a docker container to run, the running process must have a pid of 1. Your celery app might me running in the background.

Also you can run only one service as the foreground.

Alternatively You can use a process control set up. Supervisord is process control system, which will run as pid 1 as needed by the container to run and it can manage the execution on your celery worker and scheduler.

Answer 3

It is probably a good idea to run both worker and beat in the "detached" mode.

Your start.sh script should look like the following:

#! /usr/bin/env bash

celery -A ${CELERY_APP} worker \
    --detach --loglevel=info \
    --pidfile=/var/run/celery/celery-worker.pid \
    --logfile=/var/log/celery/celery-worker.log

celery -A ${CELERY_APP} beat \
    --workdir=path/to/the/working/directory \
    --detach --loglevel=info \
    --pidfile=/var/run/celery/celery-beat.pid \
    --logfile=/var/log/celery/celery-beat.log

You probably want pid and log files to be saved elsewhere, not in /var/run/celery and /var/log/celery . For starters put them in the $HOME of the user who owns the Celery processes. $CELERY_APP environment variable should contain the name of your Celery application (in your case the name is "scheduler" - which is not really a good name for Celery app).