I faced a problem that when running celery with eventlet and remote database (Redis) tasks come but are not executed, before that I used gevent and it worked correctly with both local and remote databases, but it does not have the ability to stop tasks, I tried all the options I found on the Internet and on StackOverflow, but none of them helped. I add my settings there
Django settings:
# settings.py
CELERY_BROKER_URL = REDIS_CONN_URL + "0"
CELERY_RESULT_BACKEND = REDIS_CONN_URL + "0"
CELERY_WORKER_CONCURRENCY = 1000
Celery settings:
import os
import ssl
from celery import Celery
from celery.schedules import crontab
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "SailAPI.settings")
celery_app = Celery(
"SailAPI",
broker_use_ssl={"ssl_cert_reqs": ssl.CERT_REQUIRED, "ssl_ca_certs": ssl.get_default_verify_paths().cafile},
redis_backend_use_ssl={"ssl_cert_reqs": ssl.CERT_REQUIRED, "ssl_ca_certs": ssl.get_default_verify_paths().cafile},
)
celery_app.config_from_object("django.conf:settings", namespace="CELERY")
celery_app.conf.update(
enable_utc=True,
timezone="UTC",
worker_concurrency=8,
worker_pool="eventlet",
)
celery_app.conf.beat_schedule = {
"update_user_status": {
"task": "user_app.tasks.update_user_status",
"schedule": crontab(hour="4", minute="0", day_of_week="sat"),
}
}
celery_app.autodiscover_tasks()
The problem is that tasks that are added to the queue are not executed. I use an eventlet (because it has the ability to stop tasks that are running) and when I run it locally (Windows) with a local database (redis) it works fine, accepts tasks, executes them, stops at the command, but when I threw the code on the server (Linux) and ran celery there with this command celery -A SailAPI.celery worker -l info -P eventlet -E I got the following logs
-------------- celery@sail-backend v5.3.1 (emerald-rush)
--- ***** -----
-- ******* ---- Linux-5.15.0-78-generic-x86_64-with-glibc2.35 2023-10-03 20:55:33
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: SailAPI:0x7f7f5dcf8d50
- ** ---------- .> transport: rediss://:<dedicate_database>:<dedicate_port>/0
- ** ---------- .> results: rediss://:<dedicate_database>:<dedicate_port>/0
- *** --- * --- .> concurrency: 8 (eventlet)
-- ******* ---- .> task events: ON
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. chat_gpt.tasks.send_message_to_gpt_task
. payment_system.tasks.create_payment_url
. payment_system.tasks.handle_payment_failed
. payment_system.tasks.handle_payment_success
. user_app.tasks.check_user_subscription
. user_app.tasks.update_user_status
[2023-10-03 20:55:33,395: ERROR/MainProcess] consumer: Cannot connect to rediss://:<dedicate_database>:<dedicate_port>/0: Error while reading from <dedicate_database>:<dedicate_port>: ('timed out',).
Trying again in 2.00 seconds... (1/100)
[2023-10-03 20:55:35,416: ERROR/MainProcess] consumer: Cannot connect to rediss://:<dedicate_database>:<dedicate_port>/0: Error while reading from <dedicate_database>:<dedicate_port>: ('timed out',).
Trying again in 4.00 seconds... (2/100)
[2023-10-03 20:55:39,437: ERROR/MainProcess] consumer: Cannot connect to rediss://:<dedicate_database>:<dedicate_port>/0: Error while reading from <dedicate_database>:<dedicate_port>: ('timed out',).
Trying again in 6.00 seconds... (3/100)
But if I run this on my local machine I got:
-------------- celery@DESKTOP-4FC47C2 v5.3.1 (emerald-rush)
--- ***** -----
-- ******* ---- Windows-10-10.0.19045-SP0 2023-10-03 21:01:24
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: SailAPI:0x22273aa5950
- ** ---------- .> transport: redis://localhost:16379/0
- ** ---------- .> results: redis://localhost:16379/0
- *** --- * --- .> concurrency: 1000 (eventlet)
-- ******* ---- .> task events: ON
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. chat_gpt.tasks.send_message_to_gpt_task
. payment_system.tasks.create_payment_url
. payment_system.tasks.handle_payment_failed
. payment_system.tasks.handle_payment_success
. user_app.tasks.check_user_subscription
. user_app.tasks.update_user_status
[2023-10-03 21:01:28,206: INFO/MainProcess] Connected to redis://localhost:16379/0
[2023-10-03 21:01:28,220: INFO/MainProcess] mingle: searching for neighbors
[2023-10-03 21:01:29,259: INFO/MainProcess] mingle: all alone
[2023-10-03 21:01:29,275: INFO/MainProcess] celery@DESKTOP-4FC47C2 ready.
[2023-10-03 21:01:29,284: INFO/MainProcess] pidbox: Connected to redis://localhost:16379/0.
[2023-10-03 21:01:56,854: INFO/MainProcess] Task user_app.tasks.check_user_subscription[1f39ac32-638a-4903-b489-7e2d87e26afb] received
[2023-10-03 21:01:56,858: WARNING/MainProcess] Checking user subscription...
[2023-10-03 21:01:56,879: DEBUG/MainProcess] (0.000) SELECT "user_profiles"."id", "user_profiles"."system_user_id", "user_profiles"."user_email", "user_profiles"."user_username", "user_profiles"."user_first_name", "user_profiles"."user_la
st_name", "user_profiles"."user_avatar", "user_profiles"."status", "user_profiles"."subscription", "user_profiles"."subscription_type", "user_profiles"."subscription_start_day", "user_profiles"."subscription_end_day", "user_profiles"."sub
scription_id", "user_profiles"."user_language", "user_profiles"."user_interests", "user_profiles"."assistance_interest_user", "user_profiles"."referral_utm_code", "user_profiles"."requests_per_day" FROM "user_profiles" WHERE "user_profile
s"."system_user_id" = 2 LIMIT 21; args=(2,); alias=default
[2023-10-03 21:01:56,880: WARNING/MainProcess] User subscription status: False
[2023-10-03 21:01:56,880: WARNING/MainProcess] User subscription id: None
[2023-10-03 21:01:56,880: WARNING/MainProcess] User subscription checked.
[2023-10-03 21:01:56,912: INFO/MainProcess] Task user_app.tasks.check_user_subscription[1f39ac32-638a-4903-b489-7e2d87e26afb] succeeded in 0.0470000000204891s: None
If on dedicate server I try run this command, I will receive tasks, but they will not be completed
celery -A SailAPI.celery worker --loglevel=DEBUG --concurrency=100 --without-gossip --without-mingle --without-heartbeat -Ofair
Celery Debug info:
[2023-10-03 20:27:11,424: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2023-10-03 20:27:11,425: DEBUG/MainProcess] | Worker: Building graph...
[2023-10-03 20:27:11,425: DEBUG/MainProcess] | Worker: New boot order: {StateDB, Beat, Timer, Hub, Pool, Autoscaler, Consumer}
[2023-10-03 20:27:11,754: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2023-10-03 20:27:11,754: DEBUG/MainProcess] | Consumer: Building graph...
[2023-10-03 20:27:11,762: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Agent, Heart, Gossip, event loop}
-------------- celery@sail-backend v5.3.1 (emerald-rush)
--- ***** -----
-- ******* ---- Linux-5.15.0-78-generic-x86_64-with-glibc2.35 2023-10-03 20:27:11
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: SailAPI:0x7f89c422d710
- ** ---------- .> transport: rediss://:<dedicate_database>:<dedicate_port>/0
- ** ---------- .> results: rediss://:<dedicate_database>:<dedicate_port>/0
- *** --- * --- .> concurrency: 100 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. chat_gpt.tasks.send_message_to_gpt_task
. payment_system.tasks.create_payment_url
. payment_system.tasks.handle_payment_failed
. payment_system.tasks.handle_payment_success
. user_app.tasks.check_user_subscription
. user_app.tasks.update_user_status
[2023-10-03 20:27:11,839: DEBUG/MainProcess] | Worker: Starting Hub
[2023-10-03 20:27:11,839: DEBUG/MainProcess] ^-- substep ok
[2023-10-03 20:27:11,839: DEBUG/MainProcess] | Worker: Starting Pool
[2023-10-03 20:27:11,839: DEBUG/MainProcess] ^-- substep ok
[2023-10-03 20:27:11,839: DEBUG/MainProcess] | Worker: Starting Consumer
[2023-10-03 20:27:11,839: DEBUG/MainProcess] | Consumer: Starting Connection
[2023-10-03 20:27:11,866: INFO/MainProcess] Connected to rediss://:<dedicate_database>:<dedicate_port>/0
[2023-10-03 20:27:11,866: DEBUG/MainProcess] ^-- substep ok
[2023-10-03 20:27:11,866: DEBUG/MainProcess] | Consumer: Starting Tasks
[2023-10-03 20:27:11,869: DEBUG/MainProcess] ^-- substep ok
[2023-10-03 20:27:11,870: DEBUG/MainProcess] | Consumer: Starting Control
[2023-10-03 20:27:11,870: DEBUG/MainProcess] ^-- substep ok
[2023-10-03 20:27:11,870: DEBUG/MainProcess] | Consumer: Starting event loop
[2023-10-03 20:27:11,870: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2023-10-03 20:27:11,870: INFO/MainProcess] celery@sail-backend ready.
[2023-10-03 20:27:11,870: DEBUG/MainProcess] basic.qos: prefetch_count->100
[2023-10-03 20:27:11,872: INFO/MainProcess] Task user_app.tasks.check_user_subscription[e4baa163-f3bf-4a85-a3f1-c5829e8ad90b] received
[2023-10-03 20:27:11,872: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0x7f89c418af20> (args:('user_app.tasks.check_user_subscription', 'e4baa163-f3bf-4a85-a3f1-c5829e8ad90b', {'lang': 'py', 'task': 'user_app.tasks.check_user_subscription', 'id': 'e4baa163-f3bf-4a85-a3f1-c5829e8ad90b', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': 'e4baa163-f3bf-4a85-a3f1-c5829e8ad90b', 'parent_id': None, 'argsrepr': '(2,)', 'kwargsrepr': '{}', 'origin': 'gen42524@DESKTOP-4FC47C2', 'ignore_result': False, 'stamped_headers': None, 'stamps': {}, 'redelivered': True, 'properties': {'correlation_id': 'e4baa163-f3bf-4a85-a3f1-c5829e8ad90b', 'reply_to': '90f45e4e-4760-3bb1-b8ed-6b7f5be3d767', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'redelivered': True}, 'priority': 0, 'body_encoding': 'base64', 'delivery_tag': 'e8b47759-3ee2-4177-a221-cb16093cad15'}, 'reply_to': '90f45e4e-4760-3bb1-b8ed-6b7f5be3d767', 'correlation_id': 'e4baa163-f3bf-4a85-a3f1-c5829e8ad90b', 'hostname': 'celery@sail-backend',... kwargs:{})
If I try run celery like this:
celery -A SailAPI.celery worker --loglevel=DEBUG --pool=eventlet --concurrency=100 --without-gossip --without-mingle --without-heartbeat -Ofair
[2023-10-03 21:15:47,506: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2023-10-03 21:15:47,507: DEBUG/MainProcess] | Worker: Building graph...
[2023-10-03 21:15:47,508: DEBUG/MainProcess] | Worker: New boot order: {Beat, StateDB, Timer, Hub, Pool, Autoscaler, Consumer}
[2023-10-03 21:15:47,512: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2023-10-03 21:15:47,512: DEBUG/MainProcess] | Consumer: Building graph...
[2023-10-03 21:15:47,518: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Heart, Mingle, Tasks, Control, Gossip, Agent, event loop}
-------------- celery@sail-backend v5.3.1 (emerald-rush)
--- ***** -----
-- ******* ---- Linux-5.15.0-78-generic-x86_64-with-glibc2.35 2023-10-03 21:15:47
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: SailAPI:0x7efd332d34d0
- ** ---------- .> transport: rediss://:<dedicate_database>:<dedicate_port>/0
- ** ---------- .> results: rediss://:<dedicate_database>:<dedicate_port>/0
- *** --- * --- .> concurrency: 100 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. chat_gpt.tasks.send_message_to_gpt_task
. payment_system.tasks.create_payment_url
. payment_system.tasks.handle_payment_failed
. payment_system.tasks.handle_payment_success
. user_app.tasks.check_user_subscription
. user_app.tasks.update_user_status
[2023-10-03 21:15:47,538: DEBUG/MainProcess] | Worker: Starting Pool
[2023-10-03 21:15:47,538: DEBUG/MainProcess] ^-- substep ok
[2023-10-03 21:15:47,538: DEBUG/MainProcess] | Worker: Starting Consumer
[2023-10-03 21:15:47,538: DEBUG/MainProcess] | Consumer: Starting Connection
[2023-10-03 21:15:47,566: ERROR/MainProcess] consumer: Cannot connect to rediss://:<dedicate_database>:<dedicate_port>/0: Error while reading from <dedicate_database>:<dedicate_port>: ('timed out',).
Trying again in 2.00 seconds... (1/100)
I every time will be recive that cannot connect to dedicate database, I dont know why, because if I run this for local database server(redis) everything good