Current project has few periodic celery async tasks which takes more than 1 hour to finish by celery worker. Ideally these tasks should have been properly designed and broken down into smaller tasks but it is not possible for now. What should be the best strategy now to handle those long running task. We are facing many instances where workers become un-responsive due to these tasks.
Is there any better way to handle those tasks?
One option we explored was using Kubernetes Job for those specific long running tasks.