I am building a product that uses Kubernetes. Now a user can submit requests to this cluster, which kicks off a bunch of potentially long-running tasks. I initially had intended to use Celery for this purpose: submit a bunch of tasks to other microservices. However, the crucial part is that the tasks depend on one another (I construct a DAG with the task dependencies dynamically for each request). I have found Celery to be lacking in support for general DAGs. The group and chain do some of this but it is not generic enough.
Now I am struggling to find a Celery alternative that is similarly lightweight, but allows me to submit tasks based on a DAG. The alternatives I considered:
- Apache Airflow: Native support for DAGs but is based on more fixed workflows, and constructing DAGs dynamically seems to be a bit of a second-class citizen.
- Dask: This seems like it may be able to do the trick, but also seems overkill for just submitting some HTTP requests to other micro services.
- Argo: Kubernetes-based, which is nice, but again seems overkill to spin up new kubernetes pods just to submit some HTTP requests.
- Prefect: would still use Dask under the hood (I believe) and I do not need all the additional features.
Which alternatives should I consider instead? Also feel free to point out if my conclusions about the alternatives I considered are not accurate.
I often use Python, but will happily consider a tool based on a different language.