How to monitor memory usage of individual celery tasks?

Question

I would like to know the max memory usage of a celery task, but from the documentations none of the celery monitoring tools provide the memory usage feature. How can one know how much memory a task is taking up? I've tried to get the pid with billiard.current_process and use that with memory_profiler.memory_usage but it looks like the current_process is the worker, not the task.

Thanks in advance.

Answer 1

Celery does not give this information unfortunately. With little bit of work it should not be difficult to implement own inspect command that actually samples each worker-process. Then you have all necessary data for what you need. If you do this, please share the code around as other people may need it...