Recently I have pushed around 10-15k tasks into celery by iterating in a for loop. Attaching below code for reference.
for index, htl_urls_chunk in enumerate(iteration_grouper(htl_urls, n=10, chunk_type=list)):
update_cache_for_hotel_detail_page_chunk.apply_async(kwargs={'htl_urls_chunk': htl_urls_chunk})
the length of htl_urls is around: 10-15k. Total tasks pushed ~ 1500 at given point of time.
This lead to creation of below keys in redis:
01f6ebdb-361b-32ec-afe9-2eb7c8511288.reply.celery.pidbox
6292044c-a816-349a-a159-682536ea9bfa.reply.celery.pidbox
Each key is of 150MB and total 29 keys created (4.24GB) This lead to OOM issue of my redis cluster.
Can anyone suggest why this is happening ?
I have tried to explored why these keys are being created in my redis cluster, but not able to reach any conclusion so far.