Multiprocessing Drops CPU to 0

Question

I have a list of data frames around 40k and I need to process each dataframe seperately.I have used multiprocessing pool.starmap with total cpus-2 as processes to process each dataframe parallely. Process starts without any issue and suddenly cpu usage drops to 0 and runs continuosly without stopping the process And if we manually run the process again it works sometimes.

I have tried to run in batches and with sleep time for each batch, things ran fine few times but again same issue started.

Answer 1

I don't know if you've found the answer to your problem. I ran into the same issue tonight, and after a bit of research, I found out that it was because I didn't close my pool that the process just ran continuously without stopping.

Here is my code :

cpu_pool = multiprocessing.Pool(processes=WORKERS_NUMBER)
cpu_pool.starmap(func, args)
cpu_pool.close()

Adding the .close() allowed my program to finish running. However, I also experience drops in the CPU usage throughout the execution of the function. Still don't know why, any idea ?