Creating A Multiprocessing Pool Takes Different Times

28 Views Asked by At
import multiprocessing
from time import time

def do_something(arg):
    # do something here
  
def main():
    t1 = time()
    pool = multiprocessing.Pool(processes=2)
    t2 = time()
    
    # Launch async processes to do something
    futures = pool.starmap_async(do_something, [a, b])
    
    # do some other task
    
    # Wait for processes to join
    pool.close()
    pool.join()

if __name__ == "__main__":
    for i in range(10):
        main()

The function main() is run multiple times to do some task and spawns 2 processes to do some other task in parallel, while the main() function performs its own task. However, over different runs the time (t2 - t1) is sometimes in the order of ~20 ms and other times it is approximately ~3-4s. I want to understand any possible reasons this could be happening.

I have gone through the documentation, but couldn't find a good reason. My system is a 12-core system and I have made sure that the system occupancy is low (I have tried running this piece of code after a reboot). Any help is highly appreciated.

0

There are 0 best solutions below