How can I fix python multiprocessing error?

Question

I am starting to learn about python multiprocessing. I am starting with this simple code. I am using Anaconda and the Spyder IDE to implement this code in. I am using Windows 10 Pro Operating System.

import time
import multiprocessing

start = time.perf_counter()

def do_something():
    print('Sleeping for 1 second ...')
    time.sleep(1)
    print('Done Sleeping ...')


p1 = multiprocessing.Process(target=do_something)
p2 = multiprocessing.Process(target=do_something)

p1.start()
p2.start()

p1.join()
p2.join()

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} in seconds')

When I run the above code I get the following error message:

Finished in 0.15 in seconds

Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 122, in spawn_main
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>
    exitcode = _main(fd, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Users\Mahmoud\anaconda3\Lib\multiprocessing\spawn.py", line 132, in _main
    self = reduction.pickle.load(from_parent)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: Can't get attribute 'do_something' on <module '__main__' (built-in)>

I can see that is printed the last print statement but then it gives me the above error. Why do I get this error? What do I need to do to the code to fix this error?

Answer 1

I was in your situation just a few months ago and I ended up having more success with concurrent.future than multiprocessing.

import time
import concurrent.futures

def some_cpu_bound_task(num):
    print(f"in cpu task {num}")
    range_to_sum = list(range(int(10000000)))  # create a big list to sum
    result = sum(range_to_sum)  # sum the list (CPU-intensive)
    return num ** 3  # return result of power operation

def some_io_bound_task(num):
    print(f"in io task {num}")
    time.sleep(10)  # simulated I/O delay
    return num * num

def start_calculations():
    with concurrent.futures.ProcessPoolExecutor(16) as executor:
        tasks = []
        for i in range(16):
            tasks.append(executor.submit(some_cpu_bound_task, i))
            tasks.append(executor.submit(some_io_bound_task, i))
    return tasks

def wait_all_calculations(tasks):
    for future in concurrent.futures.as_completed(tasks):
        result = future.result()
        print(f"Result: {result}")

if __name__ == "__main__":
    all_tasks = start_calculations()
    wait_all_calculations(all_tasks)

I think your code might not work because you need to be in main, like the examples in the docs.