Issue while sharing semaphore objects between process in parallel python

Question

I am facing issue while passsing any semaphore objects or synchronizing objects like Events, pipes , queues etc to a child process when used with parallel python I am getting following error when i passed a Queue to child process. ' through inheritance' % type(self).__name__ RuntimeError: Queue objects should only be shared between processes through inheritance

No issue was seeen when i used a multiprocessing library

import pp
import time
from multiprocessing import Event, Queue, Manager, Pool

e = Queue()

def startt() :
    e.put(1)
    return 1


ppservers = ()
# Creates jobserver with automatically detected number of workers        
jobServer = pp.Server(ppservers=ppservers,proto=2)
#submit the work to the job pool or job server
job = jobServer.submit(startt,(e,),(),("time", ), globals = globals())
res = job()
print e.get()
print res

Answer 1

You're having this issue because you're passing the queue object when you submit the job:

job = jobServer.submit(startt,(e,),(),("time", ), globals = globals())
#                              ^ here

Two things to note:

1. The queue is already global. You don't need to pass it as an argument.

2. You can pass the queue as an argument if you use a multiprocessing.Manager:

manager = multiprocessing.Manager()
shared_queue = manager.Queue() # this shared queue can be passed as an argument

The call to manager.Queue() will actually return a proxy to the queue rather than the queue itself, and the manager process will handle all of the synchronization necessary in the background.