Pathos "Error has occured during the function import"

Question

When I try to execute this code:

import pathos
from network import *


def simulation(error_rate=1):
    network = Network(error_rate=1)
    network.gen_transceiver_pair(1)
    network.run()
    mend = network.master_transceivers[0].cycle[0, 2]
    send = network.slave_transceivers[0].cycle[0, 2]
    return send if send > mend else mend

if __name__ == '__main__':
    p = pathos.pools.ParallelPool(nodes=10)
    result = p.map(simulation, [1, 1, 1])

I'm getting this error:

An error has occured during the function import
Traceback (most recent call last):
  File "C:\Users\user\PycharmProjects\network_simu\venv\lib\site-packages\ppft\__main__.py", line 100, in run
    six.exec_(__fobj)
  File "<string>", line 1, in <module>
  File "<string>", line 6, in Network
NameError: name 'signal' is not defined
A fatal error has occured during the function execution
Traceback (most recent call last):
  File "C:\Users\user\PycharmProjects\network_simu\venv\lib\site-packages\ppft\__main__.py", line 109, in run
    __f = locals()[ppc.str_(__fname)]
KeyError: 'simulation'

Is it because I am generating a Network object inside of the function? Ideally I'd like to run the simulation 10^6-10^7 times with different error rate (1, .9, etc.). I tried to run with multiprocessing but couldn't pickle the function hence why I am trying with Pathos.

Edit: I am using signal from the blinker module. The code works when executed normally. I've done 10^6 iterations without any problem but it takes forever.

Edit2: I am using the signal as such:

class Pool:
    def __init__(self):
        Transceiver.pool_add.connect(self.add)
        Transceiver.pool_remove.connect(self.remove)
        Network.return_cycle.connect(self.set_cycle)
        self.cycle = 0
        self.pool = []

    def set_cycle(self, sender):
        self.cycle = sender.cycle

    def add(self, sender):
        if VERBATIM:
            print("added " + sender.t_id)
        self.pool.insert(0, sender)
        sender.cycle[0, 0] = self.cycle

    def remove(self, sender):
        if VERBATIM:
            print("removed " + sender.t_id)
        self.pool.remove(sender)
        sender.cycle[0, 2] = self.cycle

class Transceiver(NetworkComponent):
    __STATUS = ["off", "t_on", "r_on", "on"]
    pool_add = signal("pool_add")
    pool_remove = signal("pool_remove")
    __ID = iter(range(1, 41))

Edit3: So I removed the use of signal. Now I get this error:

Backend TkAgg is interactive backend. Turning interactive mode on.
An error has occured during the function import
Traceback (most recent call last):
  File "C:\Users\user\PycharmProjects\network_simu\venv\lib\site-packages\ppft\__main__.py", line 100, in run
    six.exec_(__fobj)
  File "<string>", line 1, in <module>
  File "<string>", line 7, in Network
NameError: name 'np' is not defined
A fatal error has occured during the function execution

np is numpy... weird

Answer 1

I'm the pathos author. First off, you are using a ParallelPool, which uses ppft... which uses dill.source to convert objects to source code, and then passes the source code to the new process to then build a new object and execute. You may want to try a ProcessPool, which uses multiprocess, which uses dill, which uses a more standard serialization of objects (like pickle). Also, when you are serializing code (either with dill or dill.source) you should take care to make sure the code is as self-encapsulated as possible. What I mean is that:

def sin2(x):
    import numpy as np
    return np.sin(x)**2

will have a better chance of serializing than

import numpy as np

def sin2(x):
    return np.sin(x)**2

because the latter relies on the serializer to trace down all the items within sin2 that are not defined. So, simply by adding all the imports inside any function you intend to use in parallel makes it much more likely to work. (Note that both of the above should work, but the former is easier on the serializer than the latter.) It's actually similarly true for functions and any other objects that are used inside the the namespace of the object you want to serialize, but are found by reference lookup in an the enclosing namespace. Think about it like this: the serializer is guaranteed to ship whatever function you are targeting... and essentially transports it to the other processor and executes that code. If there's a reference lookup that's also required, but not copied over... then it will fail. So, help it to not fail by simplifying the namespace hierarchy enclosing the code you want to ship, and reducing the number of dependencies outside of the code you want to ship.