Python signal.SIGKILL & multiprocessing

Question

I'm trying by any means to return to prompt after Ctrl+C.

My app uses Thread executors, and multiprocess with threads as well. When I try to exit the multiprocess queues have TypeError issues. and prompt never comes back.

Traceback (most recent call last):
  File "/home/juanmf/projects/turret/env/lib/python3.11/site-packages/multiprocess/queues.py", line 248, in _feed
    obj = _ForkingPickler.dumps(obj)
...
  File "/home/juanmf/projects/turret/env/lib/python3.11/site-packages/dill/_dill.py", line 414, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/usr/lib/python3.11/pickle.py", line 578, in save
    rv = reduce(self.proto)
         ^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle '_queue.SimpleQueue' object

I'm not super interested in fixing pickle ATM. So I just tried hardcore SIGKILL. While I do get the prompt I can't restart the system due to resources RPi picamera. being taken by the python process that should have died.

Thread prints Dump Complete =======================================================

Killed
(env) juanmf@raspberrypi:~/projects/turret/src $ ps aux | grep pyth
root         956  0.0  0.4 102112 19256 ?        Ss   19:52   0:00 python /usr/sbin/wayvnc-control.py
juanmf      1100  0.0  0.8  63164 34052 ?        S    19:52   0:00 /usr/bin/python3 /usr/share/system-config-printer/applet.py
juanmf      2284  0.0  2.1 1185628 83564 pts/0   Sl   20:18   0:00 python3 Training.py
juanmf      2291  0.0  2.1 1251204 83280 pts/0   Sl   20:18   0:00 python3 Training.py
juanmf      2298  0.0  2.1 1316784 83304 pts/0   Sl   20:18   0:00 python3 Training.py

Note: My app has mainProcess and only one child Sub process, so I find it confusing that after killing Main I still see 3 processes in ps aux output. After pgrep python3 | xargs kill -9 I can re-run the app.

The nasty approach:

# Register the interrupt handler for the interrupt signal (Ctrl+C) or kill
signal.signal(signal.SIGINT, interrupt_handler)
signal.signal(signal.SIGTERM,  interrupt_handler)

...
def interrupt_handler(signum, frame):
    try:
        for process, proxyController in MultiProcessingControllerFactory.runningProcesses:
            print(process.pid, signal.SIGKILL)
            process.terminate()
            os.kill(process.pid, signal.SIGKILL)

        process = current_process()
        # exit the process
        if process._popen is not None:
            # Accessing protected because processing does not make any safety check here.
            process.terminate()

        GPIO.cleanup()
        flush_streams() // Prints buffered logs
        time.sleep(2)
    finally:
        os.kill(os.getpid(), signal.SIGKILL) // Kills this process.
        exit(0)

Edit:

I can run my app in mono process or multi process mode. On multi process I only spawn one child (the only method that runs Process.start() prints "SPAWNING!!" and I read that output only once. as well). But if I suspend (Ctrl+Z) the application and run ps aux | grep python:

mono process output

[1]+  Stopped                 python3 Training.py
(env) juanmf@raspberrypi:~/projects/turret/src $ ps aux | grep pyth
root         956  0.0  0.4 102112 19256 ?        Ss   19:52   0:00 python /usr/sbin/wayvnc-control.py
juanmf      1100  0.0  0.8  63164 34052 ?        S    19:52   0:00 /usr/bin/python3 /usr/share/system-config-printer/applet.py
juanmf      2078 43.0  3.6 1246880 142360 pts/0  TLl  20:12   0:06 python3 Training.py

multiprocess output

[1]+  Stopped                 python3 Training.py
(env) juanmf@raspberrypi:~/projects/turret/src $ ps aux | grep pyth
root         956  0.0  0.4 102112 19256 ?        Ss   19:52   0:00 python /usr/sbin/wayvnc-control.py
juanmf      1100  0.0  0.8  63164 34052 ?        S    19:52   0:00 /usr/bin/python3 /usr/share/system-config-printer/applet.py
juanmf      2269 43.3  3.7 1468804 144136 pts/0  TLl  20:18   0:07 python3 Training.py
juanmf      2284  0.1  2.1 1185644 83564 pts/0   Tl   20:18   0:00 python3 Training.py
juanmf      2291  0.0  2.1 1251204 83280 pts/0   Tl   20:18   0:00 python3 Training.py
juanmf      2298  0.0  2.1 1316784 83304 pts/0   Tl   20:18   0:00 python3 Training.py
juanmf      2306  0.0  2.1 1464968 83636 pts/0   Tl   20:18   0:00 python3 Training.py

After self-killing and exit, I see 3 processes. I guess it's MainProcess and child process that die. (5 - 2 = 3) where are those 3 unregistered process coming from? (Also, the PIDs that go are the 1st and the last.)

Answer 1

For some Reason the mutiprocess Queue's TypeError (on or after queueing & pickling poison pills) prevented the finally block from running.

• The three extra processes were due to m = Manager() invocations (from multiprocess), I wrapper id in a Singleton and call Manager() only once. Then at exit_handler:

    if SharedManager.instance:
        SharedManager.getInstance().getManager().shutdown()

SharedManager implementation

exit(0) after closing sub-processes work. Except when your app has I/O errors. I'm using sshkeyboard lib which crashes on 1st Ctrl-C, second time exits to prompt, that doesn't happen with os.kill().