mpiexec + python + ^C: __del__ method not executed (and no traceback)

Question

I have the following test_mpi.py python script:

from mpi4py import MPI
import time

class Foo:
    def __init__(self):
        print('Creation object.')
    def __del__(self):
        print('Object destruction.')

foo = Foo()
time.sleep(10)

If I execute it without recourse to mpiexec, using a simple python test_mpi.py, pressing CTRL+C after 5s, I get the following output:

ngreiner@Nathans-MacBook-Pro:~/Documents/scratch$ python test_mpi.py 
Creation object.
^CTraceback (most recent call last):
  File "test_mpi.py", line 26, in <module>
    time.sleep(10)
KeyboardInterrupt
Object destruction.
ngreiner@Nathans-MacBook-Pro:~/Documents/scratch$

If I embed it within an mpiexec execution, using mpiexec -np 1 python test_mpi.py, again pressing CTRL+C after 5s, I now get:

ngreiner@Nathans-MacBook-Pro:~/Documents/scratch$ mpiexec -np 1 python test_mpi.py 
Creation object.
^Cngreiner@Nathans-MacBook-Pro:~/Documents/scratch$

The traceback from python and the execution of the __del__ method have disappeared. The main problem for me is the non-execution of the __del__ method, which is supposed to make some clean-up in my actual application.

Any idea how I could have the __del__ method executed when the Python execution is launched from mpiexec ?

Thank you very much in advance for the help,

(My system configuration: macOS High sierra 10.13.6, Python 3.7.4, open-mpi 4.0.1, mpi4py 3.0.2.)

Answer 1

Per documentation, it is not guaranteed that del would be called. So you are lucky that it is called on non-mpi program.

For simple case, you could use try/finally to be sure that finally section is executed. Or, more generically, use context manager

Here is a quote from documentation that is important here:

It is not guaranteed that del() methods are called for objects that still exist when the interpreter exits.

Answer 2

After a bit of search, I found a solution to restore the printing of the traceback and the execution of the __del__ method when hitting ^C during mpiexec.

During a normal python execution (not launched by mpiexec, launched directly from the terminal), hitting ^C sends a SIGINT signal to python, which translates it into a KeyboardInterrupt exception (https://docs.python.org/3.7/library/signal.html).

But when hitting ^C during an mpiexec execution, it is the mpiexec process which receives the SIGINT signal, and instead of propagating it to its children processes (for instance python), it sends to its children processes a SIGTERM signal (https://www.open-mpi.org/doc/current/man1/mpirun.1.php).

It thus seems that python doesn't react similarly to SIGINT and SIGTERM signals.

The workaround I found is to use the signal module, and to use a specific handler for the SIGTERM signal, which simply raises a KeyboardInterrupt. This can be achieved by the following lines:

def sigterm_handler():
    raise KeyboardInterrupt

import signal
signal.signal(signal.SIGTERM, sigterm_handler)

The former can be included at the top of the executed python script, or, to retain this behaviour each time python is used with mpiexec and with the mpi4py package, at the top of the __init__.py file of the mpi4py package.

This strategy may have side-effects (which I am unaware of) and should be used at your own risk.

Answer 3

The answer by ngreiner helped me, but at least with Python 2.7 and all Python 3 versions, the handler function needs two arguments. This modified code snippet with dummy arguments worked for me:

import signal

def sigterm_handler(signum, frame):
    raise KeyboardInterrupt

signal.signal(signal.SIGTERM, sigterm_handler)