Why doesn't an error stop a parallel python script

33 Views Asked by At

My intension is to raise an error and stop the Python script when certain condition met. Here is the relevant code snippet.

def my_func(some arguments):
    # some code
    if X.abs().max().max() > 1e50:
        print('------------ ERROR! line 287, extremely large X={} ----------------'.format(X.abs().max().max()), file=sys.stderr)
        print('selectedFeatures: '+('-'.join(sorted(list(selectedFeatures)))), file=sys.stderr)
        print('featurePool: '+('-'.join(sorted(list(featurePool)))), file=sys.stderr)
        print('p1: '+p1+',  p2: '+p2, file=sys.stderr)
        print('p1: '+p1+',  p2: '+p2, file=sys.stderr)
        print('X.shape={}X{}'.format(X.shape[0],X.shape[1]), file=sys.stderr)
        tmp = 1/0 # to create an error
    # some code

def caller_func():
    for-loop:
        # some code
        my_func(arguments)
        # some code

Part of the output is below, and the script keeps running even though there are errors.

# some stdout output content

------------ ERROR! line 287, extremely large X=3.130749212136059e+58 ----------------
selectedFeatures: <some_string>
featurePool: <some_string>
p1: <some_string>, p2: <some_string>
p1: <some_string>, p2: <some_string>
X.shape=133615X41
/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_1.csv' mode='w' encoding='UTF-8'>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_2.csv' mode='w' encoding='UTF-8'>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "//script_heavy_ga.py", line 332, in caller_func
    X, selectedFeatures, featurePool = my_func(p1, p2, X, X_original, selectedFeatures, featurePool)
  File "//script_heavy_ga.py", line 293, in my_func
    tmp = 1/0 # to create an error
ZeroDivisionError: division by zero

# some stdout output content

------------ ERROR! line 287, extremely large X=1.1940462536258555e+75 ----------------
selectedFeatures: <some_string>
featurePool: <some_string>
p1: <some_string>, p2: <some_string>
p1: <some_string>, p2: <some_string>
X.shape=183025X41
/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_3.csv' mode='w' encoding='UTF-8'>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/multiprocessing/process.py:-1: ResourceWarning: unclosed file <_io.TextIOWrapper name='my_file_4.csv' mode='w' encoding='UTF-8'>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Traceback (most recent call last):
  File "//script_heavy_ga.py", line 332, in caller_func
    X, selectedFeatures, featurePool = my_func(p1, p2, X, X_original, selectedFeatures, featurePool)
  File "//script_heavy_ga.py", line 293, in my_func
    tmp = 1/0 # to create an error
ZeroDivisionError: division by zero

# some stdout output content

Parallel computing is used in my code from multiprocessing import Pool. Multiple instances of the function caller_func() run on multiple cores.

My question is, why isn't the script stopped by the errors? Some previous posts (and this) don't seem relevant. This might be relevant, but it did not answer why my code did not stop.

0

There are 0 best solutions below