In Python why is `KeyboardInterrupt` preventing the pickling of this object?

Question

I need a better method or I need to learn how to use the method I'm using more correctly. Normally, when I press cntrl-c my work gets pickled, but one time it did not. The subsequent time that I tried to open the pickle I got a ran out of input error. I need to know why this happened so that it does not happen again. As I'm running my code, every one hundred loops I will save the pickle and if I hit KeyboardInterrupt, theoretically my work is supposed to be pickled before the program stops. My hunch is that if I press KeyboardInterrupt while pickle.dump(obj, temp) is being executed then the file will overwrite the old file but as it's overwriting if the program is killed in the middle of the overwrite then the file will be sort of half-written. What I also do not understand is that after I hit KeyboardInterrupt the program should execute the line print("help me") but it's not, at least on the sole occasion where I tried that.

import pickle

def save_pickle(obj, file):
    temp = open(file, "wb")
    ##i have a feeling that if i hit keyboard interrupt
    ##while the below line is being executed that it won't
    ## save the pickle properly
    pickle.dump(obj, temp)
    temp.close()

class help_me():
    def __init__(self):
        pass

    def func1(self):
        try:
            self.func2()
        except KeyboardInterrupt:
            pass
        print('help me')  # this line did not get activated
        save_pickle(obj, file)

    def func2(self):
        #this function will take several days
        for i in range(600_000):
            time.sleep(1)
            if i % 100 == 0:
                save_pickle(obj, file)

Answer 1

As noted in the comments, if you terminate your program with SIGINT (i.e. CTRL+C) mid-way through pickling an object to a file, you may end up with a partially written (corrupt) file.

To avoid this, you can customize how your program responds to SIGINT so that the critical parts don't stop abruptly.

Before the loop, register a temporary signal handler for SIGINT that sets a "stop" flag instead of raising KeyboardInterrupt. In the loop, check for the stop flag, and break when it's set. After the loop, you can reset the signal handler to whatever it was before the loop.

def func2(self):
    stop = False
  
    def handle_sigint(_sig, _frame)
        nonlocal stop
        print("setting stop flag")
        stop = True

    original_handler = signal.getsignal(signal.SIGINT)
    signal.signal(signal.SIGINT, handle_sigint)

    while not stop:
        time.sleep(1)
        save_pickle(obj, file)

    signal.signal(signal.SIGINT, original_handler)

Answer 2

First, I have a few comments on your code:

• In the function save_pickle, if any exception happens while executing pickle.dump(obj, temp) such as a KeyboardInterrupt, the temp.close() statement won't be executed. To prevent that, you can use a context manager.
• Declaring the help_me class does not follow Python's idioms. Lose the parentheses and use pascal case notation like HelpMe.
• Instead of except: pass in func1 you should better use contextlib.suppress(KeyboardInterrupt): ....
• The line with print('help me') statement should get executed unless you press ctrl-c more than once. If you want that to happen anyway, you can use a finally statement.
• You can also lose the __init__ function since it doesn't do anything.

To wrap up, here's my correction of the code, with just a few modifications:

import pickle
import time

def save_pickle(obj, file_address):
    with open(file_address, "wb") as file_:
        pickle.dump(obj, file_)

class HelpMe:
    def func1(self):
        key_board_interrupt_happened = False
        try:
            self.func2()
        except KeyboardInterrupt:
            key_board_interrupt_happened = True
        finally:
            print('help me')
            if key_board_interrupt_happened:
                pass
                # You can do the final save of what is left here.

    def func2(self):
        for i in range(600_000):
            time.sleep(1)
            if i % 100 == 0:
                save_pickle(object(), "file_address")