Cleaning up old directories left around by crashing PyInstaller one-file mode programs

Question

We have a Python application built with PyInstaller into a one-file executable (for Linux) and it therefore creates a /tmp/_MEIxxxxxx directory for unpacking all the files into, before running the Python portion.

Due to the use of certain third party shared libraries, this program occasionally dumps core, which means the directory where it was unpacked does not get deleted. We have a workaround in place that recovers by restarting the application but the /tmp filesystem grows in size until we run into other problems.

We are working on fixing the underlying issue but, in the meantime, we'd like to try to clean up the directories ourselves until that happens.

I know about the --runtime-tmpdir build-time option but this appears to only affect where the individual _MEIxxxxxx directories are created, not where the code is directly unpacked to.

But even if we could cause the executable to go to one specific place (rather than a random directory under that one specific place), we have the added issue that we actually run several copies of this application on each box and each needs its own directory.

So, ideally there would be an option you could run during invocation (not at build-time) of the executable which would tell PyInstaller where all the files should go, something like:

my_prog.exe --unpack-to-dir /tmp/MYPROG-1 ... python arguments for instance 1
my_prog.exe --unpack-to-dir /tmp/MYPROG-2 ... python arguments for instance 2

That would run instance one by unpacking to /tmp/MYPROG-1 and kicking up the Python code. Ditto for instance two but in the other directory.

But, since I can't see any way to do that, I'm looking for other solutions which allow me to keep the directories as clean as possible but still keep instances away from each other.

One option I have thought of is to have a cleanup task run periodically which can recognise relevant directories for my_prog (there may be other PyInstaller things running) and, if no active process is using it, clean the directory. It would just have to run often enough to quickly delete directories that were "orphaned" and avoid conflicts in the time between "starting to unpack" and "starting to run".

Recognition could be based on certain files within the directory. Active process detection could use procfs or the use of a watchdog file that the running Python code updated every minute (for example). This seems ... sub-optimal ... so I'm hoping there's a better solution.

The Python version in use is 2.7.12 (it's a legacy product, so very little chance of upgrading unfortunately) and PyInstaller is 3.6.

Answer 1

Given the constraints, I think what you described is your best bet, with one minor tweak [1]. So just so we're on the same page:

• on startup, write a known file into your MEI temp dir, so that you are reasonably confident that the temp dir really belongs to your app, and not some other pyinstaller-based app like you mentioned. Inside that file, write out the process ID.
• also on startup, run your cleanup task. Have it look for all MEI subdirs that contain that file that identifies it as your app. If it's your app and that process no longer exists (or that PID exists but is for some other application), clean up that dir. The psutil library should make this fairly straightforward.

[1] I don't think you need to run the cleanup task in the background - if the app shuts down cleanly, the temp dir will get cleaned up, so really just doing cleanup on startup should prevent you from having a whole bunch of these directories remaining.