how do I recover an old python environment?

Question

I am in a bit of a problem, as a server was upgraded, and some old python environments are no longer working. I cannot just activate the environment, but I would like to create a new environment which is a replica of the old environment to the extent possible.

This means installing on the new environment the same version of packages from the old environment.

Is there a way to extract this information from the old environment, for example, into a requirements.txt file? There doesn't seem to be anywhere a requirements.txt file in the old environment I can use.

EDIT: Thank you for the discussion. Yes, it was my mistake for not doing pip freeze before the upgrade, but I wasn't aware things are that flimsy.

Some of the code is old (and I know I will get comments about python 2.7 being used in some of the cases), but for example, when I run pip freeze from one of the activated packages I get:

# pip freeze

/.../venv/bin/python: error while loading shared libraries: libpython2.7.so.1.0: cannot open shared object file: No such file or directory

I know python 2.7 is installed on the machine, but I looked through all files in the whole server for this object file, and couldn't find it.

Answer 1

Here is a shell script which roughly approximates the functionality of pip freeze. I have tested it on an environment where I knew the answer (the Charcoal SmokeDetector project) and the results are approximately right; but there are many Python packages whose package names do not precisely align with the file names.

#!/bin/sh

: ${1?"Syntax: $0 <sdenv>">}

site_packages=""
for d in "$1"/lib/python3.?/site-packages "$1"/lib/python3.??/site-packages; do
    test -d "$d" || continue
    [ "$site_packages" ] && echo "$0: ignoring $site_packages" >&2
    site_packages="$d"
    break
done

: ${site_packages?"No lib/python3.*/site_packages found in $1"}

for package_dir in "$site_packages"/*/; do
    case $package_dir in
        *.dist-info/ | *.egg-info/ | */__pycache__/ ) continue;;
    esac
    found=""
    for metadata in "${package_dir%/}"-*.dist-info/METADATA \
                    "${package_dir%/}"-*.egg-info/PKG-INFO; do
        test -f "$metadata" || continue
        awk '/^Name:[ \t]*/ { name=$0; sub(/^Name:[ \t]*/, "", name) }
            /^Version:[ \t]*/ {ver=$0; sub(/^Version:[ \t]*/, "", ver) }
            name && ver { print name "==" ver; found=1; exit }
            END { exit(1-found) }' "$metadata" && found=yes && break
    done
    test "$found" && continue
    package_name=$(basename "$package_dir")
    if [ -f "${package_dir}__init__.py" ]; then
        if [ "${package_name}" != "${package_name#_}" ] &&
               [ -d "$site_packages/${package_name#_}" ]; then
            continue
        fi
        PYTHONPATH="${site_packages%/site-packages}${PYTHONPATH+:}${PYTHONPATH}" \
            python3 <<________HERE 2>/dev/null && found="yes"
import $package_name
ver = $package_name.__version__
print(f'$package_name=={ver}')
________HERE
    fi
    test "$found" && continue
    echo "$0: could not extract metadata for $package_name" >&2
    echo "$package_name"
done

In addition to that, this has the usual problems of pip freeze; it will list all installed packages, not just the ones you explicitly requested. So if for example you wanted to install the package se7en and it has a dependency on six, then six will also be included in the output, even if it will obviously be unnecessary if a future version of se7en drops the dependency on six.

For the Smokey dependencies, the above script displays warnings for dns (the actual package name is dnspython), attr (I think the actual package name is attrs but it weirdly creates a directory with the singular form), msgpack (the actual package name is msgpack-python), and pkg_resources (does not expose a version number). It could arguably be changed to fix a few of these one way or another, but there are then probably other corner cases which are left uncovered. Consider this a rough sketch anyway.