Using two libraries with conflicting Python version requirements?

Question

I have two Python libraries that I need to run within the same environment. They are pptk and torch-scatter and do not have an overlapping Python version requirement; pptk <= 3.7 and torch-scatter >= 3.8. They both make somewhat heavy use of C++ to enhance their functionality and I doubt I have the technical skills required to update/downdate them for eithers required Python version.

Given that pptk is a plotting library the only solution I see is to create a Python 3.8 environment and install torch-scatter. Then write a script to take whatever data I wish to provide to pptk to display, pickle it to NamedTemporaryFile. Finally start a new process and pass the file name to it as an argument, the process would run a Python 3.7 environment with pptk installed, load the file, and display the data.

Is there are simpler solution than the one described? Is there some support in Python to call a differently Python versioned library and perform some automagic marshalling?

Answer 1

If no one else provides a better solution, and someone stumbles here in the future, here is the solution I've implemented WYSIWYG.

The function pptk_subprocess takes in the data that I display, performs some manipulations, and writes it to a NamedTemporaryFile. The file name is then passed into the commands string. This string starts up the Anaconda environment in which the pptk library lives and runs the current __file__ in that environment. In __main__ this file is loaded and the data is read to display. There is also the option to write some data back to the pptk_subprocess "context" through the same NamedTemporaryFile by shutil.copyfile data into it.

This solution seems to only work from the command line and not through an IDE's run functionality.

import os
import sys
import math
import time
import shutil
import tempfile
import subprocess
import numpy as np

import matplotlib.image as mpimg
from matplotlib.colors import ListedColormap


"""
Python 3.7 functions
"""
def subprocess_pptk_lidar_image(points, labels):
    import pptk
    """
    Plot a point cloud with pptk and return a TemporaryFile of a screenshot
    Note: User MUST .CLOSE() the file.
    """

    # Continually attempt to open pptk, plot, and capture an image. pptk sometimes selects a port in use.
    tmpimg = None
    got_image = False
    num_tries = 0
    while not got_image:
        if num_tries > 10:
            raise RuntimeError(f'Attempted to open pptk 10 times. Something is wrong.')

        tmpimg = tempfile.NamedTemporaryFile(suffix='.png', delete=False)
        try:
            v = pptk.viewer(points)
            v.attributes(labels)
            v.set(point_size=0.02)
            v.set(r=500)
            v.set(phi=math.radians(90))
            v.set(theta=math.radians(65))
            v.capture(tmpimg.name)
            time.sleep(1.5)
            got_image = True
        except Exception as e:
            num_tries += 1
            continue
    return tmpimg


def subprocess_interactive_lidar_pptk(points, labels):
    import pptk
    v = pptk.viewer(points[:, 0:3])
    v.attributes(labels)
    v.set(point_size=0.05)
    v.wait()  # Wait until the user hits enter.


"""
Python 3.8 functions
"""
def generate_colormap_from_labels_colors(labels: dict, colors: dict):
    """
    Given a dictionary of labels {int: 'label'} and colors {int: 'color'} generate a ColorMap
    """
    # If there is a label 'unclassified' label ensure its color is 'WhiteSmoke' and not 'Black'
    if 'unclassified' in labels.values():
        # Get the index of the unclassified label in the label_dict
        unclass_index = list(labels.keys())[list(labels.values()).index('unclassified')]
        colors[unclass_index] = 'WhiteSmoke'

    color_map = ListedColormap(colors.values())
    return color_map


def pptk_subprocess(points, labels=None, label_dict=None, color_dict=None, interactive=False):
    # Generate "fake" labels by using the Z values for coloring
    if labels is None:
        labels = np.copy(points[:, 2])
        labels = (labels - np.min(labels)) / np.ptp(labels)

    # Generate the labels as RGB values if a colordict is given
    if label_dict is not None and color_dict is not None:
        colormap = generate_colormap_from_labels_colors(label_dict, color_dict)
        labels = colormap(labels.astype(np.int32))

    # Package the data into a temporary file to hand to the subprocess
    datafile = tempfile.NamedTemporaryFile(suffix='.npz', delete=False)
    np.savez(datafile.name, points=points, labels=labels, interactive=np.array([interactive]))

    # Start a process that calls this file
    commands = f'C:\ProgramData\Anaconda3\Scripts\\activate.bat && conda activate torch-pptk-py37 && python {__file__} {datafile.name}'
    subprocess.run(commands, shell=True)

    # If we were not interactive the subprocess wrote and image back into the datafile
    if not interactive:
        plot_image = mpimg.imread(datafile.name)
        datafile.close()
        os.remove(datafile.name)
        return plot_image

    return None


if __name__ == '__main__':
    # Dumbly figure out which argument is the datafile path
    datafile_path = None
    for a in sys.argv:
        if os.path.isfile(a) and '.py' not in a:
            datafile_path = a

    # Load and parse the points and labels from the file
    data = np.load(datafile_path)
    points = data['points']
    labels = data['labels']
    interactive = data['interactive'][0]

    if interactive:
        # Display this plot and wait for it to close from user input
        subprocess_interactive_lidar_pptk(points, labels)
    else:
        # Generate an image of the plot and get a NamedTempFile with it as an image
        tmpimg = subprocess_pptk_lidar_image(points, labels)

        # Copy the image from the returned file into the datafile
        shutil.copyfile(tmpimg.name, datafile_path)

        # Close and delete the temporary file
        tmpimg.close()
        os.remove(tmpimg.name)