pip install on embeddable python with Jenkins

Question

I have a Windows Jenkins worker which has an embeddable python and it needs to install some packages on it. The python already has pip, installed using get-pip.py. I also modified python38._pth as seen in another SO post:

python38.zip
.
Lib
Lib/site-packages
# Uncomment to run site.main() automatically
#import site

Not sure if it helps, but the relevant part of the pipeline is:

dir('env'){
    bat '''
        .\\python.exe -m pip config --user set global.extra-index-url "<url>"
        .\\python.exe -m pip install -r requirements.txt --force-reinstall
        .\\python.exe -m pip list
    '''
        }

When using python 3.8.9 the pipeline gives the following error:

ERROR: Exception:
Traceback (most recent call last):
  File "D:\jenkins\workspace\<repo-name>_<branch-name>\env\Lib/site-packages\pip\_internal\cli\base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "D:\jenkins\workspace\<repo-name>_<branch-name>\env\Lib/site-packages\pip\_internal\cli\req_command.py", line 248, in wrapper
    return func(self, options, args)
  File "D:\jenkins\workspace\<repo-name>_<branch-name>\env\Lib/site-packages\pip\_internal\commands\install.py", line 452, in run
    installed = install_given_reqs(
  File "D:\jenkins\workspace\<repo-name>_<branch-name>\env\Lib/site-packages\pip\_internal\req\__init__.py", line 72, in install_given_reqs
    requirement.install(
  File "D:\jenkins\workspace\<repo-name>_<branch-name>\env\Lib/site-packages\pip\_internal\req\req_install.py", line 780, in install
    scheme = get_scheme(
  File "D:\jenkins\workspace\<repo-name_><branch-name>\env\Lib/site-packages\pip\_internal\locations\__init__.py", line 249, in get_scheme
    old = _distutils.get_scheme(
  File "D:\jenkins\workspace\<repo-name>_<branch-name>\env\Lib/site-packages\pip\_internal\locations\_distutils.py", line 141, in get_scheme
    scheme = distutils_scheme(dist_name, user, home, root, isolated, prefix)
  File "D:\jenkins\workspace\<repo-name>_<branch-name>\env\Lib/site-packages\pip\_internal\locations\_distutils.py", line 80, in distutils_scheme
    i.finalize_options()
  File "distutils\command\install.py", line 344, in finalize_options
  File "distutils\command\install.py", line 487, in convert_paths
  File "distutils\util.py", line 122, in convert_path
ValueError: path '/scratch/data/jenkins/workspace/<repo-name>_<branch-name>_tmp/.local' cannot be absolute

The weird part is I also tried with a python 3.10 embeddable and with the exact same pipeline, and on that version it worked without a single problem.

This is the first time I'm using Jenkins, so apologies if I seem a bit lost. Any help would be appreciated

Answer 1

It seems my mistake was having PYTHONUSERBASE defined as an environment variable in the pipeline. Even though using pip with embedded python puts the packages in the local site-packages directory by itself, the variable seems to interfere with the process. (I assume while trying to figure out where to install the packages)