Where is Python path being manipulated?

250 Views Asked by At

On one server I'm able to import a library, on a second one I'm not. The library I need to import is shared via NFS and reachable by both servers.

I found out that the Python path is different between the two servers, but the Python binary is the same, installed from standard ubuntu 16.04 repos, and $PYTHONPATH is unset on both servers.

Server 1:

$ echo $PYTHONPATH

$ python
>>> import sys; sys.path
['', '/usr/lib/python2.7', '...', '/usr/local/lib/python2.7/dist-packages', '/home/user/app/src/python', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/wx-3.0-gtk2']

Server 2:

$ echo $PYTHONPATH

$ python
>>> import sys; sys.path
['', '/usr/lib/python2.7', '...', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

How do I understand where is this library defined, in order to replicate the same behaviour to server 2?

The curious fact is that the library is placed between the paths /usr/local/lib and /usr/lib.

Edit: The python binary is the same (same version 2.7.12, and the checksums match) and it's installed from the ubuntu repos in the standard location /usr/bin/python.
I've played with the library site and all variables are the same (site.ENABLE_USER_SITE returns True).
I know I can manipulate the paths inside the script, or specify PYTHONPATH in /etc/profile.d/ for all servers, but I'd like to know where is the difference.

2

There are 2 best solutions below

0
Alberto Chiusole On BEST ANSWER

I've discovered the existence of path configuration files in Python. From the doc of the site module on Python 3:

It starts by constructing up to four directories from a head and a tail part. For the head part, it uses sys.prefix and sys.exec_prefix; empty heads are skipped. For the tail part, it uses the empty string and then lib/site-packages (on Windows) or lib/pythonX.Y/site-packages (on Unix and Macintosh). For each of the distinct head-tail combinations, it sees if it refers to an existing directory, and if so, adds it to sys.path and also inspects the newly added path for configuration files.
...
A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to sys.path. Non-existing items are never added to sys.path, and no check is made that the item refers to a directory rather than a file. No item is added to sys.path more than once. Blank lines and lines beginning with # are skipped. Lines starting with import (followed by space or tab) are executed.

In my case there was a file in /usr/local/lib/python2.7/dist-packages/easy-install.pth which was missing on the second server.

Disabling the automatic import of those files with python -S may turn out to be useful while debugging.

Unfortunately I haven't found a way to retrieve a list of directories where the path configuration files are read from.

3
Łukasz Ślusarczyk On

Sys module documentation says that sys.path is

A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default

See: https://docs.python.org/3/library/sys.html

So there must be different installation-dependent default. Check exact python versions. Maybe python modules binaries come from different apt repositories. Maybe python commands come from different locations in both servers - check it by:

which python

Also check site module - it seems that it controls extra paths. See: http://docs.python.org/3/library/site.html

I would import it and check site.ENABLE_USER_SITE Maybe it will give you some clue on where the difference is