Pyenv-installed Python creates virtualenv with empty include dir, no python-config in bin

Question

Note: There are several similar questions, which I have seen and read. None of them are the precise problem I'm having, and none of their answers work for me.

I have installed several Python versions (Python 2.7, 3.8, 3.9, and 3.10) on my macOS Ventura system using Pyenv. The installed directories include a python-config file in /bin and an /include directory with many header files, so I do not need to install python-devel or similar:

$ ls -al /Users/williamsn/.pyenv/versions/3.10.9/bin/*-config
lrwxr-xr-x  1 williamsn  staff    17 Dec 19 17:24 
/Users/williamsn/.pyenv/versions/3.10.9/bin/python-config -> python3.10-config
lrwxr-xr-x  1 williamsn  staff    17 Dec 19 17:24 
/Users/williamsn/.pyenv/versions/3.10.9/bin/python3-config -> python3.10-config
-rwxr-xr-x  1 williamsn  staff  2073 Dec 19 17:24 
/Users/williamsn/.pyenv/versions/3.10.9/bin/python3.10-config
$ ls -al /Users/williamsn/.pyenv/versions/3.10.9/include/python3.10/
total 1192
drwxr-xr-x  86 williamsn  staff   2752 Dec 19 17:24 .
drwxr-xr-x   3 williamsn  staff     96 Dec 19 17:24 ..
-rw-r--r--   1 williamsn  staff   3224 Dec 19 17:24 Python.h
...
-rw-r--r--   1 williamsn  staff   3026 Dec 19 17:24 import.h
...
-rw-r--r--   1 williamsn  staff  48878 Dec 19 17:24 pyconfig.h
...
-rw-r--r--   1 williamsn  staff   2863 Dec 19 17:24 weakrefobject.h

As a best practice, I work exclusively in virtualenvs to keep the base Python installation clean and pristine. I create my virtualenvs using the venv module built in to Python 3:

$ python3.10 -m venv my_venv

After doing so, there are two problems:

1. The my_venv/include directory is empty (does not contain the python3.10 directory or any of the header files from the parent Python and does not symlink to the parent Python)
2. There is no python-config / python3-config / python3.10-config in my_venv/bin.

So, for things requiring the Python headers:

• If it tries to find the headers using python-config (which many projects do), it will fail.
• If it tries to find the headers using the Python prefix and adding /include to it (which many projects do, notably Boost), it will fail.
• If it tries to find the headers by importing sysconfig and calling sysconfig.get_paths(), it will succeed, albeit with paths outside the virtualenv.

Now, on a case-by-case basis, I can work around this, and already have. I can manually copy over python_config and then either export CPATH to add /Users/williamsn/.pyenv/versions/3.10.9/include/python3.10 or modify the virtualenv to symlink to /Users/williamsn/.pyenv/versions/3.10.9/include. These work, but they don't seem right to me. I can't make a global workaround anywhere (such as exporting CPATH in my Bash profile), because I'm working with multiple Python versions, and I'd end up with the wrong headers half the time. It seems to me like a virtualenv that has an empty include directory and is missing python-config when its parent has a full include directory and contains python-config is a broken virtualenv.

Is there an option I'm missing to include these pieces? Is this a bug I need to file against Python/venv?

Answer 1

I am afraid there is, and has ever been, no straightforward and universal measures for this issue so far (n.b., the current most-recent stable version is Python-3.12.2 at the time of writing). Considering that the issue was once long-standing since 2011 and closed in 2019 with no applicable solutions (Issue #169: python-config should be linked), the chance for the core developers of Python/venv to provide the functionality for it in the future seems slim to me, unfortunately.

Background

Here is my guess of why it is not supported by venv. In short, there is no definitive and solid way.

There is usually no direct or reliable way from a (statically compiled) Python (binary) command to find the corresponding include or (C-type) library paths. Instead, Python provides the utility command python-config, which basically works on the assumption that the first python-config in your command-line search path corresponds to the Python command of your interest. This assumption may be unreliable in general, and this is particularly the case if you combine venv with pyenv.

In the pyenv environment, python-config is fixed at ~/.pyenv/shims/python-config (providing you have installed pyenv in your home directory). The script python-config of pyenv, which is a small-size wrapper script common to all Python versions, works fine as long as you stay in the pure pyenv environment. However, it does not work as expected as soon as you get out of it with venv. Examine the following situation, for example.

% cd ~
% pyenv shell 3.12.2
% python -m venv .venv3_12
% pyenv shell 3.9.0
% python --version  # => 3.9.0
%
% source ~/.venv3_12/.venv3_12/bin/activate
(.venv3_12) % python --version     # => 3.12.2
(.venv3_12) % which python-config  # => ~/.pyenv/shims/python-config
(.venv3_12) % python-config --prefix
   # => ~/.pyenv/versions/3.9.0  (!!)

As you see in the last line, the return value of python-config does not correspond to the currently venv-activated Python. This is because venv does not create its own python-config in its path (which should come at the head of the command-line search path), so the first python-config in your command-line search path is the one managed with pyenv; and as far as pyenv is concerned, the currently-activated version of Python is 3.9.0, as opposed to venv-managing 3.12.2.

To add, I am afraid there is no universally-applicable way for Python-venv to know how to create python-config. Remember pyenv-controlled python-config is, on its face value, just a very short shell script, and so simply copying it under a venv-controlled directory does not work. Of course, the Python core-developer team (including the venv maintainers) does not need to or perhaps does not care about pyenv, which is a mere third-party tool.

As such, I am afraid it is tricky for venv to provide an equivalent command to python-config.

Potential measures

Therefore, the remedy for this issue should be (at least for now) case-by-case basis. One of the OP's way of setting the environmental variable(s) CPATH or else appropriately is another. Or, depending on the configure script, you may symlink the correct include and lib files or directoies to/under /Your/Venv/Dir/{include,lib} . Or, a past answer on Stackoverflow "Why pyvenv does not install python-config?" provides one solution (I have not tested it personally).

Or, an alternatively way is to employ the conda framework to replace pyenv and venv altogether. In the conda framework, you can access the right python-config in your currently-activated Python environment. If you frequently need to compile user-modules of Python, this may be an easier solution.