Poetry Install Subprocess PoetryException

Question

I am facing the below issue with poetry install in Cloud composer. Failing at installation of URLObject. Looking for suggestions on resolving the same as my online search didnt get any results.

The shell script portion is

cat ${SPARK_HOME}/conf/spark-defaults.conf
export POETRY_HOME=/etc/poetry
export PATH="/etc/poetry/bin:$PATH"
curl -sSLfo poetry-install.py https://install.python-poetry.org \
    && python3 poetry-install.py --version 1.3.2
apt-get update && apt-get install -y pkgconf libhdf5-dev
cd $CODE_DIR/test && poetry config virtualenvs.in-project true && poetry install && source .venv/bin/activate
pip3 install shortuuid

I believe the line "cd $CODE_DIR/test && poetry config virtualenvs.in-project true && poetry install && source .venv/bin/activate" is causing the issue.

[2024-01-09, 12:24:50 UTC] {pod_manager.py:431} INFO - [base]   • Installing urlobject (2.4.3)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   • Installing websocket-client (0.59.0)
2024-01-09T12:24:55.208257615Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   CalledProcessError
2024-01-09T12:24:55.208333478Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   Command '['/tmp/backend/ingest/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/tmp/backend/ingest/.venv', '--no-deps', '/root/.cache/pypoetry/artifacts/23/1e/82/ffa13ae9fb790ca9aab416980cfab9da947b36f6fe14267ae27813c9f3/URLObject-2.4.3.tar.gz']' returned non-zero exit status 1.
2024-01-09T12:24:55.208628370Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   at /usr/lib/python3.8/subprocess.py:516 in run
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        512│             # We don't call process.wait() as .__exit__ does that for us.
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        513│             raise
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        514│         retcode = process.poll()
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        515│         if check and retcode:
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     →  516│             raise CalledProcessError(retcode, process.args,
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        517│                                      output=stdout, stderr=stderr)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        518│     return CompletedProcess(process.args, retcode, stdout, stderr)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        519│
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        520│
2024-01-09T12:24:55.385496695Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base] The following error occurred when trying to handle this error:
2024-01-09T12:24:55.385507722Z 

2024-01-09T12:24:55.385512832Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   EnvCommandError
2024-01-09T12:24:55.385524223Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   Command ['/tmp/backend/ingest/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/tmp/backend/ingest/.venv', '--no-deps', '/root/.cache/pypoetry/artifacts/23/1e/82/ffa13ae9fb790ca9aab416980cfab9da947b36f6fe14267ae27813c9f3/URLObject-2.4.3.tar.gz'] errored with the following return code 1, and output:
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   Processing /root/.cache/pypoetry/artifacts/23/1e/82/ffa13ae9fb790ca9aab416980cfab9da947b36f6fe14267ae27813c9f3/URLObject-2.4.3.tar.gz
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     Installing build dependencies: started
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     Installing build dependencies: finished with status 'error'
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     error: subprocess-exited-with-error
2024-01-09T12:24:55.385596508Z     

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     × pip subprocess to install build dependencies did not run successfully.
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     │ exit code: 1
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     ╰─> [15 lines of output]
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         Collecting setuptools>=40.8.0
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]           Downloading setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         Collecting wheel
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]           Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         Downloading setuptools-69.0.3-py3-none-any.whl (819 kB)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 819.5/819.5 kB 10.0 MB/s eta 0:00:00
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]             unknown package:
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]                 Expected sha256 177f9c9b0d45c47873b619f5b650346d632cdc35fb5e4d25058e09c9e581433d
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]                      Got        e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2024-01-09T12:24:55.386460841Z         

2024-01-09T12:24:55.386465642Z         

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         [notice] A new release of pip is available: 23.3 -> 23.3.2
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         [notice] To update, run: pip install --upgrade pip
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]         [end of output]
2024-01-09T12:24:55.386485896Z     

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     note: This error originates from a subprocess, and is likely not a problem with pip.
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   error: subprocess-exited-with-error
2024-01-09T12:24:55.386500760Z   

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   × pip subprocess to install build dependencies did not run successfully.
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   │ exit code: 1
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   ╰─> See above for output.
2024-01-09T12:24:55.386520313Z   

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   note: This error originates from a subprocess, and is likely not a problem with pip.
2024-01-09T12:24:55.386538557Z   

2024-01-09T12:24:55.386554330Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   at /etc/poetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1540 in _run
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1536│                 output = subprocess.check_output(
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1538│                 )
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1539│         except CalledProcessError as e:
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     → 1540│             raise EnvCommandError(e, input=input_)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1541│
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1542│         return decode(output)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1543│
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]       1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
2024-01-09T12:24:55.553353534Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base] The following error occurred when trying to handle this error:
2024-01-09T12:24:55.553448024Z 

2024-01-09T12:24:55.553459187Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   PoetryException
2024-01-09T12:24:55.553539955Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   Failed to install /root/.cache/pypoetry/artifacts/23/1e/82/ffa13ae9fb790ca9aab416980cfab9da947b36f6fe14267ae27813c9f3/URLObject-2.4.3.tar.gz
2024-01-09T12:24:55.554136296Z 

[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]   at /etc/poetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:58 in pip_install
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        54│
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        55│     try:
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        56│         return environment.run_pip(*args)
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]        57│     except EnvCommandError as e:
[2024-01-09, 12:24:55 UTC] {pod_manager.py:431} INFO - [base]     →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
[2024-01-09, 12:24:57 UTC] {pod_manager.py:431} INFO - [base]        59│
2024-01-09T12:24:55.560534096Z 

[2024-01-09, 12:25:00 UTC] {pod_manager.py:431} INFO - [base] Warning: The file chosen for install of s3transfer 0.8.1 (s3transfer-0.8.1-py3-none-any.whl) is yanked.
[2024-01-09, 12:25:00 UTC] {pod_manager.py:431} INFO - [base] Collecting shortuuid
[2024-01-09, 12:25:00 UTC] {pod_manager.py:431} INFO - [base]   Downloading shortuuid-1.0.11-py3-none-any.whl (10 kB)

Answer 1

You are correct with the line which causes the error. Specific it seems poetry install seems the faulty command.

From what I see the poetry.lock causes the issue. The problem seems that you used maybe used a different version of poetry before, but your cache libraries resulting from previous runs.

Following line (except it is a warning) give the right hints:

Warning: The file chosen for install of s3transfer 0.8.1 (s3transfer-0.8.1-py3-none-any.whl) is yanked.

This Feature was added in 1.2.0rc1

This might not be the solution for your issue, since it is difficult to reproduce. But in the past I had a similar issue and I needed to clean up my caches and recreated poetry.lock.

Keep in mind that recreation of poetry.lock could make diff this file more cumbersome.