Docker-compose: ModuleNotFoundError: No module named 'core'

Question

I use fastapi, docker, docker-compose, kafka. I try to up the modules-consumer:

docker-compose up consumer-modules

But I received ModuleNotFoundError:

ModuleNotFoundError: No module named 'core'

Import in modules_consumer.py:

from core import settings

where core - project folder, not library.

Dockerfile:

ENV PYTHONUNBUFFERED 1

COPY . /app
WORKDIR /app
RUN rm -rf .venv/
RUN apt-get update && \
    apt-get install -y locales \
    python-dev-is-python3 libldap2-dev libsasl2-dev libssl-dev \
    libmagic1 && \
    echo ru_RU.UTF-8 UTF-8 >> /etc/locale.gen && \
    locale-gen && \
    python -m pip install --upgrade pip && \
    python -m pip install poetry
RUN poetry config virtualenvs.create false && poetry install --no-root --no-dev

EXPOSE 80

Consumer-modules in docker-compose.yaml:

consumer-modules:
    build:
      context: ../app
    volumes:
      - ../app:/app
    depends_on:
      - db
    command: sh -c "python3 consumers/modules_consumer.py"

Project structure:

app/
├─ consumers/
│  ├─ modules_consumer.py
│  ├─ __init__.py
├─ core/
│  ├─ settings.py
│  ├─ __init__.py
├─ Dockerfile
├─ docker-compose.yaml

I found issues with a similar problem on stackoverflow but they did not help me.

What could be the problem?

Answer 1

This is a path issue. It's best resolved by modifying the $PYTHONPATH environment variable, which tells the Python process where to find files it requires.

The problem

You have explained that the directory structure is

/app
├─ core/
│  ├─ settings.py
├─ consumers/
│  ├─ modules_consumer.py

When you run docker compose up, the consumer-modules service is configured to run the command python3 consumers/modules_consumer.py. This runs inside the container in /app.

Now, what happens is this: the code in modules_consumer.py executes from the subdirectory in which the file is located. If it were located at /app/modules_consumer.py, the code it contains - from core import settings - would run just fine. The Python process would look in all the immediate subdirectories (core/ and consumers/) for settings.py and would find it in core/.

By contrast, because modules_consumer.py is in consumers/, this cannot happen (without help). After failing to find a file called core.py in consumers/, the Python process is going to look for any subdirectories in consumers/ and won't find any.

The solution

The way to resolve this is to add /app to the $PYTHONPATH environment variable. This way, the Python process will know that - as well as the current directory - it should look for imports in /app.

I think in your case the simplest thing to do is to overwrite this variable in your docker-compose.yml:

consumer-modules:
    build:
      context: ../app
    volumes:
      - ../app:/app
    depends_on:
      - db
    environment:
      - PYTHONPATH=/app
    command: sh -c "python3 consumers/modules_consumer.py"

(if it turns out to be more complicated, e.g. because this overwrites some things that are needed and already in the $PYTHONPATH by default, there are many alternative ways to set and/or extend this environment variable)