How to make docker-compose ".env" file take precedence over shell env vars?

Question

I would like my docker-compose.yml file to use the ".env" file in the same directory as the "docker-compose.yml" file to set some envrionment variables and for those to take precedence for any other env vars set in the shell. Right now I have

$ echo $DB_USER
tommyboy

and in my .env file I have

$ cat .env
DB_NAME=directory_data
DB_USER=myuser
DB_PASS=mypass
DB_SERVICE=postgres
DB_PORT=5432

I have this in my docker-compose.yml file ...

version: '3'

services:

  postgres:
    image: postgres:10.5
    ports:
      - 5105:5432
    environment:
      POSTGRES_DB: directory_data
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: password

  web:
    restart: always
    build: ./web
    ports:           # to access the container from outside
      - "8000:8000"
    environment:
      DEBUG: 'true'
      SERVICE_CREDS_JSON_FILE: '/my-app/credentials.json'
      DB_SERVICE: host.docker.internal
      DB_NAME: directory_data
      DB_USER: ${DB_USER}
      DB_PASS: password
      DB_PORT: 5432
    command: /usr/local/bin/gunicorn directory.wsgi:application --reload -w 2 -b :8000
    volumes:
    - ./web/:/app
    depends_on:
      - postgres 

In my Python 3/Django 3 project, I have this in my application's settings.py file

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': os.environ['DB_NAME'],
        'USER': os.environ['DB_USER'],
        'PASSWORD': os.environ['DB_PASS'],
        'HOST': os.environ['DB_SERVICE'],
        'PORT': os.environ['DB_PORT']
    }
}

However when I run my project, using "docker-compose up", I see

maps-web-1       |   File "/usr/local/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
maps-web-1       |     connection = Database.connect(**conn_params)
maps-web-1       |   File "/usr/local/lib/python3.9/site-packages/psycopg2/__init__.py", line 127, in connect
maps-web-1       |     conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
maps-web-1       | psycopg2.OperationalError: FATAL:  role "tommyboy" does not exist

It seems like the Django container is using the shell's env var instead of what is passed in and I was wondering if there's a way to have the Python/Django container use the ".env" file at the root for it's env vars.

Answer 1

I thought at first I had misread your question, but I think my original comment was correct. As I mentioned earlier, it is common for your local shell environment to override things in a .env file; this allows you to override settings on the command line. In other words, if you have in your .env file:

DB_USER=tommyboy

And you want to override the value of DB_USER for a single docker-compose up invocation, you can run:

DB_USER=alice docker-compose up

That's why values in your local environment take precedence.

---

When using docker-compose with things that store persistent data -- like Postgres! -- you will occasionally see what seems to be weird behavior when working with environment variables that are used to configure the container. Consider this sequence of events:

1. We run docker-compose up for the first time, using the values in your .env file.

2. We confirm that we can connect to the database us the myuser user:

   $ docker-compose exec postgres psql -U myuser directory_data
   psql (10.5 (Debian 10.5-2.pgdg90+1))
   Type "help" for help.
   
   directory_data=#
   

3. We stop the container by typing CTRL-C.

4. We start the container with a new value for DB_USER in our environment variable:

   DB_USER=tommyboy docker-compose up
   

5. We try connecting using the tommyboy username...

   $ docker-compose exec postgres psql -U tommyboy directory_data
   psql: FATAL:  role "tommyboy" does not exist
   

   ...and it fails.

What's going on here?

The POSTGRES_* environment variables you use to configure the Postgres are only relevant if the database hasn't already been initialized. When you stop and restart a service with docker-compose, it doesn't create a new container; it just restarts the existing one.

That means that in the above sequence of events, the database was originally created with the myuser username, and starting it the second time when setting DB_USER in our environment didn't change anything.

The solution here is use the docker-compose down command, which deletes the containers...

docker-compose down

And then create a new one with the updated environment variable:

DB_USER=tommyboy docker-compose up

Now we can access the database as expected:

$ docker-compose exec postgres psql -U tommyboy directory_data
psql (10.5 (Debian 10.5-2.pgdg90+1))
Type "help" for help.

directory_data=#

Answer 2

I cannot provide a better answer than the excellent one provided by @larsks but please, let me try giving you some ideas.

As @larsks also pointed out, any shell environment variable will take precedence over those defined in your docker-compose .env file.

This fact is stated as well in the docker-compose documentation when taking about environment variables, emphasis mine:

You can set default values for environment variables using a .env file, which Compose automatically looks for in project directory (parent folder of your Compose file). Values set in the shell environment override those set in the .env file.

This mean that, for example, providing a shell variable like this:

DB_USER= tommyboy docker-compose up

will definitively overwrite any variable you could have defined in your .env file.

One possible solution to the problem is trying using the .env file directly, instead of the environment variables.

Searching for information about your problem I came across this great article.

Among other things, in addition to explaining your problem too, it mentions as a note at the end of the post an alternative approach based on the use of the django-environ package.

I was unaware of the library, but it seems it provides an alternative way for configuring your application reading your configuration directly from a configuration file:

import environ
import os

env = environ.Env(
    # set casting, default value
    DEBUG=(bool, False)
)

# Set the project base directory
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

# Take environment variables from .env file
environ.Env.read_env(os.path.join(BASE_DIR, '.env'))

# False if not in os.environ because of casting above
DEBUG = env('DEBUG')

# Raises Django's ImproperlyConfigured
# exception if SECRET_KEY not in os.environ
SECRET_KEY = env('SECRET_KEY')

# Parse database connection url strings
# like psql://user:[email protected]:8458/db
DATABASES = {
    # read os.environ['DATABASE_URL'] and raises
    # ImproperlyConfigured exception if not found
    #
    # The db() method is an alias for db_url().
    'default': env.db(),

    # read os.environ['SQLITE_URL']
    'extra': env.db_url(
        'SQLITE_URL',
        default='sqlite:////tmp/my-tmp-sqlite.db'
    )
}

#...

If required, it seems you could mix the variables defined in the environment as well.

Probably python-dotenv would allow you to follow a similar approach.

Of course, it is worth mentioning that if you decide to use this approach you need to make accesible the .env file to your docker-compose web service and associated container, perhaps mounting and additional volume or copying the .env file to the web directory you already mounted as volume.

You still need to cope with the PostgreSQL container configuration, but in a certain way it could help you achieve the objective you pointed out in your comment because you could use the same .env file (certainly, a duplicated one).

According to your comment as well, another possible solution could be using Docker secrets.

In a similar way as secrets works in Kubernetes, for example, as explained in the official documentation:

In terms of Docker Swarm services, a secret is a blob of data, such as a password, SSH private key, SSL certificate, or another piece of data that should not be transmitted over a network or stored unencrypted in a Dockerfile or in your application’s source code. You can use Docker secrets to centrally manage this data and securely transmit it to only those containers that need access to it. Secrets are encrypted during transit and at rest in a Docker swarm. A given secret is only accessible to those services which have been granted explicit access to it, and only while those service tasks are running.

In a nutshell, it provides a convenient way for storing sensitive data across Docker Swarm services.

It is important to understand that Docker secrets is only available when using Docker Swarm mode.

Docker Swarm is an orchestrator service offered by Docker, similar again to Kubernetes, with their differences of course.

Assuming you are running Docker in Swarm mode, you could deploy your compose services in a way similar to the following, based on the official docker-compose docker secrets example:

version: '3'

services:

  postgres:
    image: postgres:10.5
    ports:
      - 5105:5432
    environment:
      POSTGRES_DB: directory_data
      POSTGRES_USER: /run/secrets/db_user
      POSTGRES_PASSWORD: password
    secrets:
       - db_user
  web:
    restart: always
    build: ./web
    ports:           # to access the container from outside
      - "8000:8000"
    environment:
      DEBUG: 'true'
      SERVICE_CREDS_JSON_FILE: '/my-app/credentials.json'
      DB_SERVICE: host.docker.internal
      DB_NAME: directory_data
      DB_USER_FILE: /run/secrets/db_user
      DB_PASS: password
      DB_PORT: 5432
    command: /usr/local/bin/gunicorn directory.wsgi:application --reload -w 2 -b :8000
    volumes:
    - ./web/:/app
    depends_on:
      - postgres
    secrets:
       - db_user

secrets:
   db_user:
     external: true

Please, note the following.

We are defining a secret named db_user in a secrets section.

This secret could be based on a file or computed from standard in, for example:

echo "tommyboy" | docker secret create db_user -

The secret should be exposed to every container in which it is required.

In the case of Postgres, as explained in the section Docker secrets in the official Postgres docker image description, you can use Docker secrets to define the value of POSTGRES_INITDB_ARGS, POSTGRES_PASSWORD, POSTGRES_USER, and POSTGRES_DB: the name of the variable for the secret is the same as the normal ones with the suffix _FILE.

In our use case we defined:

POSTGRES_USER_FILE: /run/secrets/db_user

In the case of the Django container, this functionality is not supported out of the box but, due to the fact you can edit your settings.py as you need to, as suggested for example in this simple but great article you can use a helper function to read the required value in your settings.py file, something like:

import os

def get_secret(key, default):
    value = os.getenv(key, default)
    if os.path.isfile(value):
        with open(value) as f:
            return f.read()
    return value

DB_USER = get_secret("DB_USER_FILE", "")

# Use the value to configure your database connection parameters

Probably this would make more sense to store the database password, but it could be a valid solution for the database user as well.

Please, consider review this excellent article too.

Based on the fact that the problem seems to be caused by the change in your environment variables in the Django container one last thing you could try is the following.

The only requirement for your settings.py file is to declare different global variables with your configuration. But it didn't say nothing about how to read them: in fact, I exposed different approaches in the answer, and, after all, is Python and you can use the language to fill your needs.

In addition, it is important to understand that, unless in your Dockerfile you change any variables, when both the Postgres and Django containers are created the will receive exactly the same .env file with exactly the same configuration.

With these two things in mind you could try creating a Django container local copy of the provided environment in your settings-py file and use it between restarts or between whatever reason is causing the variables to change.

In your settings.py (please, forgive me for the simplicity of the code, I hope you get the idea):

import os
import ast

env_vars = ['DB_NAME', 'DB_USER', 'DB_PASS', 'DB_SERVICE', 'DB_PORT']

if not os.path.exists('/tmp/.env'):
    with open('/tmp/.env', 'w') as f:
        for env_var in env_vars:
            f.write(env_var)
            f.write('=')
            f.write(os.environ[env_var])
            f.write('\n')



with open('/tmp/.env') as f:
    cached_env_vars = f.read()
      
cached_env_vars_dict = ast.literal_eval(cached_env_vars)

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': cached_env_vars_dict['DB_NAME'],
        'USER': cached_env_vars_dict['DB_USER'],
        'PASSWORD': cached_env_vars_dict['DB_PASS'],
        'HOST': cached_env_vars_dict['DB_SERVICE'],
        'PORT': cached_env_vars_dict['DB_PORT']
    }

    #...
}

I think any of the aforementioned approches is better, but certainly it will ensure environment variables consistency accross changes in the environment and container restarts.

Answer 3

Values in the shell take precedence over those specified in the .env file.

If you set TAG to a different value in your shell, the substitution in image uses that instead:

export TAG=v2.0

docker compose convert

 
version: '3'
services:
  web:
    image: 'webapp:v1.5'

Please refer link for more details: https://docs.docker.com/compose/environment-variables/