Airflow not show the overwrited values on config list command (EmailOperator dag case)

21 Views Asked by At

I was trying to setup the EmailOperator of Airflow to finish my pipeline with a notification. Following the various steps I found this problem: Aiflow is not actually showing the variables I passed through the environment key in docker. It seems to update it but there is no way to print (for check/verify) them if not by testing directly the task. Indeed, the airflow config list command is not usefull and retrive the default values. The problem is extended to the variable like AIRFLOW__CORE__EXECUTOR: CeleryExecutor that the default docker (created by the airflow team) is builded with. Is there any solution to this?

Given the airflow docker compose from the official api here

I tried to add few variables for a simple mail send:

  • AIRFLOW__SMTP__SMTP_HOST: smtp.gmail.com
  • AIRFLOW__SMTP__SMTP_USER: [email protected]
  • AIRFLOW__SMTP__SMTP_PASSWORD: xxx
  • AIRFLOW__SMTP__SMTP_PORT: 587
  • AIRFLOW__SMTP__SMTP_MAIL_FROM: [email protected]

I also changed the property AIRFLOW__CORE__LOAD_EXAMPLES to false to see a difference in the gui.

Here the code:

`x-airflow-common: &airflow-common
  image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.8.4}
  environment: &airflow-common-env
    AIRFLOW__CORE__EXECUTOR: CeleryExecutor
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
    AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
    AIRFLOW__CORE__FERNET_KEY: ""
    AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
    AIRFLOW__CORE__LOAD_EXAMPLES: "false"
    AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
    #  OTHER VARIABLE HERE
    AIRFLOW__SMTP__SMTP_HOST: smtp.gmail.com
    AIRFLOW__SMTP__SMTP_USER: [email protected]
    AIRFLOW__SMTP__SMTP_PASSWORD: xxx
    AIRFLOW__SMTP__SMTP_PORT: 587
    AIRFLOW__SMTP__SMTP_MAIL_FROM: [email protected]
    AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: "true"
    _PIP_ADDITIONAL_REQUIREMENTS: ${_PIP_ADDITIONAL_REQUIREMENTS:-}
  volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
  user: "${AIRFLOW_UID:-50000}:0"
  depends_on: &airflow-common-depends-on
    redis:
      condition: service_healthy
    postgres:
      condition: service_healthy`

After running the docker compose up --build Im trying to list the properties to check the result with the command: airflow config list The result is (always):

[core]
load_examples = True
[smtp]
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
\# smtp_user = 
\# smtp_password = 
smtp_port = 25
smtp_mail_from = [email protected]
smtp_timeout = 30
smtp_retry_limit = 5

The fact is that if go in the gui i actually dont see the examples, And i can send the mail too. So everithing works, but the aiflow config list or the airflow config get-value returns the old values. Why that? How can i actually check the properties?

0

There are 0 best solutions below