I have Airflow 2.5.1 deployed on a Kubernetes Cluster with the official helm chart. The executor is KubernetesExecutor.
I want to write and read the airflow logs remotely on Stackdriver.
In my code I followed the steps described in the official airflow documentation so I modified the airflow helm chart adding the following env variables:
# Environment variables for all airflow containers
env:
- name: "AIRFLOW__LOGGING__REMOTE_LOGGING"
value: "True"
- name: "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER"
value: "stackdriver://airflow.task"
- name: "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID"
value: "google_cloud_default"
In Stackdriver I created the bucket "airflow.task".
In airflow, I have a dummy task like this:
from airflow.models import Variable
# import the logging module
import logging
# get the airflow.task logger
task_logger = logging.getLogger("airflow.task")
def print_hello_world(ds):
# with default airflow logging settings, DEBUG logs are ignored
task_logger.debug("This log is at the level of DEBUG")
# each of these lines produces a log statement
print("This log is created with a print statement")
task_logger.info("This log is informational")
task_logger.warning("This log is a warning")
task_logger.error("This log shows an error!")
task_logger.critical("This log shows a critical error!")
When I run the task I have two problems:
- Airflow's logs are written in the Stackdriver bucket "_Default" instead of "airflow.task"
- In the airflow UI and in Stackdriver I can see only build-in task logs, but not my custom logs (see example below).
[2023-04-18, 10:41:38 UTC] {standard_task_runner.py:55} INFO - Started process 36 to run task
[2023-04-18, 10:42:49 UTC] {local_task_job.py:276} WARNING - State of this instance has been externally set to success. Terminating instance.
[2023-04-18, 10:42:49 UTC] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 36. PIDs of all processes in the group: [36]
[2023-04-18, 10:42:49 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 36
Someone have any suggestion? I have check online for this problem, but I did not find much information.