Setting a custom statsd stat_name_handler in Airflow 2 (GCP Composer)

71 Views Asked by At

I'm trying to set a custom statsd stat_name_handler for Airflow 2.6.3 (part of GCP Composer 2) to prevent the DAGs from failing with airflow.exceptions.InvalidStatsNameException - e.g. due to the stat_name exceeding a length of 250 characters.

I haven't found much documentation about this topic, but from the Airflow source it seems the function need to be referenced with a dotted path.

What I tried

I tried to follow the Airflow module management guide and to create the folder structure under the plugins folder - which should be added to the PYTHONPATH by Airflow - with the following configuration:

Folder structure:

plugins/
└── my_company/
    ├── __init__.py
    └── custom_handlers.py

custom_handlers.py

def shorten_stat_name_handler(stat_name: str) -> str:
    # my logic
    return stat_name

configuration value: my_company.custom_handlers.shorten_stat_name_handler

Outcome

airflow-triggerer No module named 'my_company'

What is the correct way to pass a custom stat_name_handler?

0

There are 0 best solutions below