Airflow DAG starts duplicate when execution is slow

Question

I'm having problems with some dags that are running duplicated. Example: I have a dag that runs every hour, generally it runs quickly for a maximum of 15 minutes, but it can happen that it runs for more than an hour and when this happens a new instance is executed even though the old one is still working.

Does anyone know if there is any parameter that can be placed in the dag so that this doesn't happen?

I searched in several places and didn't find anything like that, just something about catch-up

Answer 1

This is one of the Airflow features, where you can have multiple runs for the same dag running in parallel.

To control how many runs you want to have, you can set the dag parameter max_active_runs.

If you want to re-create the second run if the first dag run finishes after more than one hour (after the logical date of the second run), you need to set catchup to True, and if you want to skip creating it, you need to set it to False.