Trigger Airflow DAG on MinIO file upload

Question

I would like to trigger an Apache Airflow DAG when a specific file is uploaded to a specific bucket in MinIO.

I've been looking into MinIO webhooks thinking that might be a solution, but I haven't quite figured it out.

I'm currently working locally, I have a Docker container running MinIO and others running Airflow.

Answer 1

There are no event-based triggers in the Airflow itself. You should either use external services to trigger Airflow API or just use sensors. In your case, you can use S3KeySensor

S3KeySensor(
    task_id='s3_file_check',
    poke_interval=60,
    timeout=180,
    soft_fail=False,
    bucket_key=s3_locat,
    bucket_name=s3_buckname,
    aws_conn_id='customer_demo',
    dag=dg
)