I am running airflow in docker. I have an Extract task that extracts data from some API and stores it to a csv file in my docker container (path is /opt/airflow/raw/data.csv
I need to add a load task in my DAG to load the data from the csv file in docker container to a snowflake table. The code for my dag task is
from airflow import DAG
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator
load_task = SnowflakeOperator(
task_id='load_to_snowflake',
snowflake_conn_id='snowflake_conn',
sql=f"""
COPY INTO rates
FROM 'file:///opt/airflow/raw/data.csv'
FILE_FORMAT=(TYPE=CSV SKIP_HEADER=1)
""",
autocommit=True,
)
I get error that the url file:///opt/airflow/raw/data.csv is invalid. Please any ideas?