Using Composer to Migrate Data from GCS to BigQuery

23 Views Asked by At

I am writing a script to merge 500 files in my gcs bucket and move the merged file to BigQuery. I am using composer to run the script. I created the dag, but I am getting the following error. I do not know what the error means.

[2024-02-19, 22:24:46 UTC] {subprocess.py:97} INFO - Command exited with return code -9 [2024-02-19, 22:24:50 UTC] {taskinstance.py:1853} ERROR - Task failed with exception Traceback (most recent call last): File "/opt/python3.8/lib/python3.8/site-packages/airflow/operators/bash.py", line 196, in execute raise AirflowException( airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code -9.

load_to_bq_task = BashOperator(
    task_id='load_to_bq_task',
    bash_command ='gsutil cp gs://merge-file-code/load_to_bq.py /tmp && python /tmp/load_to_bq.py',
    dag=dag,
)

Any suggestions are welcome 

0

There are 0 best solutions below