I am trying to see how to pass values between tasks in Airflow.
There are multiple blogs and question in SO, but most of then illustrate this using PythonOperator, where ti.xcom_push and ti.xcom_push are used within the python callable.
How can we do this inside other operators where we do not have the option of a python callable?
I tried using what is mentioned as Jinja tempaltes, but I am seeing template render errors.
Example: This works: a.
'{{ ti.xcom_pull(task_ids="get_fusion_args") }}'
But this throws a template error: b.
'{{ ti.xcom_pull(task_ids="get_fusion_args", key="fusion_args") }}'
-> I am trying to get the value pushed by a previous task using the key "fusion_args".
Why is b. throwing a template error (unable to render template)
I have tried searching for answers including blogs and posts on StackOverflow. Could not find satisfactory answers.
Reiterating the questions:
- What ways are available (need some sample code) to pass values between Airflow tasks?
- What is wrong in the below code? Why is it throwing a jinja template error (cannot render template) '{{ ti.xcom_pull(task_ids="get_fusion_args",key="fusion_args) }}'
I found some answers to the questions:
Q1. What ways are available (need some sample code) to pass values between Airflow tasks? A. Link1 and Link2 mention how to pull xcom values within BashOperator (i.e., without using a python callable). The same can used with other operators.
Q2. Seems there is nothing wrong in the syntax of ti.xcom_pull(task_ids="get_fusion_args",key="fusion_args). What is wrong is that the pull returns a LazyXComAccess([1 items]) which is not getting rendered. I found this by printing the value in the earlier step which pushes the xcom. If the push has been a proper value, then the pull step works fine.