How to assign a list of strings in airflow XCOM to a global variable X using xcom_pull?

82 Views Asked by At

I have KubernetesPodOperator that execute a python code in python project and returns a list of strings, i set Xcom_push = True, and i'm able to see the results pushed to airlfow/xcom/returns.json

Now, I'm trying to grab source_names values {'source_names': ['a', 'b', 'c', 'd', 'e']} from Xcom and assign it to global variable X. I tried

x = '{{ task_instance.xcom_pull("python_source_names")["source_names"] }}'

and

x = "{{ task_instance.xcom_pull('python_source_names')[\\"source_names\\"] }}"

But i got this error:

airflow.exceptions.AirflowException: The key (test-{) has to be made of alphanumeric characters, dashes, dots and underscores exclusively

Because the below code does expect a list of strings but it seems something else is being passed

datasets = x
ml_model = [
    KubernetesPodOperator(
        task_id=f"test-{dataset}",
        name=f"test-{dataset}",
        arguments=[
            "--action",
            "ML_MODEL",
            "--dataset",
            dataset,

        ],
        **ml_config,
    )
   for dataset in datasets
0

There are 0 best solutions below