Accessing value of {{ts_nodash}} in airflow

Question

I need to access the value of {{ts_nodash}} to format it as a datetime and subtract an hour from it. But this gives me an error on dag saying: ValueError: time data '{{ ts_nodash }}' does not match format '%Y%m%dT%H%M%S' This is how my method looks like:

          ts_nodash = datetime.strptime("{{ ts_nodash }}", "%Y%m%dT%H%M%S")
          get_numbers = gke_wrapper.execute_gke_operator(
                task_id=f"get_numbers_{ctype_name}",
                labels=get_cost_labels(
                    pod=Pod.pod,
                    service="service",
                    id="mmoe",
                    environment="dev",
                    component="airflow",
                ),
                cmd=["sh"],
                args=[
                    "get_numbers.sh",
                    output_table_name,
                    dump_table_name,
                    model_path,
                    model_path,
                    ctype_name,
                    config.variants[ctype_name],
                    (ts_nodash - timedelta(hours=config.delay_interval)).strftime(
                        "%Y%m%dT%H%M%S"
                    ),
                ],
                nodepool=_NODE_POOL,
                image=_IMAGE,
                resource=_RESOURCES,
            )

Can someone mention how can I access the template variable value? I tried many different ways of passing {{ts_nodash}} within args=[] but apparently the datetime parsing operation occurs before the template value is made available. Any help will be much appreciated.

Answer 1

Templates evaluated during the Dag run (i.e. inside the execute method of the operator). Looks like you are trying to fetch ts_nodash outside the run time of the dag.

Below is an example of using templates:

op = BashOperator(
    task_id="pre-validation",
    bash_command="echo 'this is ts_nodash {{ ts_nodash }}'\n"
                 "echo 'this is dag_run.logical_date {{ dag_run.logical_date }}'",
    dag=dag,
)

So you will have to pass the template itself to the operator like below and it will be evaluated during the run time:

...
args = [
    "get_numbers.sh",
    output_table_name,
    dump_table_name,
    model_path,
    model_path,
    ctype_name,
    config.variants[ctype_name],
    "{{ ts_nodash }}"
],
...