Pass runtine args to stored procedure Bigquery in google Workflows

168 Views Asked by At

I have a Google Cloud Platform workflow that invokes a stored procedure with specified parameters. The input for this operation is provided in the form of a JSON structure during runtime.

My objective is to dynamically substitute the parameters of the stored procedure routine with values extracted from the JSON input at runtime. This entails parsing the JSON input and mapping its values to the corresponding parameters in the stored procedure.

Below, you can see that I have attempted to incorporate query parameters. However, it doesn't work well for the first parameter of the array type; it only takes one string value from the ARRAY input "sku_code" and not the "axe_code" ...

main:
    params: [args]
    steps:
    - init_global:
        assign:
            - service_account: ${text.split(sys.get_env("GOOGLE_CLOUD_SERVICE_ACCOUNT_NAME"), "/")[3]}
    - test_technical_testing:
        call: googleapis.bigquery.v2.jobs.query
        args:
            projectId: ${args.params.project_name}
            body:
              query:
                  DECLARE granularite_columns ARRAY<STRING>;
                  DECLARE primary_key STRING;
                  DECLARE project_name STRING;
                  DECLARE dataset_name STRING;
                  DECLARE table_name STRING;
                  DECLARE result STRING;
                  CALL `oa-opsfin-demandvisibility-dv.dmndvsby_ds_c3_clean_eu_dv._quality_check_v3`(
                    @granularite_columns, @primary_key, @project_name, @dataset_name, @table_name
                    );
              useLegacySql: false
              queryParameters:
                  - name: "granularite_columns"
                    parameterType: { "type": "ARRAY", "arrayType": { "type": "STRING" } }
                    parameterValue: { "arrayValues": [{"value": "${args.params.granularite_columns}"}]}
                  - name: "primary_key"
                    parameterType: { "type": "STRING" }
                    parameterValue: { "value": "${args.params.primary_key}"}
                  - name: "project_name"
                    parameterType: { "type": "STRING" }
                    parameterValue: { "value": "${args.params.project_name}"}
                  - name: "dataset_name"
                    parameterType: { "type": "STRING" }
                    parameterValue: { "value": "${args.params.dataset_name}"}
                  - name: "table_name"
                    parameterType: { "type": "STRING" }
                    parameterValue: { "value": "${args.params.table_name}"}

json input runtime args:

{
  "params": {
    "dataset_name": "dmndvsby_ds_c3_clean_eu_dv",
    "granularite_columns": [
      "sku_code",
      "axe_code"
    ],
    "primary_key": "sku_code",
    "project_name": "oa-opsfin-demandvisibility-dv",
    "table_name": "t_sku_clean_v2"
  }
}

Thanks in advance for your help

0

There are 0 best solutions below