DBT- Nested_Table Merge

136 Views Asked by At

I want to do a merge operation in DBT. my target table is nested. I wrote the code in 2 steps. In the 1st stage, I bring it to the level I want in my target table. In part 2, I merge it. but since I use expect in the select step, I see 6 joins running in the back. Actually, there should be one full join from the first temp and one merge needle. What can I use instead of Except?

Do you have any suggested solution?

Query:

{{config(
     materialized='incremental',
     unique_key='customer_id'
        )}}
WITH scd_target_array_tmp1 AS(

select ifnull(r.customer_id,l.customer_id) AS customer_id,
       ifnull(R.customerNO,l.customerNO) as customerNO,
       ARRAY_AGG(STRUCT(ifnull(l.name,r.name) as name ,ifnull(l.start_date,r.start_date) as start_date,
       case when r.customer_id is null then coalesce(l.end_date,current_date()) end as end_date,
       coalesce(r.is_current_record,'0')as is_current_record,ifnull(r.table_key,l.table_key) as table_key)) AS authors
from(
    select  p.customer_id,
            p.customerNO, b.name as name,
            b.start_date as start_date,
            b.end_date as end_date,
            b.is_current_record,
            b.table_key
    from presales-sandbox-346209.jaffle_shop.scd_target_array p, unnest(authors)b
    ) L
    full join(
    select customer_id,
           customerNO,Name,
           current_date() as start_date,
           cast(null as DATE) as end_date,
           '1'as is_current_record,
           to_hex(sha256(to_json_string(struct(customer_id,customerNO,Name)))) as table_key
    from presales-sandbox-346209.stripe.scd_table
     ) R on l.customer_id = R.customer_id and l.table_key=r.table_key
GROUP BY ifnull(r.customer_id,l.customer_id), ifnull(R.customerNO,l.customerNO)
)

SELECT customer_id,
       customerNO,
       authors
FROM scd_target_array_tmp1 a
{% if is_incremental() %}
WHERE customer_id IN (SELECT customer_id
                      FROM scd_target_array_tmp1 a , UNNEST(authors)b
                      WHERE  b. table_key in (select b.table_key from scd_target_array_tmp1 , unnest(authors) b
                                              except distinct
                                              select b.table_key from presales-sandbox-346209.jaffle_shop.scd_target_array, unnest(authors) b
                                              )
                      )
{% endif %}

I used expect in the select step in the last part of the query. What else can be?

0

There are 0 best solutions below