I want to do a merge operation in DBT. my target table is nested. I wrote the code in 2 steps. In the 1st stage, I bring it to the level I want in my target table. In part 2, I merge it. but since I use expect in the select step, I see 6 joins running in the back. Actually, there should be one full join from the first temp and one merge needle. What can I use instead of Except?
Do you have any suggested solution?
Query:
{{config(
materialized='incremental',
unique_key='customer_id'
)}}
WITH scd_target_array_tmp1 AS(
select ifnull(r.customer_id,l.customer_id) AS customer_id,
ifnull(R.customerNO,l.customerNO) as customerNO,
ARRAY_AGG(STRUCT(ifnull(l.name,r.name) as name ,ifnull(l.start_date,r.start_date) as start_date,
case when r.customer_id is null then coalesce(l.end_date,current_date()) end as end_date,
coalesce(r.is_current_record,'0')as is_current_record,ifnull(r.table_key,l.table_key) as table_key)) AS authors
from(
select p.customer_id,
p.customerNO, b.name as name,
b.start_date as start_date,
b.end_date as end_date,
b.is_current_record,
b.table_key
from presales-sandbox-346209.jaffle_shop.scd_target_array p, unnest(authors)b
) L
full join(
select customer_id,
customerNO,Name,
current_date() as start_date,
cast(null as DATE) as end_date,
'1'as is_current_record,
to_hex(sha256(to_json_string(struct(customer_id,customerNO,Name)))) as table_key
from presales-sandbox-346209.stripe.scd_table
) R on l.customer_id = R.customer_id and l.table_key=r.table_key
GROUP BY ifnull(r.customer_id,l.customer_id), ifnull(R.customerNO,l.customerNO)
)
SELECT customer_id,
customerNO,
authors
FROM scd_target_array_tmp1 a
{% if is_incremental() %}
WHERE customer_id IN (SELECT customer_id
FROM scd_target_array_tmp1 a , UNNEST(authors)b
WHERE b. table_key in (select b.table_key from scd_target_array_tmp1 , unnest(authors) b
except distinct
select b.table_key from presales-sandbox-346209.jaffle_shop.scd_target_array, unnest(authors) b
)
)
{% endif %}
I used expect in the select step in the last part of the query. What else can be?