Running a dbt snapshot model for the first time works, but on the second run, it executes indefinitely

239 Views Asked by At

I'm encountering an issue while running a snapshot on a table with over 1,000,000 rows, following the schema detailed here:

Table Schema.

The initial run of the snapshot generates the table correctly. However, on the second execution, where the snapshot is expected to compare the data to generate updates, the execution never completes.

The table I'm transforming has a composite primary key consisting of 3 columns: 'document', 'change', and 'proposal'. I'm using these columns for the unique_key check, and check_cols to validate specific columns. Despite trying various approaches, I haven't found a solution yet. I'm unsure if the issue lies in the snapshot code or the dbt configuration.

Below is the code snippet for the snapshot:

{% snapshot snapshot_quiver_documentos %}

{{
    config(
      target_schema='data_lake_snapshot',
      unique_key='id_documento',
      strategy='check',
      check_cols=['grupo_hierarquico', 'cliente', 'documento', 'alteracao', 'calculo', 'proposta', 'proposta_cia', 'apolice', 'endosso', 'seguradora', 'produto', 'tipo_documento', 'tipo_endosso', 'nome_tipo_negocio', 'situacao', 'premio_liqdesc', 'adicional', 'premio', 'perc_comissao', 'perc_cocorret', 'data_proposta', 'data_inclusao', 'data_emissao', 'inicio_vigencia', 'termino_vigencia', 'data_entrada', 'data_cancel', 'tipo_emis', 'usuario', 'usuario2', 'premio_liqprop', 'premio_total' ],
      file_format='delta'
    )
}}

SELECT tabela_documentos.*,
       {{ dbt_utils.generate_surrogate_key(['documento', 'alteracao', 'proposta']) }} AS id_documento
FROM {{ source('source_silver', 'fato_documentos_producao')}} tabela_documentos

{% endsnapshot %}

I've tried creating another column for a unique key and also attempted avoiding the use of an asterisk in the SELECT statement, specifying the names of all columns, but the issue persists. The snapshot needs to process correctly for tracking changes in this data source and creating a fact.

0

There are 0 best solutions below