dbt external tables with dynamic time range

52 Views Asked by At

in my dbt project I now encounter that the external tables I work with can't handle the amount of data any more, because the number of files are just too much. In order to keep my external tables as small as possible I was wandering if somehow there is a way to dynamically query the last ten days of the external table?

The source of my tables are s3 buckets.

This is how I query the buckets now, which won't really work when the years are changing...

- name: NAME_EXTERNAL_TABLE_1
    external:
      location: "Path/to/s3/bucket"
      file_format: "(type = json strip_outer_array = true)"
      auto_refresh: true
      partitions:
      - name: metadata_filename
        expression: metadata$filename
        data_type: string
      - name: date_part
        expression: to_date(parse_date), 'yyyy-mm-dd')
        data_type: date    

Thank you so much for your help

0

There are 0 best solutions below