in my dbt project I now encounter that the external tables I work with can't handle the amount of data any more, because the number of files are just too much. In order to keep my external tables as small as possible I was wandering if somehow there is a way to dynamically query the last ten days of the external table?
The source of my tables are s3 buckets.
This is how I query the buckets now, which won't really work when the years are changing...
- name: NAME_EXTERNAL_TABLE_1
external:
location: "Path/to/s3/bucket"
file_format: "(type = json strip_outer_array = true)"
auto_refresh: true
partitions:
- name: metadata_filename
expression: metadata$filename
data_type: string
- name: date_part
expression: to_date(parse_date), 'yyyy-mm-dd')
data_type: date
Thank you so much for your help