How to handle duckdb IOexception while reading multiple files

35 Views Asked by At

I have list of numbers generated from a function. Using that, I am trying to form a s3 bucket path and query parquet files. The numbers act as folder names in s3, but not all the folders will be present what gets generated by that function.

while running the below, i am getting IOexception (404 not found, some folder is not present)

folder_names = ["s3:/bucket/123", "s3:/bucket/456", "s3:/bucket/789"]   # just an example
duckdb.sql(f"""SELECT * FROM 
    read_parquet({folder_names});
    """)

How to handle such scenarios?

0

There are 0 best solutions below