Iceberg metadata folder size growing in S3

Question

Table have small inserts but we are looking options to let not grow of metadata folder in S3 for Athena Iceberg table. Tried with config values ofoptimize_rewrite_delete_file_threshold = 1 and vacuum_max_snapshot_age_seconds=60 .

Running both OPTIMIZE & VACUUM commands but metadata folder size is not reducing.

Also, not able to set property write_max_metadata_files_to_keep to Athena table . But , AWS documentation says it will reduce the size of metadata folder but can't set this property to table and it is failing with error unsupported table property key

Answer 1

You need to expire_snapshots which will remove old data files as well as metadata files i.e. metadata.json, manifest_list.avro and manifest.avro.

CALL hive_prod.system.expire_snapshots('db.sample', TIMESTAMP '2021-06-30 00:00:00.000', 100);

Iceberg documentation for spark procedure

Also, you can consider merging smaller manifest file with rewrite_manifests