I am new to these table formats that allow time travel (Dela/iceberg/lance), but it's unclear to me how to manage dependencies across tables.
Suppose you have tables A, B, and C, where C builds on B which builds on A.
A -> B -> C
What are some good patterns to check what version of A generated C?
I could look at the timestamps of the versions, but you quickly end on checking that the timestamps be increasing in complicated ways.
I looked to see what is the minimum version of A that generates a monotonic increasing version timestamp up to C
Iceberg does provide time-travel by basically using the snapshots of the table as long as that snapshot exist in the system and data files related to these snapshots exist. There are basically two ways to do time travel.
FOR VERSION AS OFsyntax where you can pass thesnapshot_id. Snapshot ID is stored insnaphotsmetadata tableFOR TIMESTAMP AS OFsyntax where you can pass the timestamp at which how the data was.SELECT * FROM prod.db.table TIMESTAMP AS OF '1986-10-26 01:21:00';