Glue to Iceberg to Snowflake forcing TIMESTAMP_LTZ

41 Views Asked by At

I've got an AWS Glue 4.0 job that reads a Postgres table and writes it to an Iceberg table in an S3 bucket.

The Iceberg tables are in Snowflake as EXTERNAL_VOLUME tables pointed at the S3 Iceberg tables. So I can't modify the tables in Snowflake itself.

There is a column in the Postgres table that is type = timestamp.

The Glue job exports it to the Iceberg table as timestamptz.

{
  "id" : 9,
  "name" : "created_on",
  "required" : false,
  "type" : "timestamptz"
}

In Snowflake, timestamptz is interpreted as TIMESTAMP_LTZ. I need to this to be TIMESTAMP_NTZ. There is no timezone information in the dataframe that gets written to the Iceberg table.

I tried converting the timestamp with to_timestamp_ntz() but this is only supported as of pyspark 3.5. The Spark TimestampNTZType was introduced in 3.4. And AWS latest spark version is 3.3.

My goal is to get the value set as "type" : "timestamp" in the Iceberg table so that Snowflake will interpret it as TIMESTAMP_NTZ.

Is there any way to do this?

0

There are 0 best solutions below