Write parquet file with local timestamp with avro schema

416 Views Asked by At

I'm trying to write local timestamps to a Parquet file with Java. As far as I understand, Avro supports local timestamps (https://avro.apache.org/docs/1.11.0/spec.html), which corresponds to Parquet's TIMESTAMP(isAdjustedToUTC=false, unit=MILLIS . My schema definition looks like:

LocalTimestampMillis datetimeType = LogicalTypes.localTimestampMillis();
field = new Schema.Field(attrDesc.getAttributeName(), Schema.createUnion(datetimeType.addToSchema(Schema.create(Schema.Type.LONG)), Schema.create(Schema.Type.NULL)), null, null);

Apache Drill and duckdb interpret the field as Bigint / int64 and do not show the timestamp but just the integer value.

Did I miss something in the field definition or is this an incompatibility between libraries?

I'm using avro-1.11.1, parquet-avro-1.12.3, hadoop-core-1.2.1 and parquet-hadoop-1.12.3.

0

There are 0 best solutions below