com.datastax.spark.connector.types.TypeConversionException after upgrading spark from 2.3.0 to 3.2.0

33 Views Asked by At

After upgrading spark 2.3.0 (scala 2.11) to spark 3.2.0 (scala 2.12), I am getting following runtime exception. Any properties to set for backward compatibility?

Stacktrace:

Caused by: com.datastax.spark.connector.types.TypeConversionException: Failed to convert column date_column_name of abc.tablename to java.sql.Date: 2023-06-01T00:00:00Z
    at com.datastax.spark.connector.mapper.GettableDataToMappedTypeConverter.tryConvert(GettableDataToMappedTypeConverter.scala:137)
    at com.datastax.spark.connector.mapper.GettableDataToMappedTypeConverter.convertedColumnValue(GettableDataToMappedTypeConverter.scala:160)
    at com.datastax.spark.connector.mapper.GettableDataToMappedTypeConverter.setterParamValue(GettableDataToMappedTypeConverter.scala:209)
    at com.datastax.spark.connector.mapper.GettableDataToMappedTypeConverter.$anonfun$invokeSetters$2(GettableDataToMappedTypeConverter.scala:249)
    at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:985)
    at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
    at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:984)
    at com.datastax.spark.connector.mapper.GettableDataToMappedTypeConverter.com$datastax$spark$connector$mapper$GettableDataToMappedTypeConverter$$invokeSetters(GettableDataToMappedTypeConverter.scala:248)
    at com.datastax.spark.connector.mapper.GettableDataToMappedTypeConverter$$anonfun$convertPF$1.applyOrElse(GettableDataToMappedTypeConverter.scala:262)
    at com.datastax.spark.connector.types.TypeConverter.convert(TypeConverter.scala:44)
    at com.datastax.spark.connector.types.TypeConverter.convert$(TypeConverter.scala:43)
    at com.datastax.spark.connector.mapper.GettableDataToMappedTypeConverter.convert(GettableDataToMappedTypeConverter.scala:19)
    at com.datastax.spark.connector.rdd.reader.ClassBasedRowReader.read(ClassBasedRowReader.scala:33)
    at com.datastax.spark.connector.rdd.CassandraJoinRDD.$anonfun$fetchIterator$5(CassandraJoinRDD.scala:146)
0

There are 0 best solutions below