We're migrating our MLpipeline from Spark 2.4(scala 2.11.11) to Spark 3.3.0(scala 2.12.17) we were not able to read the existing MLModel with spark 3. This is because scala won’t support BC with major release upgrade. Seems this is not a Spark issue, As I was able to load the PipelineModel. The issue that I am facing now is we're storing a case class as binary object which is having Map datatype in scala 2.11.11 and trying to read that object file back with scala 2.12.17. While doing that I am getting the below exception,
java.io.InvalidClassException: scala.collection.immutable.Map$Map4; local class incompatible: stream classdesc serialVersionUID = -7746713615561179240, local class serialVersionUID = -7992135791595275193
Sample case class(just adding one column here): case class LearningModelOutput( transformerState: Map[String, Any])
The object that we're storing is like
0 = {Tuple2@8285} (targetColumn,)
1 = {Tuple2@8286} (consideredColumns,[Ljava.lang.String;@35b49c11)
2 = {Tuple2@8287} (schema,StructType(StructField(col1,IntegerType,true), StructField(col2,LongType,true
3 = {Tuple2@8288} (transformerStages,Map())
When trying to read this objectfile using spark.sparkcontext.objectfileMap[String, Any] getting the above mentioned serialization error.