We have data in avro format in kafka. It's schema is getting registered in apicurio registry. When trying to consume this data via pyspark streaming, getting below error as we have multiple schema versions for a single batch . Malformed records are detected in record parsing. Current parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as 'PERMISSIVE'.
Approaches tried: We are deserialising data by iterating Dstream in avro format based on schema version that is present in apicurio registry for each batch. Limitations : We have to manage Dictionary containing all the schema versions and iterating each batch over this dictionary. Please suggest a work around which is more optimal than above approach.