I have a dataframe that contains avro bytes and the schema string. I would like to convert the avro bytes using the schema.
Here is my sample code to convert the avro using spark sql:
val rsDf = ss.sql("SELECT from_avro(avro_payload, avro_schema_string) FROM testDF")
but I got this error:
Caused by: org.apache.spark.sql.AnalysisException: Undefined function: 'from_avro'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7
at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$16.$anonfun$applyOrElse$121(Analyzer.scala:2084)
at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:53)
Any suggestion on how to call from_avro using spark-sql ?