Register "from_avro" function in spark sql

101 Views Asked by At

I have a dataframe that contains avro bytes and the schema string. I would like to convert the avro bytes using the schema.

Here is my sample code to convert the avro using spark sql:

val rsDf = ss.sql("SELECT from_avro(avro_payload, avro_schema_string) FROM testDF")

but I got this error:

Caused by: org.apache.spark.sql.AnalysisException: Undefined function: 'from_avro'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 7
    at org.apache.spark.sql.catalyst.analysis.Analyzer$LookupFunctions$$anonfun$apply$16.$anonfun$applyOrElse$121(Analyzer.scala:2084)
    at org.apache.spark.sql.catalyst.analysis.package$.withPosition(package.scala:53)

Any suggestion on how to call from_avro using spark-sql ?

0

There are 0 best solutions below