Dynamically parsing XML in Databricks

Question

a table in my database contains a column with large XML string data. The issue is that I want to dynamically pass a string to parse the XML.

This example shows more or less what kind of result I would like to achieve:

SELECT
  from_xml(
    '<fruits><fruit>banana</fruit><fruit>apple</fruit></fruits>',
    schema_of_xml(
      '<fruits><fruit>banana</fruit><fruit>apple</fruit></fruits>'
    )
  );

However, the function schema_of_xml only accepts a static value.

I decided to try to solve it using pyspark, below are the examples that works for individual rows:

and

There was also an attempt to write a UDF to use it with the "map" function. However, I've encountered some issues with utilizing RDDs: