Our use case pyspark structural streaming will read csv from kafka & one column in csv will have json with nested fields. This nested json can be of different format for different records. so we can not define the json schema. We need to save it to elastic search with proper indexing (as json object) so that elastic query can be done.
Problem is from_json() needs one schema. As of now I am using MapType(StringType(), StringType()) & 1st level of json is getting parsed as object in elasticsearch & but the nested levels are are getting stored as string.
Is there any way to store nested json in elastic search as json object with out specifying the schema in from_json or if there is any other way to do it.
pyspark version : 3.2.1