Azure Data Factory: How to import a complex json object from Avro file

Question

I am using ADF to parse a series of Avro files. The tricky part about this exercise is that the payload will contain a metrics array of varying sizes. And within each metric, there will be slightly different schemas.

This is what I have so far. I have flattened down the file so that I have this column.

So I will have an array of "payload metrics"....

..... and as you can see, there might be one or more levels of nesting at this level.

Long story short, what is the best way to proceed if I want to continue to break up this json node? Is there a way to dynamically break out the columns in the payload?

Answer 1

• In Flatten settings, click +Add mapping -> Rule based mapping

• Then in mapping, give the matching condition expression as true() and check theDeep column traversal checkbox.

This will dynamically flatten all the subcolumns of complex field.

Reference: https://learn.microsoft.com/en-us/azure/data-factory/data-flow-flatten#rule-based-mapping