Azure Function to be triggered when there is new data inserted in ADX

Question

I have a usecase where the data is inserted in ADX via the Event Hub. I want to track the changes in ADX i.e. when ever there is a new data inserted in ADX i want my function to be triggerd.

Also, whenever a new data is inserted in ADX, i want the organization name from the record inserted and pass it to my function. The data inserted from Event Hub to ADX will be in JSON format.

Could anyone help with an optimized approach on how i can achieve this?

I tried using event grid and logic app but i don't think they support direct integration with ADX currently. There is no data export available in ADX or in case of logic apps there is not Azure Data Explorer trigger currently available to track the changes in ADX.

Answer 1

There is no direct "ingest" trigger or something like that on ADX. You could, for example, create a second consumer group on the same Event Hub and have an Azure Function listen to that in addition to ADX. Its not a direct link, but if it's good enough for your scenario to assume that ADX will consume the data at the same time as the Function, build your trigger logic in the Function.

Answer 2

Adding to the comment that was mentioned by @silent, we can do this as a Function App. A combination of timer and ADX bindings is one way to attempt this flow.

To maintain the state of the timer, you may need to use a durable function. A barebone example of this without Durable function is available in the repo

Disclosure: We maintain this repo. Happy to help further if this fits your case.

Answer 3

It depends on the latency you are willing to live with.

One option I can think of is to use the ingestion_time() special function to discover new data. You can add a scheduled function (with what ever time you want, and once you discover new records (T | where ingestion_time() > last_check), you can send them to another function down stream, or just process it there. If you don't want to persist the last_check time, an easy way around that is to have an update_policy to another table with a small retention (say, 5 minutes). Again, it all depends on what latency is good for you.

Another option is using continues export, which would garauntee extactly once, and adding an event subscription to the target storage, and wire that to your function. This is much easier to manager, but can in theory miss the intervalBetweenRuns, so it will be less predictable.