Exit loop condition when running the synpase notebooks based on metadata dependencies

Question

screenshot is showing what is running inside my Until activity in azure ADF. the lookup- Get parentNotebook is trying to check and see if there any parent notebooknames whose child have succeeded and retreive the name of it. the exit condition for loop is that check for any notebooknames with status of "New" in the pipelines_log table. Inside the tmpl_run_master(execute pipeline activity) i have a logic that if the child notebook fails automatically update the parentnotebook status to "Failed". Suppose if there is P1 which is dependent on D1,D2,D3 when D1 failes P1 is set to failed. But D2 and D3 will still have a status of "New" in the log table. so the loop keeps iterating and the input is blank from the stored proc as there are no more parent notebooks to execute. How to handle this scenario in my pipeline.

Answer 1

If I understand you want to exit the loop when any of the child notebook execution fails. Currently your exit loop condition is based on the status of "New" and for multiple child notebook this would not be sufficient.

Along with the current exit condition add another OR condition to check if any failure in the execution.

Declare a variable with a default value like

varIsFail = 0

Set a variable activity in the Failure flow of Execute pipeline activity to change this variable when execute pipeline activity fails.

varIsFail = 1

And add this varIsFail = 1 in your loop condition to exit the loop.

Hope this helps.