Azure Synapse pipeline giving errors when triggered but runs fine if executed via Debug option

Question

I have a Azure Synapse pipeline which we migrated from DEV environment to our Pre Prod environment. When I execute it via the debug option, the pipeline executes fine. However if I trigger it via 'Trigger Now' or via setting up a trigger the job fails at a particular data flow. This particular data flow uses a cache sink; so I am unable to set 'verbose logging'.

The manual execution and the trigger use the same run time.

Following is the error message that I have been receiving:

{ "errorCode": "BadRequest", "message": "The request failed with status code '"BadRequest"'.", "failureType": "UserError", "target": "Dataflowname ", "details": "" }

Answer 1

I agree with @celaodar. Debug runs the pipeline you see and provides the output on screen. However, trigger runs can be performed only when publishing the pipeline.

I have encountered a similar kind of issue with datatype mismatch between source and sink.

I received "BadRequest" status code like below:

,"failureType":"UserError","target":"Data flow1","errorCode":"DFExecutorUserError"}

Check the data flow transformations to ensure that they are correct and not causing any data type mismatches or other errors.

I have made necessary changes for the data types for the columns, and the pipeline ran successfully.

Answer 2

Solved!!!

We posted this question to Microsoft and they looked at this and identified the issue. The Integration Run Time names across our DEV and Pre Prod had slight differences. The name of the IR, which was embedded in the pipelines definitions (which got migrated to Pre Prod), had a capital alphabet in DEV and Pre Prod had it in lower case. Once the names of the IR were rectified, the jobs were working fine when triggered.

MS advised that when you run it manually via the Debug option, the differences in the name of the IR doesnt matter. However when its triggered the IR names have to be consistently across all the environments, if the names are embedded in the pipeline definitions.