I'm using ADF to copy data from on-premise to blob (machine telemetry data in the form of compressed files of ~10kB). Then I would like to ingest this data into Azure Data Explorer (Kusto). Because the data is already in blob, I thought I could use the Kusto Ingestion Rest API to just add the blobs to the ingestion queues.
My ADF pipeline is running successfully when I have a sequential for each operation, that uploads a single file to blob and then adds it to the queue (one by one). But if I try and do it in batches, only some of the files make it into ADX. I've rerun the pipeline multiple times and got the same result. I can't find anything using the show failed ingestions command and I'm struggling to find anything of meaning in the diagnostic logs.
- What would you suggest I can do to make sure all the blobs make it onto the queue and into ADX?
- Should I try and monitor the failed/success ingestion queue endpoints?
- Is there a way to add multiple blobs to the queue with one Rest API call as I suspect my issue is related to too many rest calls in a short period of time? I think the Rest API used to allow an array of blobs but I'm struggling to find in the documentation if this is still possible and how to implement it.
I'm not sure exactly what you do (REST API and so). ADF has Kusto connectors which ought to handle this for you.
If this is a onetime effort, you may consider doing this through Kusto ingest data wizard, https://dataexplorer.azure.com/oneclick/ingest?sourceType=file
Please mind the assumption is all your files are similarly structured and map well into the table scheme you created. If they are not you may run into errors trying to process the whole batch.