Azure copy activity

Question

Azure copy activity is taking so much time and want to reduce it. Basically I have around 200 folders and every day data is getting updated with the latest date folder. And I have to copy only few files from current date only. Can we reduce the scanning time instead of it to scan for all 200 folders it should only scan for current date ?

I tried filters with last modified but it is still scanning all files and only fetching current modified files. Taking same time

Answer 1

As you have date folders, just build the current date folder string in a variable and use wild card path in the copy activity like below.

Based on the date folders format, genarate the date string in ADF.

For sample, I took the date folders like below.

So, generate the current date string in the same format yyyy-MM-dd and store the string in a variable. You need to change the format as per your date folder.

@utcnow('yyyy-MM-dd')

In the source dataset, give the file path till the container inputdata and give this variable in the copy activity source wild card path like below.

Give target folder in the target dataset.

Debug the pipeline and it will copy all the files from the current date folder.