How to write large data file (csv) to Azure Storage with encryption

85 Views Asked by At

I need to implement below in C# (background web job):

  • Call a stored procedure and get results (rows can be in millions - i.e. 100m so will be implementing pagination to return 500K at a time)
  • Generate a .CSV file
  • Zip it with encryption and store it in Azure Storage

As I need to deal with millions of records, I will need to implement batching - for instance, call the stored procedure, get 500K records, write the .CSV directly to Azure storage, and then fetch next batch, append the data in same file.

Here are my thoughts:

  • Create blob
  • Fetch batch, write to blob, repeat until all batches are written.
    Potential issue: I think I cannot append data to existing .CSV stored in Azure Storage. I cannot load existing file as it will be too big, I have to append the new data to an existing file without loading it in memory. Any suggestions here please.
  • Encrypt it, and zip it

Is this possible, any suggestions?

0

There are 0 best solutions below