Backup large number of files without enumerating

Question

Is there a way in .NET to backup a directory containing multiple sub directories each containing potentially 10,000 or more files of roughly 100kb-500kb in size without enumerating? The use case here is incrementally backing up files to USB storage and a NAS, but due to file count, it can take a really long time. I'm familiar with VSS and have created some custom backup applications utilizing it, but I was wondering if there is a way to snapshot a volume containing these files and save just the snapshot, without having to expose the snapshot as a mounted image and copy each file. End game is to shorten the amount of time the copy operation takes.

Answer 1

Windows Server Backup Feature might be a good alternative to custom built solutions. If you are running Windows Server 2016 you may find this article explaining the new features interesting too.

Answer 2

An MFT (master file table) record is only 1K or 1,024 bytes in size, so retrieving the name, attributes, etc. only needs a tiny bit of I/O in comparison to the reads required to move the file data itself. So enumerating over file directory records is not the problem, and eliminating that part of the processing won't cause more than a tiny flicker on your speedometer, so to speak.

I suppose you could try a low level BIOS "clone" (not copy) of the disk data itself, using cloning software. This may be a bit faster because the disk head won't have to zig zag around. On the other hand, this option will have nearly no effect if your hard disk is solid state, since random access will be as fast as sequential access.

Another option is to back up only files that have changed, by inspecting each file's last modified date time stamp. An easy way to do this is to use the XCOPY command with the /D switch (with no argument) which will compare the date/time of the last backup with the current file and only perform the copy if the file on your hard disk is newer than the file on the external drive.

Answer 3

You can call a different process from .Net and use RoboCopy to move the files across. It has lot of parameters like specifying number of threads to be used, or Timestamp checking etc.

public static void Main()
{
    Process myProcess = new Process();

    try
    {
        myProcess.StartInfo.UseShellExecute = false;
        myProcess.StartInfo.FileName = "C:\\Windows\\System32\\robocopy.exe Source Destination";
        myProcess.StartInfo.CreateNoWindow = true;
        myProcess.Start();
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
}