Mass copy files in windows without windows explorer
Last summer I had a situation where I needed to migrate files away from a server that was at the end of it's life. The raid controller was about to give up and time was of the essence. As I had only given myself one night for the migration from the old system to the new, I wanted to make sure that I could succeed. As I was not certain that the restore of the backup would help me in time, I worked out an alternative solution which is what I want to discuss here. The idea was that betting on 2 horses is always better than one. In my case, this second horse really made my day and that is why I want to discuss it here. (for my future reference and in case it can help you!)
I have always know that copying files through file explorer in windows is being done sequentially (one file at a time) and that this process is really slow. (for example: if one file transfer is slow, the entire process becomes slow). That Is why there exist tools such as RoboCopy that allow you to speed up this process considerably. In this case I knew I had to use this tool, but because it can do so much more than "just copying faster", it comes with a wide range of flags and parameters that you can pass into. In that context, it is always a bit of a hassle to get started and that is why I took the time to write down what worked for me so that I can re-use it in the future and so that it is there for your reference!
This is the command that I used:
1robocopy <sourcelocation> <destinationlocation> /E /Z /ZB /R:5 /W:5 /TBD /NP /V /MT:64
Important to note that both sourelocation and destinationlocation should be the full path to the folders that are impacted. This can be both network and local folders.
RoboCopy has a lot of features that you can use, and in the command shown in this guide, we’re using the following options to make copy reliable and fast.
| flag | explanation |
|---|---|
| /E | Copy Subdirectories, including empty ones |
| /Z | Copy files in restartable mode |
| /ZB | Uses restartable mode, if access denied use backup mode |
| /R:5 | Retry 5 times (you can specify a different number, default is 1 million) |
| /W:5 | Wait 5 seconds before retrying (you can specify a different number, the default is 30 seconds) |
| /TBD | Wait for sharenames To Be Defined (retry error 67) -> don't really know what it means, but it seems useful! |
| /NP | No Progress – don’t display percentage copied -> together with /V you ensure that you have an overview of the "issues" and that you can focus on what is important |
| /V | Produce verbose output, showing skipped files |
| /MT:xx | Do multithreaded copies with n threads (default is 8) -> if your system is up to it, then you should consider putting this higher. The more you can do in parallel, the faster you will be out of the danger zone 😃 |
The big difference with copying the files through Windows explorer is that you can use the /MT flag to parallelize the process and thus to speed it up. It often pays of to take a bit of time up front (case per case) to see what works bet and to see "how far" you can go. (Do you go aggressive and put some load on the system? Or Do you prefer to approach it more cautious and to spare the system? That is a decision that you'll have to take...) In my case, the "waiting time" was reduced from 24 hours to only a few. Additionally, knowing that the server and perhaps also discs were acting funny, this really shortened the time that I was in a danger zone and that was in turn positive for my stress at that time!
Additionally, you can also look at some of the following flags, which I didn't use, but which seem interesting to me:
| flag | explanation |
|---|---|
| /J | copy using unbuffered I/O (recommended for large files) |
| /PURGE | delete dest files/dirs that no longer exist in source |
| /MIR | Mirror a directory tree (equivalent to /E plus /PURGE) |
| /MOV | Move files (delete from source after copying) |
| /MOVE | Move files AND dirs (delete from source after copying) |
And at last, this little gem offers a lot of other stuff, so when you are interested, please check it out by looking in the help by running
1robocopy /?
And even more info can be found on the Azure Docs site on RoboCopy
I hope it helps!
Tim