rclone slow transfer from bucket to filesystem

Question

Im using rclone to tranfer data between a minio bucket and a shared storage. Im migrating a store and The amount of data is around 200GB of product pictures. Every single picture have his own folder/path. So there are a lot of folders that needs to create to. Rclone is installed on the new server and the storage is connected to the server via san. The transfer is running over a week and we are at 170GB right now. Everything works fine but it is really slow in my opinion. Is it normal that a transfer out of a bucket into a classic filesystem is that slow?

Answer 1

(Doing the math, the speed is only 2.3Mbps. I am honestly not going to pay anything for that speed.)

Perhaps you should break down the issue and diagnose part by part. Below are several common places to look out for slow transfer (generally speaking for any file transfer):

First of all, network and file systems are usually not performant with lots of small files, so to isolate the issue, upload a bigger file to minio first (1GB+). And for each step, test with big file first.

1. Is the speed of the source fast enough? Try copying the files from minio to a local storage or Ramdisk (/tmp is usually tmpfs and in turn stored in RAM, use mount to check).
2. Is the speed of the destination fast enough? Try dd or other disk performance testing utility.
3. Is the network latency to source high? Try pinging or curling the API (with timing)
4. Is the network latency to destination high? Try iostat
5. Maybe the CPU is the bottleneck? As encoding and decoding stuff takes quite a lot of computing power. Try top when a copy is running.

Again, try these steps with the big file and fragmented file separately. The is quite a chance that fragmented files is an issue. If that is the case, I would try to look for concurrency option in rclone.

Answer 2

I had the same problem copying hundreds of thousands of small files from a S3-compatible storage to a local storage. Originally I was using s3fs+rsync. Very (very) slow, and it was getting stuck on the largest folders. Then I discovered rclone, and finished the migration within a few hours with these parameters:

rclone copy source:/bucket /destination/folder --checkers 256 --transfers 256 --fast-list --size-only --progress

Explanation of the options (from https://rclone.org/flags/)

--checkers 256 Number of checkers to run in parallel (default 8)

--transfers 256 Number of file transfers to run in parallel (default 4)

--fast-list Use recursive list if available; uses more memory but fewer transactions

--size-only Skip based on size only, not mod-time or checksum (wouldn't apply in your case if copying to an empty destination)

--progress Show progress during transfer