I always got the impression that rsync should be at least as fast as scp. But here I have a problem. When I use rsync -avr to copy data from the remote machine, I get something like 11 mb/s. If I use scp on the same file, I get 28 mb/s.
How come? It is a directiory with many files, but the largest one is a single 1 GB file. I mean, something is wrong with the connection, it should be 1 GBit and I only get ~300 mbit, but still, its weird that rsync is so much slower here...
Any suggestions? Thx.
43 Answers
One difference between scp and rsync is that rsync verifies transfered files whereas scp doesn't.
From the rsync manpage:
Note that rsync always verifies that each transferred file was correctly reconstructed on the receiving side by checking a whole-file checksum that is generated as the file is transferred, but that automatic after-the-transfer verification has nothing to do with this option’s before-the-transfer "Does this file need to be updated?" check.
So this could be one source for a performance difference.
1.. when it is transfering a large file it sticks around 11 mb/s.
without knowning, you could've caused a partial transfer when comparing transfer rates, making the CPU a bottleneck.
Rsync does a delta-transfer by default if it sees a partial file on the receiver (leftover from the scp test?). The "rolling checksum" method of comparing what needs to be transfered will take its toll in CPU usage on the Sender side.
I noticed a 4-fold decrease in transfer rate on a large partial binary file with a weakish CPU as Sender (Intel Atom or smaller VM cloud instances). On bigger machines the transfer rate stayed up after the initial partial file scan when continuing. The difference in throughput to plain netcat then should be the encryption via ssh.
Explicitly telling rsync to not to use partial transfers (-W|--whole-file) will put transfer rates in the same ballpark again. If needed, --append is another way to resume an interrupted file transfer without causing a delta-transfer. get -a in sftp can resume a download too.
See Rsyncs architecture description for further explanation.
You forgot about your HDD speed. SATA 2 or SATA 3? Max write speed for SATA 2 would be approx 300mb/s and for SATA 3 600mb/s, so let's say, the real speed you can get is 35-150mb/s. Only NVME drive would be able to handle 1gbps network speed, so you could get 600-800mb/s. :-)
P.S. Use rsync -a --progress /home...Zip process is killing your CPU, this way transfer will go up to max of your HDD write speed.
