rsync - Apr 2018 - Much improved speeds of rsync via SSH - something to consider

Dear rsync devs,

I recently concluded a bug hunt to trace why my rsync-ing to an SBC was
much slower than the corresponding iperf3-reported speeds. To give a
concise summary of the situation, in slow wifi links using SSH with
ProxyCommand tremendously speeds up things:

    $ dd if=/dev/urandom bs=1M count=50 of=sample.data
    50+0 records in
    50+0 records out
    52428800 bytes (52 MB, 50 MiB) copied, 0.24135 s, 217 MB/s

    $ rsync -avz --progress ./sample.data root at 192.168.1.150:/dev/shm/
    sending incremental file list
    sample.data
         52,428,800 100%  575.23kB/s    0:01:28 (xfr#1, to-chk=0/1)

    sent 52,458,757 bytes  received 35 bytes  543,614.42 bytes/sec
    total size is 52,428,800  speedup is 1.00

    $ ssh root at 192.168.1.150 'rm /dev/shm/sample.data'

    $ rsync -avz --progress -e 'ssh -o "ProxyCommand nc %h
%p"'
./sample.data root at 192.168.1.150:/dev/shm/
    sending incremental file list
    sample.data
         52,428,800 100%    3.40MB/s    0:00:14 (xfr#1, to-chk=0/1)

    sent 52,458,757 bytes  received 35 bytes  2,690,194.46 bytes/sec
    total size is 52,428,800  speedup is 1.00

If it isn't clear - the speed of the upload went from 543 kbytes/sec to
2690 kbytes/sec.

If you want to see why - and how I traced this down - you can read my
complete report in the UNIX StackExchange forum. To avoid being classified
as a spammer, I won't include a link - search for question 434825 in the
search bar. The executive summary is that SSH disables Nagle's algorithm by
default - and in slow links this can cause tremendous impact (as you see
above).

Hope this helps people that backup over slow links (wi-fi or otherwise).

Cheers,
Thanassis Tsiodras, Dr.-Ing.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20180402/3418b8d8/attachment.html>

On Mon, 2 Apr 2018 15:05:14 +0200
Thanassis Tsiodras via rsync <rsync at lists.samba.org> wrote:
> I recently concluded a bug hunt to trace why my rsync-ing to an SBC
> was much slower than the corresponding iperf3-reported speeds. To
> give a concise summary of the situation, in slow wifi links using SSH
> with ProxyCommand tremendously speeds up things:
>     $ rsync -avz --progress -e 'ssh -o "ProxyCommand nc %h
%p"'
> ./sample.data root at 192.168.1.150:/dev/shm/
> If it isn't clear - the speed of the upload went from 543 kbytes/sec
> to 2690 kbytes/sec.
> 
> If you want to see why - and how I traced this down - you can read my
> complete report in the UNIX StackExchange forum. To avoid being
> classified as a spammer, I won't include a link - search for question
> 434825 in the search bar. The executive summary is that SSH disables
> Nagle's algorithm by default - and in slow links this can cause
> tremendous impact (as you see above).
> 
> Hope this helps people that backup over slow links (wi-fi or
> otherwise).
In the interest of collecting ssh-related issues in a single
thread I add this note:

The HPN patches to the portable OpenSSH implementation can
improve speed over high speed links with long round trip times.

  https://www.psc.edu/hpn-ssh

It also helps in those cases where CPU is a bottleneck and
encryption of transmitted data unnecessary.

The FAQ is also interesting.

The executive summary is:

  SCP and the underlying SSH2 protocol implementation in OpenSSH is
  network performance limited by statically defined internal flow
  control buffers.

Regards,

Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein