rsync - Oct 2003 - Speeding up Transfers of 1000s of files

Hi all,

I have to transfer thousands of files across my network. We are looking at using
rsync via a squid proxy to help improve the network tuning. What I am witnessing
now is that with each successive file, the transfer speed resets to 0 and ramps
up to 2.5MB/s.

Is there anyway that I can group these transfers together to take better
advantage of our bandwidth? I should be able to peak at 5+MB/s.

Thanks in advance,
Max

--
Max Clark <maxc at beast.clarksys.com>
My Blog http://www.clarksys.com
>> spamtrap: spam@clarksys.com - do NOT ever send email to this address
<<

Rsync works in a very linear fashion. Try running multiple rsyncs in paralell
on different directories of course.

One speedup would be to transfer the checksums and data asyncronously. I
havent checked the source on how this is done but I suspect that even this is
done in a very linear way wich explains why data transfers have to wait for
checksum calculation, transfer and compare.

Hans Eric

----- Original Message ----- 
From: "Max Clark" <maxc-rsync@beast.clarksys.com>
To: <rsync@lists.samba.org>
Sent: Monday, October 27, 2003 7:02 PM
Subject: Speeding up Transfers of 1000s of files


Hi all,

I have to transfer thousands of files across my network. We are looking at
using rsync via a squid proxy to help improve the network tuning. What I am
witnessing now is that with each successive file, the transfer speed resets to
0 and ramps up to 2.5MB/s.

Is there anyway that I can group these transfers together to take better
advantage of our bandwidth? I should be able to peak at 5+MB/s.

Thanks in advance,
Max

--
Max Clark <maxc at beast.clarksys.com>
My Blog http://www.clarksys.com
>> spamtrap: spam@clarksys.com - do NOT ever send email to this address
<<
-- 
To unsubscribe or change options:
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html