bruce at sqls.net
2014-Feb-10 14:38 UTC
Rsync performance with large exchange database files
I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to backup all the servers at my day job. This works pretty good overall but on one server it's not working so well :) We have an Exchange 2003 server with 4 separate mail store databases. One of them is roughly 900GB the others are ~200GB, ~160GB, and ~50GB. Rsync seems to spend a lot of time trying to find the differences in the files. On the Windows server where rsync is kicked off there's very little CPU or RAM usage for the rsync client. On the server rsync (rsyncd, no ssh) is using around 70-85% of a cpu (well, half a cpu due to hyper threading). I'm using VSS on the windows server to take a snapshot and expose it then running rsync from that to avoid locking issues. Is there anything I should check to help narrow down "problems?" or any settings I should try that could help speed things up any? Below is the final output of the last two rsync runs to give you an idea. It's taking 30-40+ hours to finish even though it's only transferring 80-160GB of change. Right now I'm testing this against a local rsync server so it should get pretty fast network performance. Eventually it will be moved to our off-site backup but that connection is still pretty fast (20 MBbit) and the backup is only hitting 800-1000 Kbytes/sec. Number of files: 19 Number of files transferred: 6 Total file size: 1265.74G bytes Total transferred file size: 1057.06G bytes Literal data: 160.67G bytes Matched data: 896.39G bytes File list size: 482 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 160.71G Total bytes received: 73.74M sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec total size is 1265.74G speedup is 7.87 [sender] _exit_cleanup(code=0, file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, line=1052): about to call exit(0) real 2833m1.324s user 2225m55.906s sys 45m10.015s Number of files: 11 Number of files transferred: 6 Total file size: 1268.78G bytes Total transferred file size: 1251.04G bytes Literal data: 83.43G bytes Matched data: 1167.61G bytes File list size: 216 File list generation time: 1.360 seconds File list transfer time: 0.000 seconds Total bytes sent: 83.48G Total bytes received: 87.25M sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec total size is 1268.78G speedup is 15.18 real 1745m5.647s user 1129m14.000s sys 39m58.875s Thanks (in advance) for the help :) -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20140210/90876c82/attachment.html>
bruce at sqls.net
2014-Feb-10 15:22 UTC
Rsync performance with large exchange database files
------ Original Message ------ From: bruce at sqls.net To: rsync at lists.samba.org Sent: 2/10/2014 8:38:06 AM Subject: Rsync performance with large exchange database files>I'm using a mixture of FreeBSD w/ ZFS+snapshots and rsync to backup all >the servers at my day job. This works pretty good overall but on one >server it's not working so well :) > >We have an Exchange 2003 server with 4 separate mail store databases. >One of them is roughly 900GB the others are ~200GB, ~160GB, and ~50GB. >Rsync seems to spend a lot of time trying to find the differences in >the files. On the Windows server where rsync is kicked off there's very >little CPU or RAM usage for the rsync client. On the server rsync >(rsyncd, no ssh) is using around 70-85% of a cpu (well, half a cpu due >to hyper threading). I'm using VSS on the windows server to take a >snapshot and expose it then running rsync from that to avoid locking >issues. > >Is there anything I should check to help narrow down "problems?" or any >settings I should try that could help speed things up any? > >Below is the final output of the last two rsync runs to give you an >idea. It's taking 30-40+ hours to finish even though it's only >transferring 80-160GB of change. Right now I'm testing this against a >local rsync server so it should get pretty fast network performance. >Eventually it will be moved to our off-site backup but that connection >is still pretty fast (20 MBbit) and the backup is only hitting 800-1000 >Kbytes/sec. > > >Number of files: 19 >Number of files transferred: 6 >Total file size: 1265.74G bytes >Total transferred file size: 1057.06G bytes >Literal data: 160.67G bytes >Matched data: 896.39G bytes >File list size: 482 >File list generation time: 0.001 seconds >File list transfer time: 0.000 seconds >Total bytes sent: 160.71G >Total bytes received: 73.74M > >sent 160.71G bytes received 73.74M bytes 991.84K bytes/sec >total size is 1265.74G speedup is 7.87 >[sender] _exit_cleanup(code=0, >file=/home/lapo/package/rsync-3.0.9-1/src/rsync-3.0.9/main.c, >line=1052): about to call exit(0) > >real 2833m1.324s >user 2225m55.906s >sys 45m10.015s > >Number of files: 11 >Number of files transferred: 6 >Total file size: 1268.78G bytes >Total transferred file size: 1251.04G bytes >Literal data: 83.43G bytes >Matched data: 1167.61G bytes >File list size: 216 >File list generation time: 1.360 seconds >File list transfer time: 0.000 seconds >Total bytes sent: 83.48G >Total bytes received: 87.25M > >sent 83.48G bytes received 87.25M bytes 836.85K bytes/sec >total size is 1268.78G speedup is 15.18 > >real 1745m5.647s >user 1129m14.000s >sys 39m58.875s > >Thanks (in advance) for the help :)rsync options I'm using on the client are : -rltihv --progress --stats --inplace --modify-window=1 On the windows client I'm using cygwin + rsync 3.0.9 but I'm going to test 3.1.0 there and see if there's a difference. On the server it's rsync 3.1.0 running rsyncd. Perhaps useful bit of information :). -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20140210/a2546d50/attachment.html>