rsync - Feb 2007 - rsync hangs, leaves defunct process

Hello,

I'm experiencing a weird problem which I haven't been able to track down
for 8+ weeks now.

Situation is as follows: There is a nigthly backup from our main server 
to a backup server. The server base directory is mounted there read-only 
via NFS. Then we do something like this:

1. Duplicate the contents of yesterday's backup into
    /sicherungen/2007/02/02/01.41/
2. Rsync the server directory against that with the following command:

rsync -va --delete --timeout=180 --no-detach /server_ro/ 
/sicherungen/2007/02/02/01.41/

[I've added the --timeout and --no-detach opitons for testing.]

Now, this usually works for a few days, then rsync hangs, leaving a 
defunct process:

root     30729 30695  0 01:55 ?        00:00:02 rsync -va --delete 
--timeout=180 --no-detach /server_ro/ /sicherungen/2007/02/02/01.41/
root     30730 30729  0 01:55 ?        00:00:00 [rsync] <defunct>

The last thing the log says is: "building file list ..."

Since the processes seem not to be killable, the only "remedy" is to 
reboot the backup- server. Then it will work for a few days, until it 
hangs again.

Installed are two SuSE 9.3 EE Linuxes, both fully updated. I'm not sure 
which kernel versions, but both sh?uld be 2.6.x I think. If that is 
important please tell me how I could provide you with that information.

BTW it's "rsync  version 2.6.3  protocol version 28"

I'm kind of lost here, and not exactly into Linux. I would appreciate 
any help.

Kind Regards,
David

On Fri 02 Feb 2007, David Mann wrote:
> 
> Situation is as follows: There is a nigthly backup from our main server 
> to a backup server. The server base directory is mounted there read-only 
> via NFS. Then we do something like this:
> 
> 1. Duplicate the contents of yesterday's backup into
>    /sicherungen/2007/02/02/01.41/
> 2. Rsync the server directory against that with the following command:
I'd let rsync do the duplication on the fly, using the --copy-dest
option.
> rsync -va --delete --timeout=180 --no-detach /server_ro/ 
> /sicherungen/2007/02/02/01.41/
> 
> [I've added the --timeout and --no-detach opitons for testing.]
> 
> Now, this usually works for a few days, then rsync hangs, leaving a 
> defunct process:
> 
> root     30729 30695  0 01:55 ?        00:00:02 rsync -va --delete 
> --timeout=180 --no-detach /server_ro/ /sicherungen/2007/02/02/01.41/
> root     30730 30729  0 01:55 ?        00:00:00 [rsync] <defunct>
> 
> The last thing the log says is: "building file list ..."
> 
> Since the processes seem not to be killable, the only "remedy" is
to
> reboot the backup- server. Then it will work for a few days, until it 
> hangs again.
Of course, you don't have to immediately reboot, unless the rsync
process is using up memory; simply leave the hanging process be.
Anyway, an unkillable process points to it hanging in some system call,
which in this case is probably a problem with the NFS. What are your NFS
mount options? I'd recommend using soft,intr.

Apart from that, rsyncing from an NFS filesystem is sub-optimal, as
rsync is designed to minimize network traffic at the expense of
filesystem IO. If the filesystem IO is going over the network, you're
losing.  Is there no way of running rsync on the server, instead of
having to go through the NFS mount?
> Installed are two SuSE 9.3 EE Linuxes, both fully updated. I'm not sure
> which kernel versions, but both sh?uld be 2.6.x I think. If that is 
> important please tell me how I could provide you with that information.
Kernel shouldn't matter that much, NFS kernel errors notwithstanding...
> BTW it's "rsync  version 2.6.3  protocol version 28"
That's pretty old... current is 2.6.9, with 3.0.0 pending.



Paul Slootman