rsync - Apr 2009 - rsync fails with "Permission denied" errors on random files over NFS

Hi

I use rsync for daily backups from a mailserver.

Every day the mailserver pushes backup to the backupserver.
Cron runs the command:
rsync -q -zrlpt --specials --chmod=Du+rwx \
--rsh 'ssh -F /root/ssh/config -i /root/ssh/private.key -o
UserKnownHostsFile=/root/ssh/known_hosts'  \
--files-from=/root/include \
--exclude-from=/root/exclude \
--link-dest=../previous \
/ backupuser@backupserver:backup/current

The rsync command is run as the root user on the mailserver.

/root/include contains the line:
/data

/root/exclude is empty at this moment.


The problem is that almost every day, I get an e-mail from cron saying
that rsync has failed.
It gives errors like these:
rsync: readlink
"/data/mail/virtual/xx-net.dk/martin/cur/1240839853.V11I228039M884633.mail:2,RS"
failed: Permission denied (13)
rsync: readlink
"/data/mail/virtual/xx-net.dk/martin/cur/1240830282.V11I228031M150189.mail:2,S"
failed: Permission denied (13)
rsync: readlink
"/data/mail/virtual/xx-net.dk/martin/cur/1240830490.V11I228036M432719.mail:2,S"
failed: Permission denied (13)

I have checked that the files are ineed readable on the mailserver.
Here is ls -l from one of them:
-rw------- 1 mail mail 560383 2009-04-27 13:08
/data/mail/virtual/xx-net.dk/martin/cur/1240830490.V11I228036M432719.mail:2,S


The files that gives the error is appearently changing randomly. One
day there can be one or two files with an error. Other times several
hundred.
I have seen it give errors on files in mail boxes that is currently
unused, so it cannot be because the file is currently used by some
other application (e.g. the imap server).

I noticed that it is readlink that gives the error. I'm not sure why
readlink is used in this case. This is a regular file and not a
symlink.


I hope you have some suggestions on how to debug this further.


Details on the servers's file systems:

Server backupserver:
/data is mounted like this:
nfs:/data/nfs/nfs0/data/xx/backup on /data type nfs
(rw,timeo=60,intr,rsize=16384,wsize=16384,addr=78.xx.xx.xx)


Server mailserver:
/data is mounted like this:
nfs:/data/nfs/nfs0/data/xx/mail on /data type nfs
(rw,timeo=60,intr,rsize=16384,wsize=16384,addr=78.xx.xx.xx)


Med venlig hilsen/Kind regards
Thomas Damgaard Nielsen
http://thomasdamgaard.dk

Bump.

I have still not found a solution for this. Do you have any ideas on
how to debug it?

Thanks.

Med venlig hilsen/Kind regards
Thomas Damgaard Nielsen
http://thomasdamgaard.dk



On Tue, Apr 28, 2009 at 2:35 PM, Thomas Damgaard<thomasdn@gmail.com>
wrote:
> Hi
>
> I use rsync for daily backups from a mailserver.
>
> Every day the mailserver pushes backup to the backupserver.
> Cron runs the command:
> rsync -q -zrlpt --specials --chmod=Du+rwx \
> --rsh 'ssh -F /root/ssh/config -i /root/ssh/private.key -o
> UserKnownHostsFile=/root/ssh/known_hosts' ?\
> --files-from=/root/include \
> --exclude-from=/root/exclude \
> --link-dest=../previous \
> / backupuser@backupserver:backup/current
>
> The rsync command is run as the root user on the mailserver.
>
> /root/include contains the line:
> /data
>
> /root/exclude is empty at this moment.
>
>
> The problem is that almost every day, I get an e-mail from cron saying
> that rsync has failed.
> It gives errors like these:
> rsync: readlink
>
"/data/mail/virtual/xx-net.dk/martin/cur/1240839853.V11I228039M884633.mail:2,RS"
> failed: Permission denied (13)
> rsync: readlink
>
"/data/mail/virtual/xx-net.dk/martin/cur/1240830282.V11I228031M150189.mail:2,S"
> failed: Permission denied (13)
> rsync: readlink
>
"/data/mail/virtual/xx-net.dk/martin/cur/1240830490.V11I228036M432719.mail:2,S"
> failed: Permission denied (13)
>
> I have checked that the files are ineed readable on the mailserver.
> Here is ls -l from one of them:
> -rw------- 1 mail mail 560383 2009-04-27 13:08
>
/data/mail/virtual/xx-net.dk/martin/cur/1240830490.V11I228036M432719.mail:2,S
>
>
> The files that gives the error is appearently changing randomly. One
> day there can be one or two files with an error. Other times several
> hundred.
> I have seen it give errors on files in mail boxes that is currently
> unused, so it cannot be because the file is currently used by some
> other application (e.g. the imap server).
>
> I noticed that it is readlink that gives the error. I'm not sure why
> readlink is used in this case. This is a regular file and not a
> symlink.
>
>
> I hope you have some suggestions on how to debug this further.
>
>
> Details on the servers's file systems:
>
> Server backupserver:
> /data is mounted like this:
> nfs:/data/nfs/nfs0/data/xx/backup on /data type nfs
> (rw,timeo=60,intr,rsize=16384,wsize=16384,addr=78.xx.xx.xx)
>
>
> Server mailserver:
> /data is mounted like this:
> nfs:/data/nfs/nfs0/data/xx/mail on /data type nfs
> (rw,timeo=60,intr,rsize=16384,wsize=16384,addr=78.xx.xx.xx)
>
>
> Med venlig hilsen/Kind regards
> Thomas Damgaard Nielsen
> http://thomasdamgaard.dk
>