rsync - Aug 2007 - rsync out of memory at 8 MB although ulimit is 512MB

Hello again,

I encountered something amazing. First I thought there is not
enough memory allowed through ulimit. ulimit is now set to 
(almost) 512MB but rsync still gets out fo memory at 8MB. 

Can anyone tell me why?

That's my configuration:

  rsync version 2.6.2
  from AIX 5.3 to SuSE Linux 9 (also has rsync 2.6.2)
  ulimit -a (AIX)

ulimit -a AIX (source):
-------------------------
time(seconds)        unlimited
file(blocks)         2097151
data(kbytes)         131072
stack(kbytes)        512000
memory(kbytes)       512000
coredump(blocks)     2097151
nofiles(descriptors) unlimited

ulimit -a Linux (target):
-------------------------
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 137216
virtual memory        (kbytes, -v) unlimited

rsync output:
------------------------------
$ rsync -a -v -v --rsh=ssh /thumbnail/0009/
<user>@<server>:/data/thumbnail/0009/
opening connection using ssh <server> -l <user> rsync --server
-vvlogDtpr . /data/thumbnail/0009/
building file list ...
[sender] expand file_list to 131072 bytes, did move
[sender] expand file_list to 262144 bytes, did move
[sender] expand file_list to 524288 bytes, did move
[sender] expand file_list to 1048576 bytes, did move
[sender] expand file_list to 2097152 bytes, did move
[sender] expand file_list to 4194304 bytes, did move
[sender] expand file_list to 8388608 bytes, did move
ERROR: out of memory in receive_file_entry
rsync error: error allocating core memory buffers (code 22) at util.c(116)

Kind regards,
  Marco
-- 
Marco Jacob, PMP
Prisma GmbH, Germany

On 8/27/07, mjacob@union06.de <mjacob@union06.de>
wrote:
> I encountered something amazing. First I thought there is not
> enough memory allowed through ulimit. ulimit is now set to
> (almost) 512MB but rsync still gets out fo memory at 8MB.
> [sender] expand file_list to 131072 bytes, did move
> [sender] expand file_list to 262144 bytes, did move
> [sender] expand file_list to 524288 bytes, did move
> [sender] expand file_list to 1048576 bytes, did move
> [sender] expand file_list to 2097152 bytes, did move
> [sender] expand file_list to 4194304 bytes, did move
> [sender] expand file_list to 8388608 bytes, did move
> ERROR: out of memory in receive_file_entry
Rsync's output is a bit misleading.  The 8 MB is the size of the array
of *pointers* to the file-list entries.  It's very plausible that the
allocated entries themselves could take up 512 MB while the array of
pointers takes up only 8 MB.  You need to either raise the memory
limit even further, break the rsync run up into smaller ones, or use
the incremental recursion mode in rsync 3.0.0 (which keeps only part
of the file list in memory at a time).

Matt

Wayne,

The attached patch to the current CVS rsync corrects the misleading message.

Matt
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix-flist_expand-misleadingness.diff
Type: text/x-patch
Size: 631 bytes
Desc: not available
Url :
http://lists.samba.org/archive/rsync/attachments/20070829/a5f8e691/fix-flist_expand-misleadingness.bin

Thanks, Matt, this is a really good answer to my question.
I already wrote a script recursing into directories only
doing smaller parts because I needed a fast solution.

However this needs a little more work because some
inodes in the tree can be removed or changed so
at the target directory there might be some directories
files left. So far this is no problem to me/us.

Unfortunately new versions are not build for AIX
by IBM and our company rely on them. So this will
last a long time ...

Maybe I can argue them into build it ourself. :-)

Kind regards,
  Marco
--
Marco Jacob, PMP
Prisma GmbH, Germany
>On 8/27/07, mjacob@union06.de <mjacob@union06.de> wrote:
>> I encountered something amazing. First I thought there is not
>> enough memory allowed through ulimit. ulimit is now set to
>> (almost) 512MB but rsync still gets out fo memory at 8MB.
>
>> [sender] expand file_list to 131072 bytes, did move
>> [sender] expand file_list to 262144 bytes, did move
>> [sender] expand file_list to 524288 bytes, did move
>> [sender] expand file_list to 1048576 bytes, did move
>> [sender] expand file_list to 2097152 bytes, did move
>> [sender] expand file_list to 4194304 bytes, did move
>> [sender] expand file_list to 8388608 bytes, did move
>> ERROR: out of memory in receive_file_entry
>
>Rsync's output is a bit misleading.  The 8 MB is the size of the array
>of *pointers* to the file-list entries.  It's very plausible that the
>allocated entries themselves could take up 512 MB while the array of
>pointers takes up only 8 MB.  You need to either raise the memory
>limit even further, break the rsync run up into smaller ones, or use
>the incremental recursion mode in rsync 3.0.0 (which keeps only part
>of the file list in memory at a time).
>
>Matt