out of memory in flist_expand

[Rob klein Gunnewiek]

Hello list,

I get the following error using rsync 3.0.3 (both sides):

...
data/xxxxxxxx/admin_docs/2/8/9/9/0/7/9/msg-75682-748.msg is uptodate
data/xxxxxxxx/admin_docs/2/8/9/9/0/8/0/msg-75682-749.txt is uptodate
ERROR: out of memory in flist_expand [generator]
rsync error: error allocating core memory buffers (code 22) at util.c(117)
[generator=3.0.3]
rsync error: received SIGUSR1 (code 19) at main.c(1286) [receiver=3.0.3]

Command was: rsync -avvz remotehost:/var/www /backup

These things I have tried to fix the problem:

After starting rsync, it says 'receiving incremental file list', which I read
indicates it won't consume too much memory.

I have increased memory by adding an extra swap space of 4GB (I know,
overdone..). There is 1GB of RAM in the system. There is plenty of diskspace as
well. I have also tried modifying the maximum stack size, which is now 512 MB
for a program.

In the Rsync FAQ I read Rsync uses approximately 100 bytes per file for
indexing. The number of files to copy from the remote server is 3902364, so
rsync would need about 400MB... right?

Both sides run FreeBSD 6.x, and rsync 3.0.3.

What am I missing?

--
Rob klein Gunnewiek
BWSS B.V. Deventer (http://www.bwss.nl)
Tel +(31) 0570-665140
Fax +(31) 0570-665141
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

[Rob klein Gunnewiek]

Could someone please help me with this problem?

[Wayne Davison]

On Fri, Aug 15, 2008 at 05:28:43PM +0200, Rob klein Gunnewiek wrote:
> ERROR: out of memory in flist_expand [generator]

With -vv, you should be seeing messages like this:

[generator] expand file_list pointer array to N bytes, did move

That will tell you what the last realloc attempt was before the error.
If it is a really huge value, check to see if you have a directory with
a huge number of files in it. You should also check your process's
memory ulimit values to see if they are too small, and try using a
different compiler or malloc package to see if your malloc is having
trouble with memory fragmentation.

..wayne..

[Rob klein Gunnewiek]

Thanks for your answer...

Wayne Davison schreef:

> On Fri, Aug 15, 2008 at 05:28:43PM +0200, Rob klein Gunnewiek wrote:
>> ERROR: out of memory in flist_expand [generator]
>
> With -vv, you should be seeing messages like this:
>
> [generator] expand file_list pointer array to N bytes, did move
>
> That will tell you what the last realloc attempt was before the error.

I have used -avvz, I had not noticed the '.. expand file_list pointer array to N
bytes...' type of lines. I will try this again

> If it is a really huge value, check to see if you have a directory with
> a huge number of files in it. You should also check your process's
> memory ulimit values to see if they are too small, and try using a
> different compiler or malloc package to see if your malloc is having
> trouble with memory fragmentation.

I know I have a directory with a huge number of files in it. But this shouldn't
be a big problem. Rsync from another (Linux) system works well. The linux system
has a process stack limit of 8MB. The FreeBSD system from which this fails I
gave 512MB max stack size.

Also, like I said I added 4GB of swap space in the FreeBSD system.

Here is my ulimit output:

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 524288
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 11095
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 524288
cpu time (seconds, -t) unlimited
max user processes (-u) 5547
virtual memory (kbytes, -v) unlimited

Like you suggested, I will try using another malloc library as well.

--
Rob klein Gunnewiek
BWSS B.V. Deventer (http://www.bwss.nl)
Tel +(31) 0570-665140
Fax +(31) 0570-665141

[Rob klein Gunnewiek]

Wayne Davison wrote:
> [...]

>
> That will tell you what the last realloc attempt was before the error.
> If it is a really huge value, check to see if you have a directory with
> a huge number of files in it. You should also check your process's
> memory ulimit values to see if they are too small, and try using a
> different compiler or malloc package to see if your malloc is having
> trouble with memory fragmentation.

I have compiled rsync 3.0.3 with the ptmalloc library
(http://www.malloc.de/en/), now it works!

Thanks again for your suggestion.

--
Rob klein Gunnewiek
BWSS B.V. Deventer (http://www.bwss.nl)
Tel +(31) 0570-665140
Fax +(31) 0570-665141