RE: Assistance in tuning CouchDB heap and/or replicator config

[Mat Degerholm]

Hi again,

Apologies for the re-send but as I didn't receive any replies to my earlier message I wanted to check it hadn't slipped through the cracks. Can anyone advise of a way to increase the Couch/Erlang heap?

Regards,
Mat


From: Mat Degerholm
Sent: Wednesday, 22 October 2014 5:25 p.m.
To: 'us...@couchdb.apache.org'
Cc: Mat Degerholm
Subject: Assistance in tuning CouchDB heap and/or replicator config

Hi guys,

Just looking for a little assistance in tuning the CouchDB heap and/or replicator config.

When performing an initial/full replication to an empty 2nd node using the _replicator database, the 2nd node crashes with the following error:

hend=0x00007f280d67f628
stop=0x00007f280d67f418
htop=0x00007f280d6a27b0
heap=0x00007f280d592028
beam/erl_gc.c, line 427: <0.194.0>: Overrun stack and heap
heart: Tue Oct 21 13:29:55 2014: Erlang has closed.
heart: Tue Oct 21 13:29:56 2014: Executed "/opt/couchdb/bin/couchdb -k" -> 0. Terminating.

As my replication source node does have a sizeable data set (16,000 docs, 180,000 updates, total size around 1GB), I found I could prevent the crash by reducing the replication worker batch size. But I found I had to drop down 2 orders of magnitude (from the default of 500 down to 5) before the problem disappeared!

[replicator]
; With lower batch sizes checkpoints are done more frequently. Lower batch sizes
; also reduce the total amount of used RAM memory.
worker_batch_size = 5

I'm nervous having to tweak the replicator setup so far from the defaults, so is there any way I can tune the stack/heap? I have included my ulimit output in case you could recommend any changes (note that my stack size and max memory are unlimited).

Thanks in advance,
Mat


Technical Details

CouchDB version: 1.6.0
Erlang: OTP 17.0
OS: RHEL 6.5
ERL_MAX_PORTS: 4096
ERL_FLAGS: "+A 4"


ulimit output:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 30502
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 4096
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[Alexander Shorin]

Hi,

Instead of reducing worker batch size to such low value (which means
that replication will have to make at least 3200 requests to transfer
your documents), you can also try to increase Erlang process heap size
by using +hms parameter via ERL_FLAGS environment variable. You can
read more about it here:
http://www.erlang.org/doc/man/erl.html
http://www.erlang.org/doc/efficiency_guide/processes.html

--
,,,^..^,,,