Compiling rcouch on 64 bit Ubuntu 12.04.1

[Peter Kutschera]

Hi all!

Maybe the following is of interest for someone!

It shows:
1. packages needed to be installed
2 installation of Erlang R15
3.compiling rcouch (including a patch in mochicow_upgrade.erl)
4. Starting rcouch (Very slow and uses only 1 of the available 16GB of ram)


1. On a fresh ubuntu server installation I needed the following packages to install as preparation:
 Well, maybe some are not really used....
 Mercurial and perl are needed by me to run my import scripts.
apt-get install binutils
apt-get install git
apt-get install make
apt-get install g++
apt-get install zip
apt-get install libavahi-compat-libdnssd-dev
apt-get install avahi-utils
apt-get install libavahi-glib-dev
apt-get install mercurial
apt-get install libjson-perl
apt-get install libdata-compare-perl
apt-get install libxml-simple-perl
apt-get install libicu-dev libicu48
apt-get install libwxgtk2.8-0 libwxbase2.8-0

2. installation of erlang R15

# java needed by erlang15 !!!
apt-get install java-common ca-certificates-java tzdata-java liblcms2-2 libnss3-1d icedtea-7-jre-jamvm tzdata openjdk-7-jre-headless openjdk-7-jre-lib libnss3 libnspr4


curl -O 'https://elearning.erlang-solutions.com/couchdb//rbingen_adapter//package_R15B03_precise64_1354121244/esl-erlang_15.b.3-1~ubuntu~precise_amd64.deb'
dpkg --install esl-erlang_15.b.3-1~ubuntu~precise_amd64.deb

3. Get and compile rcouch
git clone https://github.com/refuge/rcouch.git
cd rcouch
make rel --> fail!

edit _ftp/Geocouch/rcouch/deps/mochicow/src/mochicow_upgrade.erl:
%%-spec default_port(atom()) -> 80 | 443.
%%default_port(ssl) -> 443;
%%default_port(_) -> 80.

make rel
cd rel/rcouch

4. start rcouch:
bin/rcouch start

The CouchDB is Slooooow
Looking at `top` shows that beam.smp uses 1020m of memory.
How can I increase this amount?
I did not find an option in any of the./etc files, but I may have missed something there.

Any comments are welcome!

Regards
 Peter

[Benoit Chesneau]

Hi Peter,

I think you are currently build refuge not rcouch (http://rcouch.org). Rcouch doesn't need dnssd & such. Is this something you need ? Actually only rcouch is supported, new version of refuge that should appear sometimes this week introduces a radical change I described in my talk at Erlang Factory:

http://www.erlang-factory.com/upload/presentations/718/refuge_eflparis2013.pdf

(in french unfortunately)

I don't reproduce the slowness at all….  What do you call slow? About the memory, you will see it increases when building views, other than that rcouch/couchdb doesn't use so much ram or CPU on startup (same on a busy node though) once views have been bootstraped…

Couple of notes:

- rcouch and couch_core aren't using anymore cowboy and mochicow. Cowboy introduction sounded like a good idea but were introducing too much issues. They are now gone and instead we are only use ranch to improve the accepting loop of mochiweb so your issue is solved.

- On ubuntu I generally install Erlang from erlang-solutions: 

https://www.erlang-solutions.com/downloads/download-erlang-otp

It will requires to install wx too unfortunately.

- make sure your machine has an hostname and localnet correctly configured. If you don't need to bind all interfaces I woudl strongly suggest to bind rcouch to 127.0.0.1

Hope it helps,

- benoît 

[Kutschera Peter]

Hello Benoit!

Thanks for the fast answer.

I think I got the right rcouch:

> git clone https://github.com/refuge/rcouch.git
> cd rcouch
> make rel

The list of dependencies comes from my past experiments - that's why I installed the server from scratch!

Slow: Well, I have about 500 000 documents in the database.
Creation of indexes needs about 5 minutes!
That's extremely annoying.
And unusable as our use-case includes synchronisation with and interactive GUIs on smartphones.

So I assume there should be more information in the RAM to speed up things.
When looking to the "Status" screen I sometimes see that the indexer starts at e.g. 40% and proceeds very slowly.
But maybe this impression is wrong. The whole CouchDB data directory is about 1GB.

OK 500 000 documents sounds much, but that's why I use a database!

I also had a look to http://wiki.apache.org/couchdb/Performance#Disk_and_File_System_Performance but I think nothing solves my problem.
Especially there are no limits in /etc/security/limits** .
No idea where to trim the system to get faster answers.

All hints welcome!

Best regards
Peter

-----Ursprüngliche Nachricht-----
Von: Benoit Chesneau [mailto:ben...@refuge.io]
Gesendet: Donnerstag, 06. Dezember 2012 13:33
An: Kutschera Peter
Cc: rco...@googlegroups.com
Betreff: Re: Compiling rcouch on 64 bit Ubuntu 12.04.1

Hi Peter,

I think you are currently build refuge not rcouch (http://rcouch.org). Rcouch doesn't need dnssd & such. Is this something you need ? Actually only rcouch is supported, new version of refuge that should appear sometimes this week introduces a radical change I described in my talk at Erlang Factory:

http://www.erlang-factory.com/upload/presentations/718/refuge_eflparis2013.pdf

(in french unfortunately)

I don't reproduce the slowness at all.. What do you call slow? About the memory, you will see it increases when building views, other than that rcouch/couchdb doesn't use so much ram or CPU on startup (same on a busy node though) once views have been bootstraped.

Couple of notes:

- rcouch and couch_core aren't using anymore cowboy and mochicow. Cowboy introduction sounded like a good idea but were introducing too much issues. They are now gone and instead we are only use ranch to improve the accepting loop of mochiweb so your issue is solved.

- On ubuntu I generally install Erlang from erlang-solutions:

https://www.erlang-solutions.com/downloads/download-erlang-otp

It will requires to install wx too unfortunately.

- make sure your machine has an hostname and localnet correctly configured. If you don't need to bind all interfaces I woudl strongly suggest to bind rcouch to 127.0.0.1:5984

[Benoit Chesneau]

---------- Forwarded message ----------
From: Benoit Chesneau <bche...@gmail.com>
Date: Thu, Dec 6, 2012 at 5:49 PM
Subject: Re: [rcouch] AW: Compiling rcouch on 64 bit Ubuntu 12.04.1
To: Kutschera Peter <Peter.K...@ait.ac.at>

On Thu, Dec 6, 2012 at 4:15 PM, Kutschera Peter <Peter.K...@ait.ac.at> wrote:

Hello Benoit!

Thanks for the fast answer.

I think I got the right rcouch:

> git clone https://github.com/refuge/rcouch.git
> cd rcouch
> make rel

The list of dependencies comes from my past experiments - that's why I installed the server from scratch!

Slow: Well, I have about 500 000 documents in the database.
Creation of indexes needs about 5 minutes!
That's extremely annoying.
And unusable as our use-case includes synchronisation with and interactive GUIs on smartphones.

Hum, without saying you I can tell you that some of my custommers are using couchdb on mobile without any problem.  What is the size of each documents?

Also when and how are you measuring that time? Is this UPLOAD + INDEX or just INDEX ? 

Are you indexing swith all the docs in the db?

 

So I assume there should be more information in the RAM to speed up things.
When looking to the "Status" screen I sometimes see that the indexer starts at e.g. 40% and proceeds very slowly.

You may need to reduce the batch size.

 

But maybe this impression is wrong. The whole CouchDB data directory is about 1GB.

OK 500 000 documents sounds much, but that's why I use a database!

agree...

 

I also had a look to http://wiki.apache.org/couchdb/Performance#Disk_and_File_System_Performance but I think nothing solves my problem.
Especially there are no limits in /etc/security/limits** .
No idea where to trim the system to get faster answers.

All hints welcome!

If you can fill the blanks above I will try to help :) 

- benoît

[Hermann]

Hi Benoit!

Please let me come back to the problem stated a few weeks ago. Unfortunately, it still exists. The indexing takes several minutes while view execution finishes in a couple of milliseconds. Please notice my fill ins in the discussion you had below.

Another problem we ran into is the spatial view execution time. After indexing, which takes about 10min, the view file size is 180MB if we just simply emit the coordinates as key and null as value. The corresponding database holds 200.000 documents and has a file size of 115MB (compacted). Do you have any ideas on how to improve the spatial view execution time? We already tried to change the batch size - no success yet.

Best wishes,
Hermann

Am Donnerstag, 6. Dezember 2012 17:52:29 UTC+1 schrieb Benoit Chesneau:

---------- Forwarded message ----------
From: Benoit Chesneau <bche...@gmail.com>
Date: Thu, Dec 6, 2012 at 5:49 PM
Subject: Re: [rcouch] AW: Compiling rcouch on 64 bit Ubuntu 12.04.1
To: Kutschera Peter <Peter.K...@ait.ac.at>

On Thu, Dec 6, 2012 at 4:15 PM, Kutschera Peter <Peter.K...@ait.ac.at> wrote:

Hello Benoit!

Thanks for the fast answer.

I think I got the right rcouch:

> git clone https://github.com/refuge/rcouch.git
> cd rcouch
> make rel

The list of dependencies comes from my past experiments - that's why I installed the server from scratch!

Slow: Well, I have about 500 000 documents in the database.
Creation of indexes needs about 5 minutes!
That's extremely annoying.
And unusable as our use-case includes synchronisation with and interactive GUIs on smartphones.

Hum, without saying you I can tell you that some of my custommers are using couchdb on mobile without any problem.  What is the size of each documents?

A single document has an average size of between 1K and 10K. So I'd say they are rather small.
 

Also when and how are you measuring that time? Is this UPLOAD + INDEX or just INDEX ? 

It's just the INDEXING part.
 

Are you indexing swith all the docs in the db?

Yes, almost all docs.