Scalaris eats CPU and seems to hangs

[Konstantin Cherkasoff]

Hello!

I'm trying scalaris for the first time.
I installed it from prebuild package on Ubuntu 11.10 (see all versions below).
Then I tried to run scalarisctl with the same parameters as in firstnode.sh:

# scalarisctl -n S01 -p 14195 -y 8000 -m -s -f start

After that, I see erlang command prompt but there is no process that listens 8000 and 14195 ports.

Using top I can see that process beam eats 99% of CPU.

I tried
Scalaris 0.3.0 (from http://download.opensuse.org/repositories/home:/scalaris/xUbuntu_11.10/)
Erlang R14B02 (erts-5.8.3)

Thanks for any suggestions.

--

Konstantin Cherkasoff

[Nico Kruber]

Could you please verify if this is still the case with the scalaris-svn
package?

Regards,
Nico Kruber

On Saturday 17 Dec 2011 09:17:21 Konstantin Cherkasoff wrote:
> Hello!
>
> I'm trying scalaris for the first time.
> I installed it from prebuild package on Ubuntu 11.10 (see all versions
> below).

> Then I tried to run *scalarisctl* with the same parameters as in *
> firstnode.sh*:

>
> # scalarisctl -n S01 -p 14195 -y 8000 -m -s -f start
>
> After that, I see erlang command prompt but there is no process that
> listens 8000 and 14195 ports.

> Using *top* I can see that process beam eats 99% of CPU.

[Konstantin Cherkasoff]

> Could you please verify if this is still the case with the scalaris-svn
> package?

Oh, I just forgot to write. With the scalaris-svn I've got the same problem.

--
Konstantin Cherkasoff

[Nico Kruber]

in your scalaris.cfg (or scalaris.local.cfg), please set

{log_level, debug}.
{log_level_file, debug}.

and post the messages from either the command line or the log file.
When I generated the packages, I tested them with Ubuntu 11.04 and they
worked. I will set up a VM with Ubuntu 11.10 to see what happens there.

In the meanwhile, to rule out any packaging issues, could you check out
Scalaris from svn and test there?

svn checkout http://scalaris.googlecode.com/svn/trunk/ scalaris
cd scalaris
./configure
make
./bin/scalarisctl -n S01 -p 14195 -y 8000 -m -s -f start

Nico

[Konstantin Cherkasoff]

> in your scalaris.cfg (or scalaris.local.cfg), please set
>
> {log_level, debug}.
> {log_level_file, debug}.
>
> and post the messages from either the command line or the log file.

Oh, I forgot to say.
I've already tried to turn on debug. Log files were _empty_.

I'll try to build scalaris from sources.

--
Konstantin Cherkasoff

[Nico Kruber]

please also try

scalarisctl checkinstallation

we put some basic checks there to find common caveats.
(The 0.3.0 version though has some false warnings in the Ubuntu packages - try
the one from svn trunk, i.e. scalaris-svn.)

Nico

[Konstantin Cherkasoff]

Hello!

> please also try
>
> scalarisctl checkinstallation
>
> we put some basic checks there to find common caveats.
> (The 0.3.0 version though has some false warnings in the Ubuntu packages - try
> the one from svn trunk, i.e. scalaris-svn.

As far as I understand checkinstallation doesn't work without scalaris-svn-java.
After I installed scalaris-svn-java package

# scalarisctl checkinstallation

does not produce any output.

Does it mean there is no problems with intsallation ?

--
Konstantin Cherkasoff

[Nico Kruber]

On Sunday 25 December 2011 16:22:45 Konstantin Cherkasoff wrote:
> Hello!
>
> > please also try
> >
> > scalarisctl checkinstallation
> >
> > we put some basic checks there to find common caveats.
> > (The 0.3.0 version though has some false warnings in the Ubuntu packages
> > - try the one from svn trunk, i.e. scalaris-svn.
>
> As far as I understand checkinstallation doesn't work without
> scalaris-svn-java. After I installed scalaris-svn-java package

without the scalaris-svn-java package, checkinstallation should complain about
them missing but the rest should still be checked

> # scalarisctl checkinstallation
>
> does not produce any output.
>
> Does it mean there is no problems with intsallation ?

Unfortunately it only means that whatever you are experiencing has not
occurred to us yet.

How did your tests with a source checkout from svn go?

Nico

[Nico Kruber]

I just set up a VM with Ubuntu 11.10 (x86_64) and installed our scalaris-svn
package.
I did not change any config and could simply run it after installation:

===== as my user =====
nico@ubuntu1110:~$ scalarisctl -n S01 -p 14195 -y8000 -m -s -f start
mkdir: cannot create directory `/var/log/scalaris/S01@ubuntu1110': Permission
denied
Erlang R14B02 (erts-5.8.3) [source] [64-bit] [rq:1] [async-threads:0] [kernel-
poll:false]

Eshell V5.8.3 (abort with ^G)
(S01@ubuntu1110)1>
=ERROR REPORT==== 2-Jan-2012::11:30:29 ===
can not open logfile "/var/log/scalaris/S01@ubuntu1110/scalaris_error_logger":
enoent[error] Cannot open "/var/log/scalaris/S01@ubuntu1110/localhost.auth"
[warn] <0.120.0> (name: "main -> 127.0.0.1:14196") Send to
{{127,0,0,1},14196,service_per_vm} failed, drop message
{send_to_group_member,service_per_vm,{hi}} due to tcp_connect_failed
[warn] <0.121.0> (name: "main -> 127.0.0.1:14197") Send to
{{127,0,0,1},14197,service_per_vm} failed, drop message
{send_to_group_member,service_per_vm,{hi}} due to tcp_connect_failed
[warn] <0.122.0> (name: "main -> 127.0.0.1:14198") Send to
{{127,0,0,1},14198,service_per_vm} failed, drop message
{send_to_group_member,service_per_vm,{hi}} due to tcp_connect_failed

(S01@ubuntu1110)1>

===== as root =====
nico@ubuntu1110:~$ sudo -s
root@ubuntu1110:~# scalarisctl -n S01 -p 14195 -y8000 -m -s -f start
Erlang R14B02 (erts-5.8.3) [source] [64-bit] [rq:1] [async-threads:0] [kernel-
poll:false]

Eshell V5.8.3 (abort with ^G)
(S01@ubuntu1110)1> [warn] <0.120.0> (name: "main -> 127.0.0.1:14196") Send to
{{127,0,0,1},14196,service_per_vm} failed, drop message
{send_to_group_member,service_per_vm,{hi}} due to tcp_connect_failed
[warn] <0.121.0> (name: "main -> 127.0.0.1:14197") Send to
{{127,0,0,1},14197,service_per_vm} failed, drop message
{send_to_group_member,service_per_vm,{hi}} due to tcp_connect_failed
[warn] <0.122.0> (name: "main -> 127.0.0.1:14198") Send to
{{127,0,0,1},14198,service_per_vm} failed, drop message
{send_to_group_member,service_per_vm,{hi}} due to tcp_connect_failed

(S01@ubuntu1110)1>

=========================
I could also access our web debug interface and the system seamed operational.

nico@ubuntu1110:~$ ls -la /var/log/scalaris/
total 12
drwxr-xr-x 3 root root 4096 2012-01-02 11:34 .
drwxr-xr-x 14 root root 4096 2012-01-02 11:29 ..
drwxr-xr-x 2 root root 4096 2012-01-02 11:34 S01@ubuntu1110

I will probably liberate the access rights on the log folder in a future
commit.

Did you change anything after installation (config, etc)?
Could you also please provide the complete terminal output?

[Konstantin Cherkasoff]

Hello Nico!

> How did your tests with a source checkout from svn go?

I've tried to build Scalaris from sources (trunk).
Under my Ubuntu 11.10 configure script complains several times:
1. configure: WARNING: "" was not found or is not executable ('make
test-vts' will likely fail)
2. configure: erlang-edoc not found
3. configure: run_test not found
4. configure: toke erlang library not found
5. configure: -callback is not available

Is some of them mission critical ?
The manual (requirements) does not say that toke is required.

Anyway on the make stage I've got:
Recompile: src/comm_layer/comm_acceptor
src/comm_layer/comm_acceptor.erl:112: can't find include lib
"kernel/include/inet.hrl"
src/comm_layer/comm_acceptor.erl:117: record hostent undefined

This is minimal system Ubuntu 11.10 on VPS.
Riak and Couchdb running fine.

--
Konstantin Cherkasoff

[Konstantin Cherkasoff]

Hello Nico!

> I just set up a VM with Ubuntu 11.10 (x86_64) and installed our scalaris-svn
> package.
> I did not change any config and could simply run it after installation:

[...]
I found what was wrong.
I mixed up distro version in /etc/apt/sources.list
So under Ubuntu 11.10 I tried to run package from .../xUbuntu_11.04/

And beam starts to eat CPU. Actualy its very strange...

--
Konstantin Cherkasoff

[Nico Kruber]

On Saturday 14 January 2012 04:20:29 Konstantin Cherkasoff wrote:
> Hello Nico!
>
> > How did your tests with a source checkout from svn go?
>
> I've tried to build Scalaris from sources (trunk).
> Under my Ubuntu 11.10 configure script complains several times:
> 1. configure: WARNING: "" was not found or is not executable ('make
> test-vts' will likely fail)
> 2. configure: erlang-edoc not found
> 3. configure: run_test not found
> 4. configure: toke erlang library not found
> 5. configure: -callback is not available

none of these is critical - all critical things let the configure process fail

> Anyway on the make stage I've got:
> Recompile: src/comm_layer/comm_acceptor
> src/comm_layer/comm_acceptor.erl:112: can't find include lib
> "kernel/include/inet.hrl"
> src/comm_layer/comm_acceptor.erl:117: record hostent undefined

this is critical and compile should have stopped

Debian and Ubuntu split the erlang package into several sub-packages. For a
full build, the following packages are needed (taken from our scalaris-
svn.dsc):
erlang-dev (>= 13.b.1), erlang-tools (>= 13.b.1), erlang-edoc (>= 13.b.1),
erlang-crypto (>= 13.b.1), erlang-common-test (>= 13.b.1)

[Nico Kruber]

that might be because of these included system .hrl files which might have
changed - or anything else.

Unfortunately I can not put requirements into the packages which depend on the
Erlang version they have been built. I can think of something for rpm-based
distros, but nothing for Debian.
Seems we need to take extra caution adding the package URLs.

Nico