Connection strategy / Node starting

[Matthieu Ghilain]

Hi everybody,

I have smalls questions about your system.

What is the best strategy to start a ring correctly, currently, i
start node one by one wait one or two second between two node
launches, all the nodes connect to the same first node, I never
launched more than 20 nodes.

Sometimes it works, sometimes some nodes never connects. I based my
observation on the web server of the management server which run on
his own node. Will eventually all nodes connect if the machine do not
crash?

So how should I start my ring, is it a good idea to always connect
first to the same node?

How should I connect to my nodes? Currently I am using one separate
node which create as many thread as node and use a different
connection to each node, never close this connection and dispatch
operations between threads. Is it a good idea to have several
connections to the same Scalaris node, or is it better for a Scalaris
node to handle one transaction at a time?

Other thing, when I run basic non conflicting tests I do not have any
problem, but as soon as I run tests for my real applications where
there are several transaction in parallel that can conflict, I got
lots of timeout/abort exception and performances drop drastically. Do
you have any advice?

One last question, I have a strange observation, I am forced to start
scalaris two times ( start it, close it, start it ) on all the nodes
in order to have java connect to it, otherwize it never connects...
this only happens in distributed settings and java can connect to the
firstnode (with name firstnode@... ) but not the other node (which are
named node1@... ) if I don't use this trick. Do you have any idea of
the problem ?

Thanks a lot for your time, and sorry for my english, I know it's far
from perfect.

Have a nice day.

[Matthieu Ghilain]

My main problem is this force restart of scalaris node,
it's impossible to do a write to other node than the first node i
absolutly don't know why, because nodes
seems correctly connected and I don't get any error message until I
try to do a write on another node than
the first node. Restarting the other nodes ( not the first one ) solve
the problem, I can now do write on other
nodes too, but I then got a lot of error message saying incarnation of
old nodes... How can I cleanly restart
a node without receiving such messages...

[Florian Schintke]

Hi,

sorry, but I cannot reproduce that behaviour. Via which API are you
accessing the nodes when you trigger the write? All started nodes
should join the system. When their first attempt fails, they will
retry it after about 30 seconds. So maybe you just have to wait a bit
longer?

For the restart of a node: How did you performed the restart? The
messages you see are not critical and should after some time (a few
minutes) stop to appear.

Florian

[Matthieu Ghilain]

> --
> You received this message because you are subscribed to the Google Groups "scalaris" group.
> To post to this group, send email to scal...@googlegroups.com.
> To unsubscribe from this group, send email to scalaris+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scalaris?hl=en.

Florian

[Matthieu Ghilain]

I simply kill the beam process.

I tried access it through the web server interface and the java
interface. I can create a connection to the node but any transaction
on it
will result in a time out.

To solve this I restart it by killing the process and launching the
node again, then i start a timer and when the timer reach the end ( 20
seconds ) I test again a write and
if it fails restart against, but this is really time consumming and
does not alway work well

Here are the log I received after having started a ring of 8 nodes and
trying to write a value through the web interface of another node than
the first node...

2011-05-27 14:13:20.234052 [warn] [ Node <0.109.0> ] join timeout hit,
starting over...
2011-05-27 14:13:20.442334 [error] [ CL ] wrong call to comm:send:
{undefined,14196,<0.119.0>} ! {get_node_details_response,[{node,{node,
{{10,244,42,148},14196,<0.109.0>},
95994823025181571142405103898020939872,0}}]}
2011-05-27 14:13:20.442709 [error] [ CL ] stacktrace:
[{comm_layer,send,2},{dht_node,on,2},{gen_component,loop,5},
{gen_component,try_loop,4},{gen_component,start,4}]
2011-05-27 14:20:39.548032 [error] No result for commit received!


And each time i try a new transaction I get a No result for commit
received!

And on the first node I checked the log and here is what i get ( there
is a lot more ) :

2011-05-27 14:13:31.664274 [warn] [ <0.109.0> ] Routing is damaged!!
Trying again...
myrange:{'(',266136006485650802874092407613905045600,
308671302350768110807014233542876072032,']'}
db_range:[]
msgfwd:[]
Key:266136006485650802874092407613905045600
2011-05-27 14:13:31.665012 [warn] [ <0.109.0> ] Routing is damaged!!
Trying again...
myrange:{'(',266136006485650802874092407613905045600,
308671302350768110807014233542876072032,']'}
db_range:[]
msgfwd:[]
Key:266136006485650802874092407613905045600
2011-05-27 14:13:31.665658 [warn] [ <0.109.0> ] Routing is damaged!!
Trying again...
myrange:{'(',266136006485650802874092407613905045600,
308671302350768110807014233542876072032,']'}
db_range:[]
msgfwd:[]
Key:266136006485650802874092407613905045600
2011-05-27 14:13:31.668476 [warn] [ <0.109.0> ] Routing is damaged!!
Trying again...
myrange:{'(',266136006485650802874092407613905045600,
308671302350768110807014233542876072032,']'}
db_range:[]
msgfwd:[]
Key:266136006485650802874092407613905045600


I really need to have Scalaris working properly and I can't get rid of
all those errors.

Here is my configuration file :

% IP Address, Port, and label of the boot server
{mgmt_server, {{10,245,65,19}, 14195, mgmt_server}}.

% IP Address, Port, and label of a node which is already in the
system
{known_hosts, [{{10,245,67,29}, 14195, service_per_vm}]}.

The management server is running on his own node. As I said before and
because I find it so strange, the first node is the
only on which I can do writes. The other need to be restarted. Maybe
is it because they are all named node1? Is it a problem?

On May 27, 9:57 am, Florian Schintke <schin...@gmail.com> wrote:
> Hi,
>
> sorry, but I cannot reproduce that behaviour. Via which API are you
> accessing the nodes when you trigger the write? All started nodes
> should join the system. When their first attempt fails, they will
> retry it after about 30 seconds. So maybe you just have to wait a bit
> longer?
>
> For the restart of a node: How did you performed the restart? The
> messages you see are not critical and should after some time (a few
> minutes) stop to appear.
>
> Florian
>
> [Matthieu Ghilain]
>
>
>
>
>
>
>
>
>
> > My main problem is this force restart of scalaris node,
> > it's impossible to do a write to other node than the first node i
> > absolutly don't know why, because nodes
> > seems correctly connected and I don't get any error message until I
> > try to do a write on another node than
> > the first node. Restarting the other nodes ( not the first one ) solve
> > the problem, I can now do write on other
> > nodes too, but I then got a lot of error message saying incarnation of
> > old nodes... How can I cleanly restart
> > a node without receiving such messages...
>

[Matthieu Ghilain]

I have one more problem... Maybe this you can have an answer more
easily.

Sometimes i just can't write a value whatever the node.

I am doing a benchmark writing 10000 values with 4 nodes, and
frequently there is one or two values that I can't
write and give abort whatever the time between the tries. I am waiting
10 seconds after I see I have a problem
with this value, I check that my node are up ( CPU nearly idle ) and
nothing to do this wirte aborts.

I also tried with the web interface on several nodes and it gives the
same result : abort...

Maybe you have an idea :).

I am really sorry for all those questions.

[Matthieu Ghilain]

Hello again,

I still have the problem that I need to start two times a node for him
in order to do write on it.

But I have another problem, even when I am able to launch correctly
the ring, after some thousand of transactions I got problems
and some nodes seem to never stop being overloaded and start to abort
all the times.
I observe that at then end my test tell me that it remains some values
to write but can never write it.
I tried with the web interface it's not possible neither to write
those value, response is always abort.
Here are a part of the logs of the machine overloaded :

2011-05-28 15:51:36.575358 [error] Error: exception error:{badmatch,
{new,
{{tx_id,
{16983051,
{{10,195,189,6},
14198,<9109.119.0>}}},
tx_state,unknown,unknown,[],[],
[],undecided,0,
0,0,0,0,0,uninitialized,[]}}} during
handling of {learner_decide,{tx_item_id,{16983052,{{10,195,189,6},
14198,<9109.119.0>}}},
{{16983054,{{10,195,189,6},14198,<9109.119.0>}}},
abort} in module tx_tm_rtm in
({[],dht_node_118253675_tx_tm_rtm_tx_rtm3,tx_rtm3,<0.138.0>,
{{10,242,78,55},14195,<0.139.0>}}) with on - stacktrace:
[{tx_tm_rtm,on,2},

{gen_component,loop,5},
{gen_component,try_loop,4},
{gen_component,start,4}]

2011-05-28 15:51:40.483452 [warn] Takeover by RTM was necessary.
2011-05-28 15:51:40.484751 [warn] Takeover by RTM was necessary.
2011-05-28 15:51:40.493285 [warn] Takeover by RTM was necessary.
2011-05-28 15:51:40.503350 [warn] Takeover by RTM was necessary.
2011-05-28 15:51:40.503812 [error] duplicate learner initialize for id
{{60257129,
{{10,202,34,97},
14196,<8956.119.0>}}}
2011-05-28 15:51:40.503842 [error] Duplicate proposer:initialize for
paxos id {{60257129,
{{10,202,34,97},
14196,<8956.119.0>}}}
Just triggering instead

2011-05-28 15:51:40.507566 [warn] Takeover by RTM was necessary.
2011-05-28 15:51:40.511322 [error] duplicate learner initialize for id
{{60257130,
{{10,202,34,97},
14196,<8956.119.0>}}}
2011-05-28 15:51:40.512435 [error] Duplicate proposer:initialize for
paxos id {{60257130,
{{10,202,34,97},
14196,<8956.119.0>}}}
Just triggering instead

Even after stopping my test waiting 10 minuts I can't write those
value ( tried via the web interface ).

Maybe is it because the micro instances of amazone are too weak to
support Scalaris.
Do you have any idea of the minimal requirement of the machine in
order to support correctly
Scalaris?

[Florian Schintke]

Regarding the connectivity: Does admin:number_of_nodes() list all the
nodes? Can all the servers ping each other on the operating system
level? Does all servers have the same configuration in respect to the
management-server and the known hosts? Your log indicates, that some
nodes where not able to determine their own IP-address, which is
usually done by contacting the management-server. Is the
management-server reachable?

Scalaris currently does not handle high load situations very well. We
are aware of that and I am currently developing a solution, so
Scalaris can handle overload situations. It is mainly finished and I
already ran high loads successfully, but some failure szenarios (when
more than a majority of transaction participants were killed) are not
sufficiently handled. Maybe I can submit the status quo next week and
do the more elaborate scenarios in a separate commit afterwards.

For the time being, just don't put such a high load on Scalaris.

Florian

[Matthieu Ghilain]

First thanks for answering :).

What is the management used for? I launched it separately from the
first node in order to lower as much as possible the load on nodes
and to be fair with respect for other node but is it a good idea?

Here is the configuration of my installation, I launched three amazone
nodes : one management server, one first node, one classical node.
First machine are started then on all the machines I copy the same
configuration file with the IP address of the management server and
the IP address of
the first node. I put it below

% IP Address, Port, and label of the boot server

{mgmt_server, {{10,245,119,226}, 14195, mgmt_server}}.

% IP Address, Port, and label of a node which is already in the
system

{known_hosts, [{{10,195,207,232}, 14195, service_per_vm}]}.

I also copy a host file constaining the matching between the hosts
names (returned by hostname --long) and the ip address of the nodes,
I saw that it was sometimes needed by Erlang just a precaution.

Hosts file :
10.245.119.226 ip-10-245-119-226.ec2.internal
10.195.207.232 ip-10-195-207-232.ec2.internal
10.195.77.23 ip-10-195-77-23.ec2.internal

Then i launch Scalaris on the three node in the following order, first
the management server, then the first node, then the normal node.
I can see when connection on the web interface of the management
server that he has 2 nodes in the ring so they seem well connected.
They can all ping each other so no problem of connectivity.

Launching management server :
/home/ubuntu/trunk/bin/scalarissctl -n mgmt_server -d -p 14195 -y 8000
-m start

Launching first node :
/home/ubuntu/trunk/bin/firstnode.sh

Launching the normal node :
/home/ubuntu/trunk/bin/joining_node.sh 1

Here is first node script :
HOSTNAME=$(hostname --long)
NAME="firstnode@$HOSTNAME"
PORT=$((14195))
YAWSPORT=$((8000))

ABSPATH="$(cd "${0%/*}" 2>/dev/null; echo "$PWD"/"${0##*/}")"
DIRNAME=`dirname $ABSPATH`

NODE_NAME_AND_PORTS="-n $NAME -p $PORT -y $YAWSPORT"
$DIRNAME/scalarisctl $NODE_NAME_AND_PORTS -s -d -f start $*


Here is the joining_node script:
ID=1

if [ 1 == $# ]; then
ID=$1
shift;
fi
HOSTNAME=$(hostname --long)
NAME="node$ID@$HOSTNAME"
CSPORT=$((14195+$ID))
YAWSPORT=$((8000+$ID))

ABSPATH="$(cd "${0%/*}" 2>/dev/null; echo "$PWD"/"${0##*/}")"
DIRNAME=`dirname $ABSPATH`

$DIRNAME/scalarisctl -n $NAME -p $CSPORT -y $YAWSPORT -d -s start $*

Launching last node :
/home/ubuntu/trunk/bin/joining_node.sh

Then I connect on the normal node and check the log, here is what I
get...

2011-05-29 07:00:53.608026 [warn] [ Node <0.109.0> ] join timeout hit,
starting over...
2011-05-29 07:00:53.914017 [error] [ CL ] wrong call to comm:send:

{undefined,14196,<0.119.0>} ! {get_node_details_response,[{node,{node,

{{10,195,77,23},14196,<0.109.0>},
45520918733171919070870689492921621086,0}}]}
2011-05-29 07:00:53.914261 [error] [ CL ] stacktrace:

[{comm_layer,send,2},{dht_node,on,2},{gen_component,loop,5},
{gen_component,try_loop,4},{gen_component,start,4}]

Could you explain me the synthax of the first error log? Because I
don't understand why this node try to contact the port 14196 as it is
the port he used for communication between DHT nodes, don't know why
he try to contact himself.

So I then try via the web interface of the normal node to add a
value :
(foo, bar)

I got a timeout after 5 seconds, which I see is the classic
configuration of scalaris.

I check the log of the normal node again a new entry appears

2011-05-29 07:17:15.449023 [error] No result for commit received!

But not much to say about that....

I checked the log on the other nodes, nothing on the management server
and one little log on the first node saying he could not connect to
management server (think I forgot to put little sleep between the
launching of the mangement server and the first node, maybe the
management server was not completely started).

I continue the experiment. Now I come back on the web interface of the
first node. I do a add (foo, bar ), reponse : ok. I go on the web
interface of the normal node and to a look up on (foo) which succeed
and return bar. So they must be connected. No new log everything
worked well, but I still can't do any add on the normal node always
return a timeout.

I then go on the first node machine do the following commands in a
small script to stop the node and restart it.

sudo killall epmd
sudo killall beam
/home/ubuntu/trunk/bin/joining_node.sh 1

I of course got some logs on the first node saying the other node has
crashed :

2011-05-29 07:27:37.758911 [warn] [ CC ] tcp closed info
2011-05-29 07:27:51.340024 [warn] [ FD {{10,195,207,232},
14195,<0.189.0>} ] reports [{{{10,195,77,23},14196,<8956.134.0>},1},
{{{10,195,77,23},14196,<8956.129.0>},1},
{{{10,195,77,23},14196,<8956.109.0>},2}] as crashed
2011-05-29 07:27:51.343845 [warn] got unsubscribe for not registered
subscription {unsubscribe,<0.109.0>,{{10,195,77,23},
14196,<8956.109.0>},'$fd_nil'}

2011-05-29 07:27:51.384590 [warn] got unsubscribe for not registered
subscription {unsubscribe,<0.120.0>,{{10,195,77,23},
14196,<8956.134.0>},'$fd_nil'}

2011-05-29 07:27:51.387980 [error] req. to delete non watched pid

2011-05-29 07:27:51.389983 [warn] got unsubscribe for not registered
subscription {unsubscribe,<0.120.0>,{{10,195,77,23},
14196,<8956.129.0>},'$fd_nil'}

2011-05-29 07:27:51.392495 [error] req. to delete non watched pid

Then I get full of messages that seems never end on the normal node :

2011-05-29 07:28:15.464778 [error] Discarding message {ping,
{{{10,195,207,232},14195,<8747.99.0>},c,{node,{{10,195,77,23},
14196,<0.109.0>},45520918733171919070870689492921621086,0}}} from
<0.123.0> to <0.109.0> in an old incarnation (2) of this node (1)

2011-05-29 07:28:36.921044 [error] Discarding message {ping,
{{{10,245,119,226},14195,<8781.78.0>},c,{{10,195,77,23},
14196,<0.109.0>}}} from <0.141.0> to <0.109.0> in an old incarnation
(2) of this node (1)

2011-05-29 07:28:45.464080 [error] Discarding message {ping,
{{{10,195,207,232},14195,<8747.99.0>},c,{node,{{10,195,77,23},
14196,<0.109.0>},45520918733171919070870689492921621086,0}}} from
<0.123.0> to <0.109.0> in an old incarnation (2) of this node (1)

The strange thing is that now I can do add on the first node via the
web interface as well. This hack only work if i restart the node
exactely the same way. If I give to the node another name/port it
still can not do any writes.

Hope it's clear, because it's not so easy to describe correctly what I
do.

> ...
>
> read more »

[Florian Schintke]

Thanks for the detailed description. It looks fine to me.

> Then I connect on the normal node and check the log, here is what I
> get...
>
> 2011-05-29 07:00:53.608026 [warn] [ Node <0.109.0> ] join timeout hit,
> starting over...
> 2011-05-29 07:00:53.914017 [error] [ CL ] wrong call to comm:send:
> {undefined,14196,<0.119.0>} ! {get_node_details_response,[{node,{node,
> {{10,195,77,23},14196,<0.109.0>},
> 45520918733171919070870689492921621086,0}}]}
> 2011-05-29 07:00:53.914261 [error] [ CL ] stacktrace:
> [{comm_layer,send,2},{dht_node,on,2},{gen_component,loop,5},
> {gen_component,try_loop,4},{gen_component,start,4}]
>
> Could you explain me the synthax of the first error log? Because I
> don't understand why this node try to contact the port 14196 as it is
> the port he used for communication between DHT nodes, don't know why
> he try to contact himself.

It is fine when a node tries to contact itself, in principle. But the
'undefined' is a bit strange here. Normally at this place, we have an
IP-Address. Just in the beginning, when a node does not know its
IP-address, we have undefined here, so some node was not able to
determine its IP-address (usually done by contacting the
management-server to determine the outgoing network interface and
associated IP).

Another strange thing is the join timeout hit. Already this should not
happen normally. I tried your steps locally on my machine and I don't
get these errors. Everything works as expected. Maybe you can also try
locally first? Or you could check whether some firewall issues in your
setup by using 'telnet <IP> <port>' between the nodes to check
connectivity...? I have no other ideas currently.

Cheers,

Florian

[Matthieu Ghilain]

Hi I have found the problem of connectivity!

Sometimes amazone machine are not reachable via ping... really
annoying situation.

But anyway even if that work I have to start Scalaris nodes two time
except the first node to make a write on those nodes.
Moreover I have no error log when I connect them the first time, only
a small log saying that no commit was received...

[Florian Schintke]

Hi, is the EC2 firewall configured properly between the nodes?
Maybe the port 14195 is blocked somehow?

Florian

[Matthieu Ghilain]

Sorry if this message appears several time got a problem while posting

Here is the firewall configuration, it's the same for all the nodes and everything is open. Moreover connectivity is finally reached as restarting the node with the same configuration ( and thus the same port ) works.

http://img10.imageshack.us/i/screenshot3qk.png/

2011/5/31 Florian Schintke <schi...@gmail.com>

[Florian Schintke]

So, do you think this node restart, which seems to be necessary on
your deployment (always? / sometimes? / rarely?), is EC2 specific or a
problem of Scalaris?

So, for Scalaris, the issue remains, that you cannot issue write
requests from the joining node, even when it was restarted?

Florian

Florian

[Matthieu Ghilain]

I did one more time a trial on my own network to avoid possible amazone problems.

4 nodes on my local machine and 3 nodes on another machine.
On the machine launching the ring all the nodes could directly do a write. On the other machine one of the three node could not
do the write and nothing was written in the log except the no commit received message.

On Amazon i have to restart 9/10 a node before being able to do write on it.
I do think it's a problem of some kind with my configuration or with Scalaris because I got the same
problem using my computers at home.

Could you tell me exactly how YOU launch your node distributed...

I personnaly always put as known host the first node, I use different name each time I launch a node
on a machine. I hope it's not a problem the fix the hostname while launching the node, I guess it's not because
it's working but you do not do so in your example script. I only have one node by machine on amazone and my management
server is running alone on other node, the first thing i launch is the management server and I wait 10 between
each node starting.

hostname=$(hostname --long)
nodeX@hostname

2011/6/1 Florian Schintke <schi...@gmail.com>

[Florian Schintke]

Hi Matthieu Ghilain,

today, I was able to reproduce your problem on one of our clusters and
got it fixed as of SVN revision r1911. We will also include this fix
in our upcoming release.

Would be nice if you could give it a try.

Thanks for your patience,

Florian