consequences of node death

[Roger Critchlow]

Hi --

I'm at a point where my application wants to test itself by losing part of the DHT ring and recovering from the failure.  I'm running my modifications on top of the scalaris svn pulled this morning.

I notice that this version of scalaris fails a node join/leave test, test #326 I believe.

I was testing my own code by sending the message {die} to one of the nodes.  The node ends itself, and the supervisor takes down a series of other processes, too.

But I get the appended errors logged during the recovery.  

Is the ring recovered or not?  

Is the failure in the test suite expected?

Is {die} the right way to kill a node to test recovery?

-- rec --

2011-02-15 12:59:16.489271 [error] Error: exception {badarg,[{comm,send_local,2},

                          {rt_chord,check,5},

                          {rt_loop,on_active,2},

                          {gen_component,loop,4},

                          {gen_component,start,4}]} during handling of {crash,{{127,0,0,1},14196,<0.199.0>}} in module rt_loop in ({{[{node,{{127,0,0,1},14196,<0.155.0>},

[3000+ lines removed]

                {{14,61},

                 {node,{{127,0,0,1},14196,<0.109.0>},

                       72021918025239873891586113773715102545,0},

                 nil,nil},

                nil}}}}}}}}}}}}}}},

 {trigger_periodic,{#Fun<rt_loop.0.16883589>,trigger_rt,#Ref<0.0.0.13663>}}}) with on_active

2011-02-15 12:59:20.48705 [error] unknown message: {activate_reregister}

 in Module: dht_node_reregister and handler on_active

 in State {trigger_periodic,{#Fun<dht_node_reregister.0.125967971>,register,

                   #Ref<0.0.0.1224>}}

2011-02-15 12:59:20.48994 [error] unknown message: {activate_cyclon}

 in Module: cyclon and handler on_active

 in State {[{{node,{{127,0,0,1},14196,<0.155.0>},

         242163101485709105623273417489599208273,0},

   0},

  {{node,{{127,0,0,1},14196,<0.109.0>},

         72021918025239873891586113773715102545,0},

   3},

  {{node,{{127,0,0,1},14196,<0.243.0>},

         157092509755474489757429765631657155409,0},

   2}],

 {node,{{127,0,0,1},14196,<0.199.0>},

       327233693215943721489117069347541261137,0},

 99,

 {trigger_periodic,{#Fun<cyclon.0.51784117>,cy_shuffle,#Ref<0.0.0.14553>}},

 cyclon}

2011-02-15 12:59:20.49523 [error] unknown message: {activate_rt,{[{node,{{127,0,0,1},14196,<0.155.0>},

                     242163101485709105623273417489599208273,0}],

              {node,{{127,0,0,1},14196,<0.5969.0>},

                    305966045283385067522656156383055747921,0},

              [{node,{{127,0,0,1},14196,<0.287.0>},

                     29486622160122565958664287844744076113,0}]}}

 in Module: rt_loop and handler on_active

 in State 

  [ state removed ]

[Florian Schintke]

[Roger Critchlow]

> Hi --
>
> I'm at a point where my application wants to test itself by losing part of
> the DHT ring and recovering from the failure. I'm running my modifications
> on top of the scalaris svn pulled this morning.
>
> I notice that this version of scalaris fails a node join/leave test, test
> #326 I believe.
>
> I was testing my own code by sending the message {die} to one of the nodes.
> The node ends itself, and the supervisor takes down a series of other
> processes, too.
>
> But I get the appended errors logged during the recovery.
>
> Is the ring recovered or not?

You may call in a Scalaris shell
admin:number_of_nodes().
admin:check_ring(). % should return ok
admin:check_ring_deep(). % should return ok

You can also take a look into the web interface at localhost:8000 on the
'Indexed Ring' tab. If the successorlists contain red entries, the ring
is not yet fixed (if you have still more than a single node).

> Is the failure in the test suite expected?

Yes. We know that the test fails. There is still a bug in the data
handover when nodes join and leaves that occurs under relatively rare
circumstences. We are working on that. A second reason is, that the
old transaction implementation cannot handle the data handover. We
will not fix that, but will get rid of the old tx implementation as
the new one now supports failures during tx execution itself.

> Is {die} the right way to kill a node to test recovery?

You may do that, yes. Another way is to start a node in a separate
erlang vm (with bin/cs_local.sh for example) and then kill this vm by
pressing Ctrl-C Ctrl-C.

Florian

[Roger Critchlow]

On Thu, Feb 17, 2011 at 5:02 AM, Florian Schintke <schi...@gmail.com> wrote:

[Roger Critchlow]

> Hi --
>
> I'm at a point where my application wants to test itself by losing part of
> the DHT ring and recovering from the failure.  I'm running my modifications
> on top of the scalaris svn pulled this morning.
>

[ ... ]

> Is the ring recovered or not?

You may call in a Scalaris shell
admin:number_of_nodes().
admin:check_ring(). % should return ok
admin:check_ring_deep(). % should return ok

 

Hi Florian --

Thanks for the suggestions, but none of that really helped.

 

What I ended up doing, after much thrashing around, is polling all live nodes for all nodes listed in their routing tables.  What I'm finding is that the dead node is still present in somebody's routing table an hour after it died.  This is using the scalaris sources from this morning.

But I also get the impression that someone is working on the problem.  I hope it's high on the to do list.

-- rec --

[Florian Schintke]

Hi Roger,

I just fixed an issue with our failure detectors that *might* have
caused also your error. Can you reproduce the wrong behaviour? Could
you please test with svn rev. 1624?

Florian

> Hi Florian --
>
> Thanks for the suggestions, but none of that really helped.
>
> What I ended up doing, after much thrashing around, is polling all live
> nodes for all nodes listed in their routing tables. What I'm finding is
> that the dead node is still present in somebody's routing table an hour
> after it died. This is using the scalaris sources from this morning.
>
> But I also get the impression that someone is working on the problem. I
> hope it's high on the to do list.
>
> -- rec --
>

> --
> You received this message because you are subscribed to the Google Groups "scalaris" group.
> To post to this group, send email to scal...@googlegroups.com.
> To unsubscribe from this group, send email to scalaris+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scalaris?hl=en.
>

Florian

[Roger Critchlow]

Hello Florian --

I just pulled up to r1627 and the problem persists.

I've added a log message to src/rt_loop.erl here:

% failure detector reported dead node

on_active({crash, DeadPid}, {Neighbors, OldRT, TriggerState}) ->

log:log(info, "[RT ~p] filter dead pid ~w", [self(), DeadPid]),

    NewRT = ?RT:filter_dead_node(OldRT, DeadPid),

    ?RT:check(OldRT, NewRT, Neighbors, false),

    new_state(Neighbors, NewRT, TriggerState);

The  test is to start a 5 node ring in a single vm and send a kill to one of the nodes.  Then poll the ring to see what's in the routing tables, and examine the log files to see what's reported.

The problem is reproducible in that the dead node always remains in the routing tables.  Ah, correction:  the dead node was completely removed in 1 test run out of 7.

The number of times the log message appears differs between runs.  I've seen 0, 2, 3, and 4 calls.

I've also seen the appended stack trace, which happens in the ?RT:check called above.

-- rec --

[Nico Kruber]

up until r1630, a {die} message only killed the dht_node process. its
sup_dht_node_core supervisor then probably re-created all its children.
The problem is that, among others, the routing table process does not belong
to this supervisor and does not get re-started and a running rt_chord does not
handle the activate_rt message (as seen in your first message) telling it the
new node id.

Processes related to the node you send the {die} to were thus totally
unsynchronised and had different opinions about the own dht_node's pid (and
possibly more).
The routing table thus still propagated an old dht_node pid. It should have
been removed by the failure detector though.

The errors from the ?RT:check you've seen are probably happening in the rt
process of the killed node until a new dht_node has been started.

Could you test with r1630 again? there, a {die} message completely kills the
dht_node and all depending processes in an ungraceful way (use {leave} for
graceful leave).

Nico

> > Florian

[Roger Critchlow]

Nico --

Thank you much, that cures the problem for me.

It does change the semantics of {die} in that the system does not try to resurrect the node that dies, but that's fine by me.  Maintaining the node coverage is probably best dealt with separately.

It also leaves two other messages in src/dht_node.erl, {churn} and {kill}, which still have the old semantics and the zombie routing table entries bug.

Thank you for dealing with this so quickly,

-- rec --