Serf Event Handlers (membership vs user)

[Dhananjay Kumar]

Hello,

I have a serf cluster consisting of 4 nodes and noticing a strange issue w.r.t event handlers. My config file looks as below:

cat serf.conf

{

    "replay_on_join": false,

    "leave_on_terminate": true,

    "tags": {

        "currentRole": "standby",

        "IP": "10.0.1.74"

            },

    "node_name": "nodeA",

    "start_join": [

        "10.0.1.74",    # IP of node A

        "10.0.1.75",    # IP of node B

        "10.0.1.76",    # IP of node C

        "10.0.1.77"     # IP of node D

    ],

    "interface": "eth0",

    "event_handlers": [

        "member-join,member-leave,member-failed=python /serf/membership.py",

        "user:down,user:active,user:standby=python /serf/userEventHandler.py"

    ]

}

When a node(say node D) joins the cluster, it sends a member-join to all others and everyone respond with their member-join. If an user event is triggered from one of the nodes(say C) at the same time, only userEventHandler.py gets invoked for C on D and membership.py doesn't get invoked at all. I can see that member-join has reached serf agent on D, but agent didn't invoke the member event handler script and there are no errors.

On node D:

    2017/08/01 03:08:48 [INFO] serf: EventMemberJoin: nodeA 10.0.1.74

    2017/08/01 03:08:48 [INFO] serf: EventMemberJoin: nodeB 10.0.1.75

    2017/08/01 03:08:48 [INFO] serf: EventMemberJoin: nodeC 10.0.1.76

    2017/08/01 03:08:48 [INFO] agent: Received event: user-event: custom    #This event was received from nodeC

    2017/08/01 03:08:49 [INFO] agent: Received event: user-event: custom

Is there any smartness/bug in serf to not invoke event handler for membership events if a user event handler is invoked at the same time for that node? I have noticed this quiet a few times and all the times member event handler doesn't get invoked if a user event has come at the same time from that node.

Appreciate any help in this regard.

Thanks,

Dhananjay 

[Armon Dadgar]

Hey Dhananjay,

The event handlers will get invoked sequentially, so it’s important that a handler not take too long to execute.

What might be happening is that an event handler is running for a long time and preventing the other handlers from executing.

If you need to do long running work based on a handler, it’s best to fork the process so that Serf can invoke the next handler.

Hope that helps!

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/serf/issues
IRC: #serfdom on Freenode
---
You received this message because you are subscribed to the Google Groups "Serf" group.
To unsubscribe from this group and stop receiving emails from it, send an email to serfdom+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/serfdom/b310b2bb-3a19-49ea-8caa-b546e4dbe1fc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dhananjay kumar]

Hi Armon,

I do have the forking part inside both of my eventHandler scripts. 

(taken from serf website):

out_log = file('/dev/null', 'a+')
os.dup2(out_log.fileno(), sys.stdout.fileno())
os.dup2(out_log.fileno(), sys.stderr.fileno())

But there are numerous user events coming at that time(i.e while recovering from a network partition or split brain, all old user events get replayed).  Can this cause agent to not invoke membership event handler? Is there a timer after which agent would drop/ignore calling the event handler.

Thanks,

Dhananjay

 

Virus-free. www.avast.com

To unsubscribe from this group and stop receiving emails from it, send an email to serfdom+unsubscribe@googlegroups.com.

[Armon Dadgar]

Hey Dhananjay,

That is doing the file descriptor redirection, but are you also forking the process?

With Python that would be something like “os.fork()"

Best Regards,

Armon Dadgar