Re: high availability using a pool of daemons

[Chad Kouse]

I'm not sure about like 90% of what you said but we present our beanstalkd pool to our application as a single point of entry maintained by a set of haproxy's. works pretty well and is simple enough. 

--
Chad Kouse

On Wednesday, October 17, 2012 at 12:43 PM, B. Estrade wrote:

I am working on a project that needs a job queue, and beanstalkd is looking very attractive. The thing is that we really want to make the application very highly available, and the fact that federation of beanstalkd is left to the client (like memcached) has lead me to consider the following approach.  I'd like to create a resource manager that maintains a pool of daemons, but presents to the client application the view as a single and highly available process that is managing these requests.

I would like to basically create a resource manager that provides failover and redundancy via a pool of distributed beanstalkd instances.  The application, however, will view things logically as a single logical instance of the queue.  It looks like that the way to go would be to manage the pool of beanstalkd instances using Paxos.

My primary question after reading the documentation is, how would one use the existing features (as atomic primitives, so to speak) of beanstalkd to create a resource manager that provides the isolation guarantees and atomic changes to the state of a queue?  As far as I can tell, this would be possible if a) there was a way to provide a temporary change to the queue (e.g., a job or set of jobs) and b) a way to make the temporary changes visible in an atomic way ( I suppose a daemon would also need to know if it's OK to make the commit, too).  I don't think that these needs are unique the to distributed commit protocol since this would be required by each daemon individually to facilitate the protocol itself.  Leader elect, I think, can be handled outside of the daemon.

I would like to provide strong consistency across all beanstalkd instances in the pool; I am not interested, really, in simply master/slave replication (which I suppose one could facilitate via the bin log of a master). I am interested in seeing if it is possible to implement a Paxos like protocol among all daemons in the pool using beanstalkd's current capabilities.  Weak or eventual consistency is also an option, but I'd like to see how possible the ideal is at the onset. 

Thanks!

Brett 

--
You received this message because you are subscribed to the Google Groups "beanstalk-talk" group.
To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/wnNdSeBC_GkJ.
To post to this group, send email to beansta...@googlegroups.com.
To unsubscribe from this group, send email to beanstalk-tal...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.

[Dave Gardner]

Highly available and redundant (and paxos) suggests to me that you have to write to more than one queue. What are you planning to do on consumption? 

Sent from my iPhone

[B. Estrade]

On Wednesday, October 17, 2012 3:17:38 PM UTC-5, Dave Gardner wrote:

Highly available and redundant (and paxos) suggests to me that you have to write to more than one queue. What are you planning to do on consumption? 

This is what I have to figure out, i.e., are there a series of things I can do to provide the isolation (temporary add to all), then the atomic switch to make it visible (durable) upon distributed consensus being met.

Basically, I want to have a single logical queue that is supported by a set of many queues that can fail or go away, but the logical queue is always there and consistent.

Thanks,
Brett
 

[B. Estrade]

On Wednesday, October 17, 2012 2:50:28 PM UTC-5, chadkouse wrote:

I'm not sure about like 90% of what you said but we present our beanstalkd pool to our application as a single point of entry maintained by a set of haproxy's. works pretty well and is simple enough. 

Thanks, I will look at haproxy.

Cheers,
Brett
 

[Andre Ruiz]

Make them failover ciclycally, only one active at a time.

Andre

> https://groups.google.com/d/msg/beanstalk-talk/-/af9VMKZ8oEMJ.

>
> To post to this group, send email to beansta...@googlegroups.com.
> To unsubscribe from this group, send email to
> beanstalk-tal...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/beanstalk-talk?hl=en.

--
Andre Ruiz <andre...@gmail.com>
Curitiba, PR, Brasil
Tel +55 (41) 8407-3847

[Chad Kouse]

Sure, from our ops team to you:

listen beanstalk-production 100.100.100.100:11300

mode tcp

option abortonclose

option log-separate-errors

maxconn 4415

option httpchk

balance roundrobin

timeout client 120s

timeout connect 120s

timeout server 120s

server bns4.shared.day1 172.0.0.4:11300 check port 9200 inter 10s fall 2 rise 5

server bns5.shared.day1 172.0.0.5:11300 check port 9200 inter 10s fall 2 rise 5

server bns6.shared.day1 172.0.0.6:11300 check port 9200 inter 10s fall 2 rise 5

for port 9200, we have a script running on each host under xinetd, that gives an HTTP 200 if the host is healthy, and a 304 if not.

-- chad

On Monday, January 7, 2013 at 9:00 PM, Drew Broadley wrote:

Chad, are you able to provide your haproxy configuration for beanstalkd in high availability mode ?

To view this discussion on the web visit https://groups.google.com/d/msg/beanstalk-talk/-/w25lHXeg7LcJ.

[Drew Broadley]

Thanks Chad, appreciate it.

Cheers,

Drew Broadley

Sent from phone

[Chad Kouse]

Consumers connect directly to individual beanstalks servers.  We have our consumers just consume from 1 beanstalkd server.  If a beanstalkd server is going to be down for a long time it's simple to point all of it's consumers to another beanstalkd server instead, but to be honest that's never happened.

in other words say we have bns4, bns5, bns6 beanstalkd servers and we have cns1, cns2, cns3 consumers

cns1 would connect to bns4, cns2 to bns5, cns3 to bns6, etc

pretty low-budget solution but works fairly well so far.

-- 

Chad Kouse

On Saturday, February 9, 2013 at 3:42 PM, News Group wrote:

How do you deal with the worker connection to the multiple beanstalk instances?

To unsubscribe from this group and stop receiving emails from it, send an email to beanstalk-tal...@googlegroups.com.

To post to this group, send email to beansta...@googlegroups.com.

Visit this group at http://groups.google.com/group/beanstalk-talk?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.