Beanstalk Relaying

[Tahir Hashmi]

Hi

I want to deploy beanstalk in an environment wherein I want a set of
apache servers to act as producers and one beanstalk instance on a non-
webserver machine as a consumer. Currently, the producers on each
webserver directly connect to the remote consumer.

I would like to have a setup where all the producers accumulate their
messages in a local beanstalk instance specific to each webserver and
that instance forwards the messages to the remote consumer. This would
save the producers from having to wait on the time taken to set up a
TCP connection to the remote consumer and send the message. It'll also
reduce the number of connections to the consumer since all the
connections from producers on a single webserver would be multiplexed
on to one webserver-consumer connection.

My questions is, is there a way to run beanstalkd in a relay mode such
that it forwards all incoming messages to a different consumer? Is
there a way to also specify that the relay should use a persistent TCP
connection to the remote consumer?

Thanks

--
Tahir Hashmi

[Keith Rarick]

On Tue, Jul 21, 2009 at 12:59 AM, Tahir Hashmi<tnha...@gmail.com> wrote:
> I would like to have a setup where all the producers accumulate their
> messages in a local beanstalk instance specific to each webserver and
> that instance forwards the messages to the remote consumer. This would
> save the producers from having to wait on the time taken to set up a
> TCP connection to the remote consumer and send the message. It'll also
> reduce the number of connections to the consumer since all the
> connections from producers on a single webserver would be multiplexed
> on to one webserver-consumer connection.

I think you'd be better off trying to get your apache instances to
maintain a persistent TCP connection to the central beanstalkd
process. This would give you all the benefits you described above, and
it would be simpler once it is set up.

Beanstalkd will be fine with tens of thousands of open connections.

> My questions is, is there a way to run beanstalkd in a relay mode such
> that it forwards all incoming messages to a different consumer? Is
> there a way to also specify that the relay should use a persistent TCP
> connection to the remote consumer?

Beanstalkd itself doesn't have that feature; it's usually not
necessary (see above), and better handled by a client. I don't know of
any clients designed to do this, but it wouldn't be too hard to write
one. The only tricky part would be finding newly-created tubes.

I made an issue on github to track this:

http://github.com/kr/beanstalkd/issues/#issue/13

If anyone wants to do this, go ahead and comment on the issue or tag
it or whatever. I won't be able to get to it for a while.

kr