High Throughput HTTP gateway for rabbitmq

[Andrew Boag]

Hi all, I'm hoping there is some expertise out there on this topic.

We have an API notification gateway for a statistics platform. Basically, lots of data comes back as HTTP requests (with a farm of GET parameters - they should be POST but they aren't). Instead of putting this stuff directly into the database, we want to throw it into rabbitMQ

We have a basic PHP script that is pushing a serialised object into the queue that represents all of the parameters. The message size is small (150 chars or so) and we have got the injestion up to about 500 messages/second ... however we are sure that this can be improved.

We have a FE web server (apache + php) and a BE rabbitMQ server. With concurrent simulated load, it's the web server that drowns, the rabbitMQ server barely breaks a sweat.

I'm sure that apache + PHP is not the most exciting approach to doing this. Doesn't anyone have any suggestions for light-weight listeners to take basic HTTP parameters, serialise them and put into a queue?

The next thing I'll try is nginx + fastcgi with PHP ... but it would probably be smarter to write this is in perl / python or another scripting language before resorting to C :-)

Any help appreciated.

[Simon MacMullen]

As a general design point, HTTP is designed around the idea of opening a
connection, doing one thing, and closing it. AMQP is designed around the
idea of opening a connection, doing lots of things for a while, then
closing it.

So AMQP connections are generally more expensive than HTTP requests. You
will likely do well if you can set up a gateway that has a few long
running pooled AMQP connections to use when handling HTTP requests.

PHP is of course badly suited to doing that sort of thing. Perl / python
I don't know, might not be much better. Java or .NET might be reasonable
to look at. Or Erlang of course :-)

Cheers, Simon

> --
> You received this message because you are subscribed to the Google
> Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to rabbitmq-user...@googlegroups.com
> <mailto:rabbitmq-user...@googlegroups.com>.
> To post to this group, send email to rabbitm...@googlegroups.com
> <mailto:rabbitm...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Gavin M. Roy]

I am in the process of porting Statelessd to Erlang using cowboy and have been 

contemplating making it a plugin to make use of the direct connection.

That being said, it’s usable and performant as a stand-alone Python daemon:

https://github.com/gmr/statelessd

Gavin

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.

[Simon MacMullen]

Ah, that looks exactly like what you want. So it can be done in Python :-)

A plugin would be very awesome too mind!

Cheers, Simon

On 18/08/14 15:15, Gavin M. Roy wrote:
> I am in the process of porting Statelessd to Erlang using cowboy and
> have been
> contemplating making it a plugin to make use of the direct connection.
>
> That being said, it’s usable and performant as a stand-alone Python daemon:
>
> https://github.com/gmr/statelessd
>
> Gavin
>
> On August 18, 2014 at 10:12:01 AM, Simon MacMullen (si...@rabbitmq.com