Re: Nginx Logging to Zeromq Module - Sparkngin

[Steve Morin]

Thanks W-Mark

[1] I have been looking at that module and a couple others including https://github.com/FRiCKLE/ngx_zeromq

[2] My understanding is that nginx uses a number of worker process that are isolated, but need to see how zeromq handles that and if a socket can me shared.  Given that they aren't thread safe I assume that is a no.  I will pay attention to this part http://zguide.zeromq.org/page:all#Getting-the-Context-Right

[3] Saw that, will look where a setup call can happen this is helpful.

[4]  I have been reading Evan Miller's guide, thanks

* Planning on using zero-copy

* Not sure if redis would be faster under high load but that would be pretty interesting to see.

On Tue, Dec 3, 2013 at 5:49 AM, W-Mark Kubacki <wmark...@hurrikane.de> wrote:

2013/11/16 Steve Morin <st...@stevemorin.com>:

> Does anyone have experience integrating zeromq with Nginx.  I am looking for
> some pointers, to see what concerns I should look out for.
>
> I am trying to contribute this code to a open source project.
> -Steve

This seems to me being a good template for what you want to accomplish:
[1] http://www.binpress.com/app/nginx-redislog-module/998

Keep in mind that »ØMQ sockets are not threadsafe« and don't share
them between threads.
[2] http://zguide.zeromq.org/page:all#Multithreading-with-MQ

See also:
[3] http://forum.nginx.org/read.php?2,243942,243986#msg-243986 (master
process → init_module callback; don't define sockets here; zmq_init
goes here)
[4] http://www.evanmiller.org/nginx-modules-guide.html#definition

A note on performance:
• Use zero-copy wherever possible. (I would make a custom string an
optional last part of a message.) — [5]
http://zeromq.org/blog:zero-copy
• Does ØMQ have UDP or similar for PUB sockets? If so, use that.
• I guess that logging or queuing through Redis will perform better
than utilizing ØMQ for that matter iff everything is done on one
machine under high load.

--
Mark

[Steve Morin]

Piotr,

  You've worked on something similar in the past what's your opinion on the performance of a zeromq module vs redis?

-Steve

On Tue, Dec 10, 2013 at 7:00 AM, W-Mark Kubacki <wmark...@hurrikane.de> wrote:

2013/12/10 MAGNIEN, Thierry <thierry...@sfr.com>:

>
> I don't know much about Zeromq performance, but we pushed redis to about 300k commands per second on our production servers (8 cores), and benchmarks on better hardware (24 cores) show it could handle close to 500k/s.

ØMQ should come pretty close or even exceed that. You will require a
free core for its process, though.
      Nevertheless implementing it looks rather straightforward (I've
done it in a different product) and with an event consumer (Nginx
being the producer) you will get one part of a fantastic local
analytics tool, similar to Piwik.

Here's an introduction to authentication and encryption:
[6] http://hintjens.com/blog:49#toc6

Here's how you can prototype the plugin using Lua:

[7a] http://wiki.nginx.org/HttpLuaModule#init_by_lua — initialize context
[7b] http://wiki.nginx.org/HttpLuaModule#Data_Sharing_within_an_Nginx_Worker
— the place for PUB sockets
[7c] https://github.com/mtourne/nginx_log_by_lua/blob/master/conf/nginx.conf
— collect data for logging, and send a multipart message here…
[7d] https://github.com/Neopallium/lua-zmq/blob/master/examples/client_multipart.lua
— … that's how the call looks like

Everything else on the consumer will be counters of some sort. You
could still use Redis there for small sites, though.

Other noteworthy data structures for the consumer:
[8] http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/
[9] http://billmill.org/bloomfilter-tutorial/
[10] http://www.igvita.com/2010/01/06/flow-analysis-time-based-bloom-filters/

I've written this in the hope that other developers of Nginx modules
have an idea on where to start (their sockets, for example ;-) ). We
should move discussion of further details to a sparkngin group.

--
Mark

_______________________________________________
nginx-devel mailing list
nginx...@nginx.org
http://mailman.nginx.org/mailman/listinfo/nginx-devel

[Steve Morin]

Mark, Thierry and Piotr,

  Going forward I would like to move discussions like this to the Sparkngin mailinglist.  Hoping all of you can join it so we can continue the discussion there.

-Steve