Is ZMQ going to be suppoted on Comm Layer?

[Sam.Park]

Hi

I notice he comm layer supports only UDP and multicast. If I'd like
there would be no data loss on S4 System,

Using ZMQ on Comm Layer will meet this need? Is there any test
scenario to find out how may messages can be handled

on specific hardware with guaranteeing no data loss?

Regards,

Park

[Bruce Robbins]

I did run some tests with ZMQ in the comm layer. However, I was using the PUB/SUB pattern, which is non-blocking. You can send messages as fast as you want: However, if the connection's queue fills, ZMQ starts throwing away messages until a slot opens up on the queue. The test showed similar results to UDP. I ran several tests at faster and faster data rates: At some threshold, both UDP and ZMQ starting dropping events. At each event rate, the percentage dropped was nearly identical between UDP and ZMQ PUB/SUB.

Even using a non-blocking ZMQ pattern or TCP/IP directly could result in data loss. TCP/IP send is reliable and blocking: That is, you cannot send messages faster than the target can receive them. But somewhere up the line, messages are being generated based on real-world events (users clicking on links, people swiping cards, hearts beating, etc.), and those real-world events are not necessarily going to slow down to accommodate how fast your nodes can receive data. If the nodes cannot keep up, a queue somewhere is going to fill up: The legacy stream's send queue, some adapter's receive or send queue, or some S4 node's receive or send queues.

>Is there any test
>scenario  to find out how may messages can be handled
>on specific hardware with guaranteeing no data loss?

For our S4-based applications, we first ran load tests at maximum expected rates (usually by generating fake events) to ensure we had enough processing power and memory to accommodate those rates. The hardware, number of nodes, and the number of adapters needed depends on the application, since some applications might be more CPU bound, others more memory bound, and some might generate many new events on the fly.

[Frank San Miguel]

One of the problems I have is system monitoring.  In particular, if I don't know how many messages are being lost, I can't know how well my app is working.

All sorts of things could cause data loss including network problems, CPU overload, software bugs and bursty inputs.  I've never used ZMQ, but if it has comparable performance to UDP and it uses queues, then I think it has an advantage over UDP because you can actually count exactly how many messages are getting lost.

[Bruce Robbins]

Maybe that's true, but when I was performing my tests, I could not
find an ZMQ API that tells me when it's throwing away messages and how
many it's throwing away. It appears to just throw them away silently.
With UDP, I can see some of the loss via netstat.

There are some other TCP/IP network libraries that I know little about
at this point, but might provide that sort of functionality.

For a TCP/IP implementation of the comm layer, in the end we'll likely
have to implement our own queues so we can keep track of what is
getting lost and favor some types of messages over others.

[Leo Neumeyer]

The nice thing about UDP is the low overhead. It occurred to me that for debugging and testing we could log events locally and do an audit after the fact. The audit report will tell you exactly what events got lost and where. For live detailed info you will need a reliable protocol.

-leo