ACKing drops TPS to a 1/4

[Rafael Mahnovetsky]

I'm finding that with a bare bones topology acking reduces TPS to less then a 1/4. Is this a problem with my config or this is the performance hit for having reliability in storm?

Here is the code, nothing special pretty much just barebones storm, one spout and one bolt: 

topology: https://github.com/rmahnovetsky/storm-perf-test/blob/master/src/main/java/com/rpm/storm/payment/topology/SpoutTestTopology2.java

bolt: https://github.com/rmahnovetsky/storm-perf-test/blob/master/src/main/java/com/rpm/storm/payment/bolt/AckBolt.java

With acking executor set to 0

Storm UI: https://www.evernote.com/shard/s225/sh/26ea6148-36e8-4e0a-bea8-e5c17c7f4ec1/725ea19cc775925965f743570fac026e

CPU/MEM: (there are 4 executors only one shown): https://www.evernote.com/shard/s225/sh/d58b22c5-5d43-4788-b993-8a0e641d07f3/1e7cebc0342b52d22969ed2590be82ae

CPU samples: https://www.evernote.com/shard/s225/sh/66c5e1a8-8a47-4873-acaf-6492f80a9769/da2236599aab6d054461b3a2c0306678

TPS = 36k/s

With acking 4 executors

Storm UI: https://www.evernote.com/shard/s225/sh/baf5c4f9-addb-4373-9443-1a1978cd0233/d9606765197609b4f73f945c1d10d1f8

CPU/MEM: https://www.evernote.com/shard/s225/sh/fa13a7b8-d8c1-4ba6-9cd6-90f31d64a130/78e81e10883a60e5ef86cc026696c6de

CPU samples: https://www.evernote.com/shard/s225/sh/ffb1dd51-e963-46c9-ae06-8812aa8bb138/a04a2b725500ae2e92c10806ab8d0470

TPS 8.8k

This test was run on a mac mini with 16gig of ram intel quad core i7 @ 2gig

 

[John Pountain]

That is a big drop in performance. I have noticed similar and performance seems to be worse on EC2 instances as well.

I am also struggling to determine if it is config related or whether it is the acking code in Storm. This is no where near the TPS that various people claim to get using Storm but perhaps they don't have acking or use small messages and lots of boxes. 

Has anyone else encountered this and better still, is there a way to speed things up?

[Bruno D. Rodrigues]

If you’d like more values for comparison (thanks for your values), my PoC:

- source json with average 300B per line, read at max 600MB/sec

- spout with a separate thread and an internal queue to split the reading of the file and the nextTupple emit

- bolt that just counts messages and bytes.

the best I can reach are around 50K msg/sec and 20MB/sec (160Mbps). It obviously degrades if more machines are involved due to the serialization and network. 

Without the thread and queue I was getting:

- reading one line and emitting, even with a buffered reader, and it went down to KB/sec

- reading lines and emitting on a loop (as I’m not expecting acks), would go up to 300MB/sec, but die with out of memory in about 5 seconds. As the memory was allocated inside zeromq, manually, it is quite a funny error because GC can’t do anything about it.

- Hence the middle ground of the thread concentrated on the IO and put on queue, and the nexttupple just taking out of it.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Atul Dambalkar]

I have noticed a similar drop in performance when Ack'ing is enabled. I have seen performance going down between 1/4th to 1/8th. 

I am curious to know has to why this should happen. Is Ack'ing a blocking or non-blocking call? I hope it's non-blocking with underlying async zeromq messaging infrastructure. And assuming it's a non-blocking call, are we saying that it's only the network traffic congestion that really causes this big drop in performance. It will be good to know what's going on under the hood?

-Atul