ACKing drops TPS to a 1/4

400 views
Skip to first unread message

Rafael Mahnovetsky

unread,
Oct 26, 2013, 8:34:39 PM10/26/13
to storm...@googlegroups.com

John Pountain

unread,
Oct 26, 2013, 9:11:43 PM10/26/13
to storm...@googlegroups.com
That is a big drop in performance. I have noticed similar and performance seems to be worse on EC2 instances as well.

I am also struggling to determine if it is config related or whether it is the acking code in Storm. This is no where near the TPS that various people claim to get using Storm but perhaps they don't have acking or use small messages and lots of boxes. 

Has anyone else encountered this and better still, is there a way to speed things up?

Bruno D. Rodrigues

unread,
Oct 27, 2013, 7:22:30 AM10/27/13
to storm...@googlegroups.com
If you’d like more values for comparison (thanks for your values), my PoC:

- source json with average 300B per line, read at max 600MB/sec
- spout with a separate thread and an internal queue to split the reading of the file and the nextTupple emit
- bolt that just counts messages and bytes.

the best I can reach are around 50K msg/sec and 20MB/sec (160Mbps). It obviously degrades if more machines are involved due to the serialization and network. 

Without the thread and queue I was getting:
- reading one line and emitting, even with a buffered reader, and it went down to KB/sec
- reading lines and emitting on a loop (as I’m not expecting acks), would go up to 300MB/sec, but die with out of memory in about 5 seconds. As the memory was allocated inside zeromq, manually, it is quite a funny error because GC can’t do anything about it.
- Hence the middle ground of the thread concentrated on the IO and put on queue, and the nexttupple just taking out of it.


--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

signature.asc

Atul Dambalkar

unread,
Dec 10, 2013, 10:52:11 PM12/10/13
to storm...@googlegroups.com, bruno.r...@litux.org
I have noticed a similar drop in performance when Ack'ing is enabled. I have seen performance going down between 1/4th to 1/8th. 

I am curious to know has to why this should happen. Is Ack'ing a blocking or non-blocking call? I hope it's non-blocking with underlying async zeromq messaging infrastructure. And assuming it's a non-blocking call, are we saying that it's only the network traffic congestion that really causes this big drop in performance. It will be good to know what's going on under the hood?

-Atul
Reply all
Reply to author
Forward
0 new messages