Cyclic topology getting stuck / dying

[Justin Kaeser]

I have a simple cyclic topology as specified below.

WaterSpout drops raindrops, FountainBolt consumes them, emits back to the fountain, and after the third use, discards into the GullyBolt.

After running for a very short time, perhaps a few thousand spout emits, the topology simply hangs, while only about 3 raindrops even made it into the gully. 

If I increase the number of fountain bolts, more tuples make it through the topology, but it still hangs quickly. With a large number, it even crashes completely (goes back to command line) without any failure notice.

Any idea what is happening here? Thanks for any pointers.

Topology definition:

    val builder = new TopologyBuilder

    builder.setSpout("rain", WaterSpout, 1)

    builder.setBolt("fountain", FountainBolt, 1)

    .shuffleGrouping("rain")

    .shuffleGrouping("fountain", "reuse")

    builder.setBolt("gully", GullyBolt, 1)

    .shuffleGrouping("fountain","discard")

    

    val topology = builder.createTopology()

    

    val conf = new Config

    conf setDebug(true)

    conf setNumAckers 0

    new LocalCluster().submitTopology("rainyday", conf, topology)

[Justin Kaeser]

I am still having the problem of hanging cyclic topologies. I reproduced the topology in plain Java (previously in Scala), but the result is mostly the same.

Here is the complete topology implementation: http://pastebin.com/JjSuN3kD

It would be really great if anybody knew what might be happening here.

[Michael Rose]

Running on Storm 0.7.2,

Jan 19, 2013 5:21:03 PM test.CyclicTopology$GullyBolt execute

INFO: drop 71000

seemed to work for me, but 0.8.x gutted how components talk to each other.

Running on Storm 0.8.2 (w/ no modifications)

Jan 19, 2013 5:24:26 PM test.CyclicTopology$GullyBolt execute

INFO: drop 0

Doesn't work, pauses after about 700 emitted tuples. The spout thread is sleeping on the SpoutSleepWaitStrategy and never wakes up again.

Doing a little more digging, it looks like the spout is running a little too fast. I'm not sure why it never recovers, that might be a question for Nathan (until I have time to dig a little deeper on how it's invoked).

By setting:

conf.setMaxSpoutPending(50); // 50 can be about anything, 500 worked quite well for me too.

Or by adding a short courtesy sleep into the spout's nextTuple method (I used 2ms) the topology continues to process messages.

maxSpoutPending is always a good idea to bound the memory usage of your topologies and not overrun buffers. :) I'd recommend that over sleeping.

-- 

Michael Rose (@Xorlev)

Senior Platform Engineer, FullContact
mic...@fullcontact.com

[Justin Kaeser]

Thanks for reproducing!

I had forgotten to mention that I also ran this on 0.8.2 with these results. 

Meanwhile, I had tried adding a courtesy sleep, and indeed, with even only 1ms sleep in the spout the topology happily chugs along. Of course, handling a peak rate of less than 1k/s may delay processing of messages more than desired.

Thanks for the maxSpoutPending hint, I'll try that as well.

[Michael Rose]

You can always scale spouts with parallelism (same as bolts) :) I'd recommend just tuning your maxSpoutPending as necessary.

Not sent from my iPhone

[Sumant Sankaran]

Hi,
Nathan had fixed a bug where in the topology hangs because of the
buffer getting filled up .
https://github.com/nathanmarz/storm/commit/1a9dca46abe4c937e6b5874a9d1b178163a95af4


Original discussion here
https://groups.google.com/forum/#!topic/storm-user/c1g_s5L8yuI/overview

Your case seems to be similar ,so to verify I just added a sleep of
1sec in your spout nextTuple method Utils.sleep(1000);
I see it progressing now .If you are using storm 0.8.2 ,this should
not occur.

Thanks
Sumant

PS:Seems weird but google keeps deleting my stuff,so if this appears
multiple times ,I shall delete it.

[Nathan Marz]

Hi Justin,

This is currently a limitation of Storm 0.8.0 – if the buffers in a cyclic topology fill up, it's possible for it to deadlock (essentially the dining philosophers problem). 

You certainly need to throttle the spout so that they don't get filled up, but it would be nice for Storm to automatically increase the size of the buffers so that they eventually get big enough for a cyclic topology. This would be good for future work, though it does conflict with our desire to also have Storm use filled up buffers as a mechanism for providing backpressure back to the spout. So we'll need to figure out the right way to resolve it (possibly only making the auto-resize buffers the ones on the "back link" in the cycle).

Prior to Storm 0.8.0, Storm didn't use bounded buffers which is why it worked in 0.7.2.

-Nathan

--
Twitter: @nathanmarz
http://nathanmarz.com

[Justin Kaeser]

Hi Sumant, Nathan,

thanks for the reply. Do I understand correctly that this problem is related to the one fixed in 0.8.2, but somehow not affected by the fix?

I don't quite understand the Storm internals enough to understand where and what buffers get filled up. Would you have any link explaining this?

Either way, setting the maxSpoutPending seems to solve the problem for now without slowing the processing unnecessarily, so thanks alot!

[Nathan Marz]

No, nothing is changed with this issue in 0.8.2. As long as you ensure the intermediate buffers involved in the cycle don't fill up, then things will work fine (which maxSpoutPending can do for you).