setNumWorkers above 1 doesn't work for a DRPC topology

[Cristian Pirvu]

Hi everyone,

I'm facing an issue that I don't know how to fix: if I deploy the topology with more than 1 worker in the config, on a storm cluster (with 1 or 2 supervisors is the same behaviour) i get an error from in the worker log:

2013-03-29 12:16:27 util [ERROR] Async loop died!

org.zeromq.ZMQException: Address already in use(0x62)

        at org.zeromq.ZMQ$Socket.bind(Native Method)

        at zilch.mq$bind.invoke(mq.clj:69)

        at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:58)

        at backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1631.invoke(loader.clj:26)

        at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375)

        at clojure.lang.AFn.run(AFn.java:24)

        at java.lang.Thread.run(Thread.java:662)

Tried with versions 0.8.2, 0.8.3-wip3 and 0.9.0-wip16, the error is the same.

If i don't use setNumWorkers() and leave it to the default value of 1, everything is ok, no exceptions and the call gets through.

The topology is build using LinearDRPCTopologyBuilder, but the same happens if i use Tridend.

The number of workers I set in Config is always lower than the ones available in the supervisor.

Any idea why this happens? What do i need to change to make a DRPC topology be able to have more workers.

Many Thanks!

[xmdxcxz]

Have the supervisor.slots.ports: defined in your storm.yaml file has been used already by other processs?

you can use the command "netstat" to check it.

Thanks.

在 2013年3月29日星期五UTC+8下午8时21分29秒，Cristian Pirvu写道：

[Cristian Pirvu]

I don't believe so. For example if i set the topology with 2 workers, i see 2 ports used in netstat (7704, 7705), i've changed the default ones, trying to find solutions to the problem.

These are both created by worker process of the supervisor. They are already on LISTENING before i call the topology from DRPC.

The ports seem to be opened as soon as i deploy the topology to Storm and it's the same for 1 or 2 workers, but it only gives an error with more than 1 worker being used.

Thanks!

[陈竞]

i met the same problem, do you solve it?

2013/4/1 Cristian Pirvu <pcri...@gmail.com>

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
陈竞，中科院计算技术研究所，高性能计算机中心
Jing Chen HPCC.ICT.AC China

[Cristian Pirvu]

Unfortunately no, and it's a shame too, because everything else about Storm seems to be so nice, but this bug is a stopper for us, we can't continue using it if we can't make use of parallel processing.

--
You received this message because you are subscribed to a topic in the Google Groups "storm-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/storm-user/iny-9QHIhDw/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to storm-user+...@googlegroups.com.

[Nathan Marz]

Are you using zmq 2.1.7 or 2.1.4?

Twitter: @nathanmarz
http://nathanmarz.com

[Cristian Pirvu]

Hi Nathan,

I'm using zmq 2.1.7, should I try with 2.1.4? I know from the docs I've read that 2.1.7 is the recommended version for Storm 0.8.x.

Thanks,

Cristi

[Nathan Marz]

Yea, try 2.1.4. Also, let us know your OS information.

[Cristian Pirvu]

Thanks Nathan, that actually worked.

I'm using CentOS 6.3 x64, kernel v2.6.32.

The only problem that i can see right now is that i cannot make a DRPC call on a topology split among multiple servers in the cluster.

If the topology runs on one supervisor only, everything is ok, if i add another supervisor, the DRPC call times out.

Thanks,

Cristi

[丛梓存]

Hi celix：
     I‘m a student from sun yat-sen university. I have the same problem too, so i wonder if you have solved the problem
 
在 2013年4月16日星期二UTC+8下午9时09分51秒，celix写道：

[Suchi Amalapurapu]

Any resolution on this? We are seeing a similar issue. nimbus.log does indicate two workers but all the requests go to only one router.

it kind of defeats the parallelism on the cluster.