worker.childopts doesn't work

Min Zhou

unread,

Dec 30, 2011, 2:03:34 AM12/30/11

to storm...@googlegroups.com

Hi all,

I set the worker.childopts in my topology driver class like below

Config conf = new Config();

conf.put(Config.WORKER_CHILDOPTS, "-Xmx768m -Xms768m -XX:MaxPermSize=256m -XX:+UseParallelGC");

and submitted the topology. The worker log showed that child.opts is correct.

2011-12-30 14:49:32 worker [INFO] Worker has topology config {"storm.id" "topn-141-1325227770", "topology.fall.back.on.java.serialization" true, "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false, "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true, "nimbus.monitor.freq.secs" 10, "java.library.path" "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64:/lib64:/usr/local/lib64", "storm.local.dir" "/disk1/storm", "supervisor.worker.start.timeout.secs" 240, "nimbus.host" "dw26.kgb.sqa.cm4", "storm.zookeeper.port" 2181, "storm.zookeeper.root" "/storm", "supervisor.enable" true, "storm.zookeeper.servers" ["dw94.kgb.sqa.cm4"], "worker.childopts" "-Xmx768m -Xms768m -XX:MaxPermSize=256m -XX:+UseParallelGC", "supervisor.heartbeat.frequency.secs" 5, "drpc.port" 3772, "supervisor.monitor.frequency.secs" 3, "task.heartbeat.frequency.secs" 3, "topology.max.spout.pending" 1024, "supervisor.slots.ports" [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs" 240, "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil, "topology.message.timeout.secs" 60, "task.refresh.poll.secs" 10, "topology.workers" 19, "supervisor.childopts" "-Xmx1024m", "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05, "worker.heartbeat.frequency.secs" 1, "nimbus.task.timeout.secs" 30, "zmq.threads" 1, "topology.state.synchronization.timeout.secs" 60, "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs" 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts" "-Xmx1024m", "topology.ackers" 2, "storm.cluster.mode" "distributed", "topology.optimize" true, "topology.max.task.parallelism" nil}

2011-12-30 14:49:32 worker [INFO] Worker 5c211114-8c4e-405d-8604-31a24ded4de8 for storm topn-141-1325227770 on 86f82bde-eee2-4e8d-92a8-ec3746769c67:6700 has finished loading

but worker process didn't take the child options when starting.

$ ps aux | grep 21603

Can any help me with this?

Thanks,

Min

--
My research interests are distributed systems, parallel computing and bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Min Zhou

unread,

Dec 30, 2011, 2:13:44 AM12/30/11

to storm...@googlegroups.com

It seems that worker.childopts isn't a per-topology option, but a per-cluster option. Supervisor didn't care about that option and lanuched the worker like this way:

Haitao Yao

unread,

Dec 30, 2011, 2:37:50 AM12/30/11

to storm...@googlegroups.com

I think the worker jvm options is decided by the ${storm_home}/conf/storm.yaml

here's the code to launch the worker:

(defmethod launch-worker

:distributed [conf shared-context storm-id supervisor-id port worker-id worker-thread-pids-atom]

(let [stormroot (supervisor-stormdist-root conf storm-id)

stormjar (supervisor-stormjar-path stormroot)

classpath (add-to-classpath (current-classpath) [stormjar])

childopts (conf WORKER-CHILDOPTS)

logfilename (str "worker-" port ".log")

command (str "java -server " childopts

" -Djava.library.path=" (conf JAVA-LIBRARY-PATH)

" -Dlogfile.name=" logfilename

" -Dlog4j.configuration=storm.log.properties"

" -cp " classpath " backtype.storm.daemon.worker "

storm-id " " supervisor-id " " port " " worker-id)]

(launch-process command)

))

Min Zhou

unread,

Dec 30, 2011, 2:42:58 AM12/30/11

to storm...@googlegroups.com

The code doesn't show where the conf is from, which is the key for this issue.

2011/12/30 Haitao Yao <yao....@gmail.com>

Min Zhou

unread,

Dec 30, 2011, 2:52:12 AM12/30/11

to storm...@googlegroups.com

All right. I did a test minutes ago, adding that options to supervisor's storm.yaml , and the result showed confirmed that Haitao is right.

However, I don't think that is practical for users since each topology would with different memory requirements and other options. Setting such options unifiedly on the supervisor side doest make sense to users who wanna make a difference between their topologies.

2011/12/30 Min Zhou <code...@gmail.com>

songhe yang

unread,

Dec 30, 2011, 2:52:22 AM12/30/11

to storm-user

You can configure the jvm opts in ./conf/storm.yaml, restart the
nimbus and supervisor, then the worker will have the correct jvm opts.

Try it.

BTW: I don't why the statement "conf.put(Config.WORKER_CHILDOPTS, "-
Xmx768m -Xms768m -XX:MaxPermSize=256m
-XX:+UseParallelGC"); " is not worked.

On Dec 30, 3:42 pm, Min Zhou <coderp...@gmail.com> wrote:
> The code doesn't show where the conf is from, which is the key for this
> issue.
>

> 2011/12/30 Haitao Yao <yao.e...@gmail.com>

>
>
>
>
>
>
>
>
>
>
>
> > I think the worker jvm options is decided by the
> > ${storm_home}/conf/storm.yaml
>
> > here's the code to launch the worker:
>
> > (defmethod launch-worker
> > :distributed [conf shared-context storm-id supervisor-id port
> > worker-id worker-thread-pids-atom]
> > (let [stormroot (supervisor-stormdist-root conf storm-id)
> > stormjar (supervisor-stormjar-path stormroot)
> > classpath (add-to-classpath (current-classpath) [stormjar])

> > childopts (conf *WORKER-CHILDOPTS*)

> > logfilename (str "worker-" port ".log")
> > command (str "java -server " childopts

> > " -Djava.library.path=" (conf *JAVA-LIBRARY-PATH*)

> > " -Dlogfile.name=" logfilename
> > " -Dlog4j.configuration=storm.log.properties"
> > " -cp " classpath " backtype.storm.daemon.worker "
> > storm-id " " supervisor-id " " port " " worker-id)]
> > (launch-process command)
> > ))
>
> > 在 2011-12-30，下午3:03， Min Zhou 写道：
>
> > Hi all,
>
> > I set the worker.childopts in my topology driver class like below
>
> > Config conf = new Config();
> > conf.put(Config.WORKER_CHILDOPTS, "-Xmx768m -Xms768m -XX:MaxPermSize=256m
> > -XX:+UseParallelGC");
>
> > and submitted the topology. The worker log showed that child.opts is
> > correct.
>
> > 2011-12-30 14:49:32 worker [INFO] Worker has topology config {"storm.id"
> > "topn-141-1325227770", "topology.fall.back.on.java.serialization" true,
> > "zmq.linger.millis" 5000, "topology.skip.missing.kryo.registrations" false,
> > "storm.zookeeper.session.timeout" 20000, "nimbus.reassign" true,
> > "nimbus.monitor.freq.secs" 10, "java.library.path"
> > "/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64:/lib64:/usr/local/lib64" ,
> > "storm.local.dir" "/disk1/storm", "supervisor.worker.start.timeout.secs"
> > 240, "nimbus.host" "dw26.kgb.sqa.cm4", "storm.zookeeper.port" 2181,
> > "storm.zookeeper.root" "/storm", "supervisor.enable" true,

> > "storm.zookeeper.servers" ["dw94.kgb.sqa.cm4"], "*worker.childopts*" "*-Xmx768m
> > -Xms768m -XX:MaxPermSize=256m -XX:+UseParallelGC*",

> > "supervisor.heartbeat.frequency.secs" 5, "drpc.port" 3772,
> > "supervisor.monitor.frequency.secs" 3, "task.heartbeat.frequency.secs" 3,
> > "topology.max.spout.pending" 1024, "supervisor.slots.ports" [6700 6701 6702
> > 6703], "topology.debug" false, "nimbus.task.launch.secs" 240,
> > "nimbus.supervisor.timeout.secs" 60, "topology.kryo.register" nil,
> > "topology.message.timeout.secs" 60, "task.refresh.poll.secs" 10,
> > "topology.workers" 19, "supervisor.childopts" "-Xmx1024m",
> > "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
> > "worker.heartbeat.frequency.secs" 1, "nimbus.task.timeout.secs" 30,
> > "zmq.threads" 1, "topology.state.synchronization.timeout.secs" 60,
> > "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
> > 600, "storm.local.mode.zmq" false, "ui.port" 8080, "nimbus.childopts"
> > "-Xmx1024m", "topology.ackers" 2, "storm.cluster.mode" "distributed",
> > "topology.optimize" true, "topology.max.task.parallelism" nil}
> > 2011-12-30 14:49:32 worker [INFO] Worker
> > 5c211114-8c4e-405d-8604-31a24ded4de8 for storm topn-141-1325227770 on
> > 86f82bde-eee2-4e8d-92a8-ec3746769c67:6700 has finished loading
>
> > but worker process didn't take the child options when starting.
>
> > $ ps aux | grep 21603
> > 50957 21603 149 0.7 1262992 193552 ? Sl 14:49 1:08 java
> > -server -Xmx768m
> > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/lib64:/lib6 4:/usr/local/lib64
> > -Dlogfile.name=worker-6700.log -Dlog4j.configuration=storm.log.properties
> > -cp /home/

Min Zhou

unread,

Dec 30, 2011, 2:56:45 AM12/30/11

to storm...@googlegroups.com

BTW: I don't why the statement "conf.put(Config.WORKER_CHILDOPTS, "-
Xmx768m -Xms768m -XX:MaxPermSize=256m
-XX:+UseParallelGC"); " is not worked.

Apparently, supervisors lanuch workers following their local conf, but not from the conf of topology.

see this code of supervisor.clj

(defn -main []

(let [conf (read-storm-config)]

...

(mk-supervisor conf nil)))

Haitao Yao

unread,

Dec 30, 2011, 3:11:31 AM12/30/11

to storm...@googlegroups.com

configuring the jvm options for the echo topology seems to be a practical way, but there's another thing to consider: the capacity planning for the supervisor machine is done before the supervisor process starts(configure how many slots can be used for the supervisor), if the worker is distributed to the supervisor machine with an unknown memory setting, maybe it will break the capacity planning for the supervisor and exceed the physical constraint of the supervisor.

so, I think Nathan implement it in a reserved and simple way.

To implement this feature, I think the work load distribution logic should be redesigned and implemented: get the available memory of every supervisor server to adjust the max worker allowed on the supervisor.

BTW, the worker memory settings for the Hadoop is also very reserved and can not change.

Do we really need this feature?

songhe yang

unread,

Dec 30, 2011, 3:18:53 AM12/30/11

to storm-user

Sounds reasonable!

On Dec 30, 4:11 pm, Haitao Yao <yao.e...@gmail.com> wrote:
> configuring the jvm options for the echo topology seems to be a practical way, but there's another thing to consider: the capacity planning for the supervisor machine is done before the supervisor process starts(configure how many slots can be used for the supervisor), if the worker is distributed to the supervisor machine with an unknown memory setting, maybe it will break the capacity planning for the supervisor and exceed the physical constraint of the supervisor.
>
> so, I think Nathan implement it in a reserved and simple way.
> To implement this feature, I think the work load distribution logic should be redesigned and implemented: get the available memory of every supervisor server to adjust the max worker allowed on the supervisor.
> BTW, the worker memory settings for the Hadoop is also very reserved and can not change.
> Do we really need this feature?
>
> 在 2011-12-30，下午3:52， Min Zhou 写道：
>
>
>
>
>
>
>
> > All right. I did a test minutes ago, adding that options to supervisor's storm.yaml , and the result showed confirmed that Haitao is right.
>
> > However, I don't think that is practical for users since each top
> > ology would with different memory requirements and other options. Setting such options unifiedly on the supervisor side doest make sense to users who wanna make a difference between their topologies.
>

> > 2011/12/30 Min Zhou <coderp...@gmail.com>

> > The code doesn't show where the conf is from, which is the key for this issue.
>

> > 2011/12/30 Haitao Yao <yao.e...@gmail.com>

Min Zhou

unread,

Dec 30, 2011, 3:24:01 AM12/30/11

to storm...@googlegroups.com

BTW, the worker memory settings for the Hadoop is also very reserved and can not change.

I don't think so. We usually define memory requirements and other things like gc strategy for each hadoop job in the production environment. And it works fine for us.

Actually, however, I agree with you on the point of resource management. We need yarn or mesos for the function, and predictably there should be a lot of works on refactoring storm.

Haitao Yao

unread,

Dec 30, 2011, 3:50:18 AM12/30/11

to storm...@googlegroups.com

I don't think so. We usually define memory requirements and other things like gc strategy for each hadoop job in the production environment. And it works fine for us.

I mean we can not change the task process memory settings dynamically in the programming way while coding the mr jobs, sorry for the misunderstanding.

Tune the gc strategy for echo topology is a reasonable request, i think.

But there's another work around for this: use different storm cluster for different types of jobs with your optimized JVM options , if your data is big enough .

Comparing to use a shared big storm cluster, only more nimbus processes are needed.

Nathan Marz

unread,

Dec 30, 2011, 6:33:16 AM12/30/11

to storm...@googlegroups.com

Yes, Storm on Mesos/YARN will give the capabilities for fine-grained resource allocation. That's for the future; for now it makes more sense to keep all workers on a cluster homogeneous in terms of resources.

I agree that it would be useful to have topology specific worker java opts, so I've opened up an issue for this here: https://github.com/nathanmarz/storm/issues/88

--
Twitter: @nathanmarz
http://nathanmarz.com

Reply all

Reply to author

Forward