passing Java system properties

[vinh]

Is there a way to pass Java system properties to Storm when deploying a topology?

For example, like:

storm jar MyTopologies.jar com.acme.Topology Topology -Dkey=value

Then in my bolt, I would get the property value via System.getProperty("key").

Since the value is dynamic and varies by the topology deployed, I cannot set it in storm.yaml as part of worker.childopts.

Thanks,

-Vinh

[Jon]

You can set things in the constructor of your bolts, and they'll be serialized before being deployed.

I'm not entirely sure, but I would imagine there is some way to pass things to your main method, and then pass it to bolt constructors...

[SKG]

Is it jvm specific or topology(application) specific?

If jvm then edit the worker.childopts.

If topology specific then pass it in command line argument like map object (K:V)
and in the main function extract this command line argument and update the configuration which will passed downstream.

For values use the data types which are serializable by storm.

[vince]

as last mail say，if for jvm，edit woker.childops.

if for topology there is 3 method to specific.

 1. in you java code set the config "conf,put(key,value)"

 2. add config in ~/.storm/storm.yaml  ,"key: value"

 3. when you submit topology add -c param as" storm jar xxx.jar mainclass -c key=value

seems method 3  match your case.

发自我的 iPad

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[vinh]

Thanks all.  Here's the use case.  I have an external properties file, but it differs by topology.  The contained properties are used by all spouts and bolts.  So, I have a singleton registry class that loads this file, and it gets the file name/path via System.getProperty().  The registry class has no knowledge or dependency on Storm.  It's just a library.

Method #3 below is closer to my use case.  But with one problem.  Storm doesn't serialize singleton objects.  So, additional plumbing code is required in all spouts/bolts to fetch the key=value from the component config and explicitly pass it to the singleton.  Or as long as spouts are created first by Storm, the code only needs to be added to the spouts.  Overall though, this means that the properties file is now managed in multiple code paths, instead of just in the singleton registry class.

I think what would be good is that if I pass system arguments to Storm as "-Dkey=value", that Storm passes those on to the workers and makes them available there, too, by populating the system properties cache via System.setProperty(…).  Then my code will work as-is when it calls System.getProperty(…).

Or perhaps I'm making this too complicated?

[SKG]

Why can't u pass the topology specific properties file as command line argument and open the file in the main method and
update the storm configuration by conf,put(key,value).

[S Melody]

The master HEAD of the storm python script that is executed with the jar command has the ability to specify system properties that are passed to the workers.  You could copy and make changes to your bin/storm file to pass the options along.  We're running a forked copy from our deployments in jenkins to do something very similar to what you're trying to do.

Look at:

https://github.com/nathanmarz/storm/blob/master/bin/storm

    exec_storm_class(
        klass,
        jvmtype="-client",
        extrajars=[jarfile, USER_CONF_DIR, STORM_DIR + "/bin"],
        args=args,
        jvmopts=[JAR_JVM_OPTS + " -Dstorm.jar=" + jarfile])

[vinh]

Thanks all.

Sean, is this script compatible with 0.8.2?

[S Melody]

I can't say from experience, but I would assume taking the 0.8.2 launcher and following the same pattern should work.

Sean