Submitting a jar with external config

[paces...@gmail.com]

Is there any way to submit a topology jar along with external config (containing things such as environment related credentials) or does all config need to be bundled in the jar? How do other deal with deploying topologies with things such as environment specific credentials?

Thanks,

Josh

[paces...@gmail.com]

To add onto this - if a particular bolt is run in a worker and that bolt requires configuration (db credentials for example) what is the best way to provide those to the bolt, and to package them?

- Josh

[Homer Strong]

I pass most of those types of configuration variables through the -c option to `storm jar`, e.g.

$ storm jar target/storm-starter-standalone.jar storm.starter.clj.core word-count -c nimbus.host=localhost

but you can also pass whatever other custom configurations you want, like

$ storm jar target/storm-starter-standalone.jar storm.starter.clj.core word-count -c secret.database.password=foo

and you could probably pass environment variables as your configured values. When `storm jar` runs it should print the configuration for you.

Then all components in your topology have access to the configuration map.

[Christopher Burke]

I've been wondering about this for best practices as well.

For what its worth, I have been passing in the name of a properties file as an argument of the topology. There's also a default version that I package with the jar in case there is not a customized version to specify on the command line. For example:

    storm jar my.jar my.Topology topo-name config-file-name

I then read in the properties and add them to the Config object so they are available to whatever portion of the topology might need them (spout, bolt, etc).

[Viral Bajaria]

+1 to Christopher's approach. I follow a similar pattern. Package a file with defaults but with the capability to overwrite by passing another file through the command line.

-Viral

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[paces...@gmail.com]

I like this.

Thanks for the responses, Christpher, Homer and Viral.

[Adam Phelps]

I've been using a combo of the two suggestions - I parse command line arguments and then within the options parser set values in the storm config for anything that will be needed by individual workers rather than just when launching the topology.

- Adam

--

[Casey Sybrandy]

This may be a dumb question, but when is the config file read by the jar file you create?  My concern was that a properties file wouldn't work since the bolts are distributed and across a cluster, therefore unless the file is packaged in the jar file or present in the exact same location on each box, it wouldn't work.  However, from what I've read, the code to set up and deploy the topology is run on the local machine.  Is that correct?

Oh, and I wish this was documented on the Storm wiki.