how to reference parent yaml config value

[John Calcote]

I have what I feel is a fairly common use case for dropwizard yaml configuration and I can't figure out if or how dropwizard supports it. Let's say my config yaml file looks like this:

sysPaths:

    runtimeDataPath: /var/lib/dude

    staticDataPath: /usr/share/dude

server:

    connector:

      - type: my-https

        keyStorePath: keystore.jks

...

My goal here is to recognize the value of server:connector[my-https]:keyStorePath as a relative path and append it to "<runtimeDataPath>/security/" without the user having to enter "/var/lib/dude/" in front of keystore.jks (and every other relative path in the configuration file). Thus, the application administrator can change the base path of the runtime data directory with a single change. If the user specifies an absolute path in the keyStorePath setting, then it would be treated as absolute.

I cannot figure out how to reference the SysPathConfiguration object from the MyHttpsConnectorConfiguration object. I'm to the point where I'm ready to remove the sysPaths section from the config file and just hard-code defaults into the application in a non-configuration-specific POJO, then allow the user to override those values with command-line arguments. It would be nice to be able to specify those defaults in the yaml file, however, rather than hard-code them.

Thoughts? Suggestions? 

Thanks in advance,

John

[John Calcote]

I guess I've got the black plague or something - no one has any advice for me. 

I've decided to come up with a system for variable interpolation within my config file - probably an extension of the jackson parser. I've looked around a bit on stack overflow and other places and there's nothing existing that fits the bill for me. For instance, yaml anchors and references might work except you can't built file paths that way because you need a space between the reference and any trailing text. There's also an environment variable enhancement, but I really need internal variables, not external ones.

Please - I'm still open to suggestions. I can't believe no one else cares about this.

John

[Paul Brown]

Hi, John --

I think the most common approach to the use case of treating paths relative to a base path is to integrate a "cd /usr/dude/sweet" into the script you use to run the service.

Another thing you can do is to integrate property interpolation into your string values in a way that's obvious, e.g., create a class InterpolatedStringValue and make that returned by your getters on the configuration class with a method that performs replacements (e.g., of ${property-name}).  An even partially general approach to interpolation will take you quickly down the slippery slope of defaulting, overrides to overrides, when to throw an error for a missing replacement, etc., but there are people who are really into that kind of thing.

Two thoughts, both of which hopefully keep your configuration short on logic and long on interpretability.

-- Paul

—
p...@mult.ifario.us | Multifarious, Inc. | http://mult.ifario.us/

--
You received this message because you are subscribed to the Google Groups "dropwizard-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dropwizard-us...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Glenn McAllister]

Typically this sort of issue is handled in a configuration management system like Puppet.

# This is the lowest level class and the most generic...

class yourapp (

  $keystore_path = $yourapp::params::keystore_path,

  $config_file_path = $yourapp::params::config_file_path,

  $config_file_owner = $yourapp::params::config_file_owner,

  $config_file_group = $yourapp::params::config_file_group,

  $config_file_mode = $yourapp::params::config_file_mode,

  # and so on ...

) {

  # Lots of things you need to do, but the key one I want to demonstrate is creating your

  # configuration file; this isn't necessarily the best way to do it, but it's enough for an 

  # example...

  file { 'my_app_server_config' : 

    path  => "$config_file_path",

    owner => "$config_file_owner",

    group => "$config_file_group",

    mode => $config_file_mode,

    ensure => present,

    content => template('yourapp/yourapp.yaml.erb'),

  } 

}

# This is the class where you do your lookups into Heira or whatever your external CM system is...

class profiles::yourapp {

  $runtime_data_path = heira('profiles::yourapp::runtime_data_path')

  $keystore_path = "$runtime_data_path/keystore.jks"

  $etc_path = heira('profiles::yourapp::etc_path')

  $config_file_path = "$etc_path/yourapp.yaml"

  # configure your application

  class { '::yourapp' :

    keystore_path => "$keystore_path",

    config_file_path => "$config_file_path", 

  }

  # Whatever other component modules you need to configure/setup your application

}

# This is the class that defines all the profiles on a given host/VM - business specific wrapper class

class roles::something::something_else {

  include profiles::yourapp

  include profiles::someotherbusinessapp

  # roles are really, really business specific. Just keep in mind that they are intended to include only profiles, and profiles are 

  # really just collections of specific technology configurations.

}

# and finally, here's a snippet of the ERB file mentioned above, which will end up on 

# the host at the path $config_file_path (probably /etc/yourapp/yourapp.yaml).

server:

    connector:

      - type: my-https

        keyStorePath: <%= @keystore_path %>

All of the above is in a version control system, probably Git these days but whatever works for you.

Do not, please, please, please, do not just have your admins randomly edit files on your live hosts. Use Puppet, Chef, Salt, Ansible, take your freaking pick, to manage the configurations on these hosts and use a VCS to track the changes that are made. It's not as hard as it sounds and you actually have a chance at knowing who made what changes, when, and why, and how to undo them when sh*t hits the fan.

Glenn

--

[Glenn McAllister]

I should have mentioned that if you use Puppet, you really need to read

http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-1/

http://garylarizza.com/blog/2014/02/17/puppet-workflow-part-2/

http://garylarizza.com/blog/2014/02/18/puppet-workflow-part-3/

http://garylarizza.com/blog/2014/03/07/puppet-workflow-part-3b/

http://garylarizza.com/blog/2014/10/24/puppet-workflows-4-using-hiera-in-anger/

Glenn

[John Calcote]

Hi Paul, 

Thanks for the response. We really can't run our application in a specified directory and have that action provide the desired effect. The main reason for this is that we have several system directory roots: a dynamic-data-path rooted in the /var/lib/<company-name>/<product> directory, a static-data-path rooted in /usr/share/<company-name>, etc. Which would we choose and how would we handle the others? This is the sort of stuff one deals with when one writes an installable service, rather than a pure web service. I supposed we could use the technique some applications employ of rooting our entire application in /opt/<company-name>/<product>, and then treating that subspace as our virtual root...

Regards,

John

[John Calcote]

Thanks for the reply Glen. I guess I should have mentioned that one of our main deployment scenarios is simply an rpm package. Thus, deployment for us is merely 'rpm -Uvh our-app-1.0.rpm'. Makes it sort of difficult to use puppet - not that we can't, but I hate to impose that requirement on our customers just because we have no internal method of configuring our own runtime paths.

Regardless, I've found a way of doing what I need to - I've added a variable interpolation module to the dropwizard framework. It's based work done by Henrik Olsson at https://github.com/minisu/Dropwizard-EnvVar-Interpolation. I've enhanced this code so that it can be made to be self-referential - in addition to environment variables, my version will also allow references of the style '${some.path.within.the.yaml.file}' and these references are replaced while the file is being read with the contents of the referenced variable. Of course, this precludes the use of default values for referenced variables (referenced values must be explicitly defined), but it does allow and manage properly command-line overrides as supported by dropwizard. 

Since I finished this module, I've also thought of a way to make default values in configuration factories work as well, but I'm not sure I'll implement it yet.

Thanks again!

John

[Glenn McAllister]

Ah, that is an entirely different kettle of fish, that's true. My brain is a little overly focused on large scale deployments of internal services ATM. :)

I'm glad you found something that works for you.