java-buildpack configuration mechanism: future evolution to avoid forks ?

[Guillaume Berche]

I understand the current way to configure the java-buildpack is to fork it. [0]

This forking practices has the following implications:
- when a security CVE is published, it's harder to get all forked buildpacks apps to upgrade.
- when making changes to the buildpack api [2]or java-buildpack internal design, its costly for lots of forkers to merge
- its a bit heavy weight to fork buildpacks in private paas lacking the nice github ui just to change a config file.

I understand the fork-based config is designed to preserve role separation between app-dev (owning the app, either source or binaries) and app-ops (owning the buildpack).

While this separation of role is still important to lots of Java companies that rely on WARs or EARs as their handover from development/qa to ops, it may make less sense for 12factor apps processes [1]

Wouldn't it make sense to extend the java buildpack configuration mechanism to some more flexible mechanisms such as :
1- env vars. Now that the buildpack api has evolved to support multi line env vars [2], we could imagine having java-buildpack config yml files there. Not sure about total env var limitation though on CF. Its 16KB on heroku [4]. For most small configs such as jvm config this might be sufficient.

2- allow buildpack config within the app with custom files in the staged app (either in WEB-INF/ or some other custom dirs like the .wls of the weblogic buildpack [3] )

I have a personal preference for #1 as to avoid having artefacts themselves depend on the target paas (and endup with conflicting paas configuration mechanism, preventing a same vendor/opensource war from running on multple paas/ private jee container).

Thanks in advance,

Guillaume.

[0] https://github.com/cloudfoundry/java-buildpack/blob/master/README.md#configuration-and-extension
[1] http://12factor.net/dev-prod-parity
[2] https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/IzCw-iiR2pU/Bz9j_lI9LCgJ
[3] https://github.com/pivotal-cf/weblogic-buildpack#application-configuration
[4] https://devcenter.heroku.com/articles/config-vars#limits

ps: Apology if this is a duplicate thread that has been discussed somewhere else, could not nail down a relevant recent one.

[Noburou Taniguchi]

Recently, we've released a configurable Java Buildpack:
https://github.com/cloudn/java-buildpack

I think it is based on the 2nd way you described.

Please take a look and try it if you feel interested.

Details of the buildpack is given in this [1] thread. (Please never mind the thread duplication. I'm pleasant to know there is another person who is interested in this issue.)

[1] https://groups.google.com/a/cloudfoundry.org/forum/#!topic/vcap-dev/8bm0IDAZNHU

By the way, my guess is that the Pivotal people may prefer the 3rd option: "offline buildpack" to configure a Java buildpack. I wonder this is right or not.

2014年6月6日金曜日 8時10分36秒 UTC+9 Guillaume Berche:

[Guillaume Berche]

Thanks Noburou, I had missed this interesting thread.

With the approach of configuring the java buildpack from extra meta data within the app, I'd be interested to hear pros/cons of where to locate the meta data.

The natural java fit seems to extend the java packaging metadata [1], ie extend the META-INF directory with either new keys within the MANIFEST file (like what is done for osgi bundles [3] and eclipse-specifics [2] or new files within the directory structure

Having a plain ."/config" or ".wls" directory within the classpath is probably easier at a first glance for developers, with some rare case of collisions with existing apps. Howevern most java build system offer easy customization of the MANIFEST generation [4] [5] [6].


What's your opinion about env variable-based customization ?

The 12 factor apps [7] defines the app config as "everything that is likely to vary between deploys (staging, production, developer environments, etc)". Some of the java buildpack configs might fall in this category:
- java_opts (e.g. lower JVM sizing in dev, or fine tuning GC trace algo in production)
- tomcat (configuration of session replication with redis_store.database)
- newrelic/app_dynamics (e.g. turning off in dev env)

The 12 factor app config section [7] provides some more arguments of why env vars is a sensible approach w.r.t. to config files checked in source code (or derivative jar files)

In terms of user-experience using CF cli, it could be similar to:

cf set-env web-servlet-2-application JAVA_OPTS.YML < prod_config/java_opts.yml 
Setting env variable 'JAVA_OPTS.YML' to '...' for app web-servlet-2-application in org myorg / space development as me@mecom... 
OK

The "prod_config" directory is outside of the JAR/EAR artefact, usually under version control (e.g. git). 
This means it does interfer with the app packaging and the app detection criteria.

App ops team would script it as shown below to make it reproductible:

for f in `ls prod_config/*.yml`; do cf set-env web-servlet-2-application $f < $f ; done

Setting env variable 'JAVA_OPTS.YML' to '...' for app web-servlet-2-application in org myorg / space development as me@mecom...
OK

Setting env variable 'TOMCAT.YML' to '...' for app web-servlet-2-application in org myorg / space development as me@mecom...
OK

Setting env variable 'postgresql_jdbc.yml' to '...' for app web-servlet-2-application in org myorg / space development as me@mecom...
OK

Thanks,

Guillaume.

[1] http://docs.oracle.com/javase/7/docs/technotes/guides/jar/jar.html#The_META-INF_directory
[2] http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fmisc%2Fbundle_manifest.html
[3] http://www.osgi.org/download/r4v41/r4.core.pdf section 3.2.1 Bundle Manifest Headers
[4] http://maven.apache.org/plugins/maven-jar-plugin/examples/manifest-customization.html
[5] https://ant.apache.org/manual/Tasks/manifest.html
[6] http://stackoverflow.com/questions/11469481/defining-custom-classpath-for-a-jar-manifest-in-gradle
[7] http://12factor.net/config

[Ben Hale]

Guillaume,

Can you elaborate a bit more on the problems you're encountering with security CVEs?  The reason I ask is that we have a couple of the forks of the buildpack that we maintain (some public, some private) but we do so in an automated way that doesn't conflict in most cases.  You can see our "Auto Merge Upstream" bot merging most commits into master without human intervention.  I'd like to get a better understand of the pain you're feeling around this.  

I'm also interested in getting an understanding of how many buildpack forks you currently maintain?  I would have guessed a single fork for each environment but nothing particularly extensive (i.e. many applications using a few buildpacks depending on their application and deployment class).  If you are seeing the need for many more buildpacks, what triggers that need?  Does the create-buildpack/update-buildpack functionality easy the burden at all?

In general, I agree that any application-specific configuration would come via environment variables rather than packed into the deployable artifact.  I'm not convinced yet, but I'm hoping that your input from the earlier questions will help me solidify my opinions on it.

-Ben Hale

Cloud Foundry Java Experience

[Noburou Taniguchi]

I'm sorry for such a delayed response.

Our approach to customize Java buildpack is basically to utilize the existing configuration file structure of the Java buildpack [1] and override one of them (oracle_jre.yml) in intuitive way.

[1] https://github.com/cloudfoundry/java-buildpack/tree/326ade4133149798386d894e49e954e4ab5d53b0/config

However, I think it's a better way (than ours) to customize Java buildpack via environment variables because it is highly uniform and universal, as you wrote.  Another reason is, Heroku is recently promoting configuration in buildpack with environment variables [2] [3] [4].

[2] https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/heroku$20buildpack$20compatibility/vcap-dev/IzCw-iiR2pU/Bz9j_lI9LCgJ
[3] https://devcenter.heroku.com/articles/buildpack-api  (around ENV_VARS)
[4] https://devcenter.heroku.com/articles/config-vars

I think it seems a downside that 'env vars-based customization' may require two cf operations ('cf push' and 'cf set-env'). However, i think this can easily be solved by adding '--set-env' option to 'cf push' command.

So, in my private opinion, env vars-based customization is the way to go, while our file-based customization is viable as a short-term solution.

2014年6月11日水曜日 16時08分37秒 UTC+9 Guillaume Berche:

[Guillaume Berche]

Hi Ben,

Sorry for late response. See my responses inline

Can you elaborate a bit more on the problems you're encountering with security CVEs?  The reason I ask is that we have a couple of the forks of the buildpack that we maintain (some public, some private) but we do so in an automated way that doesn't conflict in most cases.  You can see our "Auto Merge Upstream" bot merging most commits into master without human intervention.  I'd like to get a better understand of the pain you're feeling around this.  

The auto-merge-upstream seems great and certainly addresses the need to keep up-to-date forks with latest build fixes in the up stream java-buildpack. Thanks!
Re-thinking it through, most CVEs would be in JVM, tomcat or frameworks/libs. Apps willing to get latest version would just specify the proper version wildcard [1]. So you're right, applying CVEs is irrelevant to buildpack-custom-env-vs-fork discussion.

 

I'm also interested in getting an understanding of how many buildpack forks you currently maintain?  I would have guessed a single fork for each environment but nothing particularly extensive (i.e. many applications using a few buildpacks depending on their application and deployment class).  If you are seeing the need for many more buildpacks, what triggers that need?  

Some configs that are needed on an app-per-app basis are currently only available through fork base custom in yml file:
- java main:  arguments [2]
- tomcat: redis_store.timeout, redis_store.connection_pool_size

To some extent locking the jvm/container/framework version is also an app-specific configuration. It seems legitimate for an app team to control when such upgrades are triggered in order to manage associated regression risk with deeper tests and production monitoring following upgrades.

I therefore expect more java-buildpack forks than number of envs (e.g. dev/qa/prod). It's however too soon in my org to measure that in current deployments.

Also, in some contexts, it is necessary to automate the java-buildpack configs (e.g. when custom consoles are provided to end-users). Env-based config seem much easier than fork-based config or custom buildpacks through create-buildpack command.
 

Does the create-buildpack/update-buildpack functionality easy the burden at all?

Taking the example of the java main argument for an app, the create-buildpack command enables automating/scripting the buildpack config, but to the extend of being locked with a given version (unlike forking and auto-merging-upstream you mentionned).

In general, I agree that any application-specific configuration would come via environment variables rather than packed into the deployable artifact.  I'm not convinced yet, but I'm hoping that your input from the earlier questions will help me solidify my opinions on it.

Do you envision downsides to adding env-based config in addition to the existing fork-based config ? Brainstorming about such downsides, I could only think of the followings:
- the two step process (cf set-env prior to cf-start): but it can be scripted by teams
- hitting env space limit: does not seem yet an issue given volume of configs available

Thanks,

Guillaume.

[1] https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-repositories.md#version-wildcards
[2] https://github.com/cloudfoundry/java-buildpack/blob/master/docs/container-java_main.md#configuration