Heroku buildpack compatibility

[Matthew Sykes]

Hi.  I ran into an issue this morning that caused me to look into some differences between the Heroku[1] and Cloud Foundry [2] buildpack support.

One of the things that jumped out was Heroku's recent treatment of environment variables configured for the app.  In their model, they provide an extra argument to the compile script that points to a directory containing text files; each file has the name of the environment variable and the contents of the file is the value associated with that environment.  This is only presented to the compile script.

In CloudFoundry, application environment variables and generated variables like `MEMORY_LIMIT` are exported to the environment and are available across detect and compile.

If we want to be able to use Heroku build packs in Cloud Foundry, it seems like we should update the DEA and diego stager to start creating the `ENV_DIR` tree so build packs can use the same mechanism across both environments.  Any concerns there?

As for the existing behavior of exporting all variables when executing detect/compile, I actually think the Heroku guys got it right by avoiding that.  Does anyone have any idea what kind of fall out there would be if only the `VCAP_SERVICES`, `VCAP_APPLICATION`, and MEMORY_LIMIT generated variables were exported?

Thanks.

[1] https://devcenter.heroku.com/articles/buildpack-api

[2] http://docs.cloudfoundry.org/buildpacks/custom.html

--
Matthew Sykes
matthe...@gmail.com

[Patrick Mueller]

Sounds good to me; I assume any env var set in the app via `cf set-env` are also available.

If MEMORY_LIMIT == `cf -m` option value, why wouldn't the rest of the "app settings" be here, like (run `cf help push` to see options).  Some are pointless (path of app directory), some are controversial (startup command; previous thread in the group on this), but some potentially useful (disk limit).

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--
Patrick Mueller
http://muellerware.org

[James Bayer]

thanks for bringing this up matt, i hadn't seen the heroku buildpack api change that introduced the env var directory. the article was updated in early april, but it's hard to know when that change was introduced as i don't see it announced in their build forums [1].

i believe we should adopt the approach of a env var directory with files for each to stay consistent with heroku buildpacks. are there reasons someone can think of why we would not want to do this?

patrick, as to your comment:

I assume any env var set in the app via `cf set-env` are also available.

i believe that customized app env vars is the only thing heroku is putting in that directory. i don't see references to other heroku provided values and i haven't tried looking in that env dir on a heroku deployment yet. if so, the question now is for us to check if we stop setting the app env vars by default, which buildpacks would that break? for example, i had assumed previously in the SCM buildpack [2] that the SCM_URL is just automatically available without running the special bash script. so that buildpack would need updating before being compatible with a change like this.

how do people feel about giving notice that app env vars will not be set automatically by the system and depend on the buildpack compile script to do it and making that change the default behavior? should we introduce the behavior of the new dir and still set app env vars automatically for a period of time before turning it off? my inclination is to rip off the bandaid all at once, but what are the other views?

on the other question, it seems like CF system specific env vars like VCAP_SERVICES should be passed automatically since those are not set explicitly by the user. buildpacks that are explicitly written for CF would ideally tolerate those env vars not being present with defaults so they could work on other systems, but it seems that shouldn't be required.

[1] https://discussion.heroku.com/category/build 

[2] https://github.com/jbayer/scm-buildpack

Thank you,

James Bayer

[Mike Youngstrom]

Interesting change.  One common semi-common practice for env vars during staging is to set proxy values "http_proxy and https_proxy".  I'd imagine most buildpacks would need to explicitly support those variables if they want to work behind a proxy.

I vote not to "rip off the bandaid".  I vote to go with both behaviors for a time with a clear deprecation of the env variables being automatically set.

Are we sure we don't want to treat cf set environment variables the same way?  Previously all talk about CF compatibility with heroku has been around CF supporting Heroku buildpacks.  But some CF only buildpacks depend upon VCAP_SERVICES and VCAP_APPLICATION which aren't there in a heroku build.  If CF treated VCAP_* environment variables the same way as any other environment variable could that help add a degree of compatibility for CF buildpacks to work on Heroku?  If we treated VCAP_* variables like user provided variables and put them in the ENV_DIR could an application developed targeting a CF specific buildpack theoretically run on Heroku as well if VCAP_* config variables were added to the Heroku app by the user?

Does Heroku support buildpacks written in ruby?

Mike

[Ben Hale]

Interesting change.  One common semi-common practice for env vars during staging is to set proxy values "http_proxy and https_proxy".  I'd imagine most buildpacks would need to explicitly support those variables if they want to work behind a proxy.

A number of languages and libraries support those env vars (and variations like HTTP_PROXY and HTTPS_PROXY) as convention and so some buildpacks may not necessarily need special support.  Sadly, the Java Buildpack does, and it's been contributed recently.

 

I vote not to "rip off the bandaid".  I vote to go with both behaviors for a time with a clear deprecation of the env variables being automatically set.

Agreed on deprecation.  Any particular reason that we wouldn't leave both in place permanently?  That would give the greatest amount of compatibility.

 

Does Heroku support buildpacks written in ruby?

It would appear so.

-Ben Hale

Cloud Foundry Java Experience

[Matthew Sykes]

My concerns is around exporting variables like PATH, LIBPATH, LD_LIBRARY_PRELOAD, etc.  For variables like those, I doubt most buildpacks would expect to run with values specified for the app instance.

While I agree in principle with deprecating and transitioning, I still think the Heroku guys got it right; the staging environment is not the same as the app execution environment.

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

--
Matthew Sykes
matthe...@gmail.com

[James Bayer]

ok, based on the feedback, my recommendation is to deprecate setting all app env vars for a period of time TBD, and then transition to having to explicitly decide which env vars are set by the buildpack itself via the env-var dir that is passed to the compile script.

once the runtime team picks this work up we'll communicate more.

Thank you,

James Bayer

[James Bayer]

shannon created this story [1] to implement the you-must-opt-in style of setting app env variables in the buildpack.

since diego just started staging applications in the CI environments at the end of last week, matthew kocher is proposing making this change in diego only and when diego formally takes over staging applications, then that is the time to transition to the new style. when mark/shannon have an approximate timeline for that, we'll communicate further. it should be a minimum of several weeks before we would stop staging on DEAs and transition to only recommend staging on diego.

[1] https://www.pivotaltracker.com/story/show/70246426

[Mike Youngstrom]

What was decided for VCAP_SERVICES and VCAP_APPLICATION?

I'd like to see them handled the same as other environment variables.

I'm not a fan of using Diego as the cut over since that will require buildpack developers to support both models for a while.  Otherwise we will need to sync buildpack updates with CF deploys again.  Unfortunate now that we've finally de-coupled them. :)  Not a big deal for me as long as the buildpacks I use support both models for a while.  (nudge Ben Hale)

Mike

[Jan Dubois]

On Mon, Apr 28, 2014 at 4:48 PM, Mike Youngstrom <you...@gmail.com> wrote:
> I'm not a fan of using Diego as the cut over since that will require
> buildpack developers to support both models for a while.

Won't buildpacks that have been updated for the new model continue to
support the old model automatically?

E.g. if there is no $ENV_DIR/FOO file, but $FOO is already set, then
the buildpack will simply not source the environment variable from
file but use the current value.

I think maintaining this level of backwards compatibility is pretty
important anyways. Otherwise the new buildpacks will be broken on
older installation for people who reference them directly from e.g.
Github.

Cheers,
-Jan

[Mike Youngstrom]

E.g. if there is no $ENV_DIR/FOO file, but $FOO is already set, then
the buildpack will simply not source the environment variable from
file but use the current value.

Easier said than done as apps might reference an old buildpack fork or branch for a while.  So, this requires system owners to look at all the buildpacks used and ensure they are all updated with that logic or we break them when moving to diego.

If Diego just "export ENV" in addition to creating the file for a while this because less of an abrupt change.

Mike

[James Bayer]

yes jan, if buildpacks start implementing support for the updated compile argument for a env-var directory now, they will be backward compatible when the change switch over to diego occurs if we follow matthew's suggestion to have diego only support opt-in app env-vars.

mike, the story is a placeholder to drive out all of the details in further conversations. it should reflect the decisions when mark is back, he's currently at IBM Impact. you could always choose not to adopt diego staging to keep old buildpack functionality for a bit longer, but presumably we will only leave that code around for a certain period of time if there is enough people not ready to move to diego yet.

i pushed an example app to heroku with a modified null buildpack compile script [1] that prints the compile output for the env without doing opting in to an env vars to see what env vars they set if you do nothing and got this output [2]. compare that with the same app push to CF with this buildpack [3].

you can see that heroku does set heroku system vars like logplex info, ssh info, user info, etc in the env without the buildpack having to explicitly setting them. in order to get an app env var like FOO, the buildpack would have to explicitly ask for it. following this convention means that CF-specific env vars should be set automatically. presumably this would include:

USER=vcap
VCAP_APPLICATION={"limits":{"mem":256,"disk":1024,"fds":16384},"application_version":"65655208-793b-4585-ae61-2678b3e67965","application_name":"env-heroku","application_uris":["env-heroku.tabasco-app.cf-app.com"],"version":"65655208-793b-4585-ae61-2678b3e67965","name":"env-heroku","space_name":"development","space_id":"34abe92a-fc08-431a-a6ca-b673c021972e","uris":["env-heroku.tabasco-app.cf-app.com"],"users":null}
STAGING_TIMEOUT=900.0
VCAP_SERVICES={}
BUILDPACK_CACHE=/var/vcap/packages/buildpack_cache

MEMORY_LIMIT=256m

there may be other env vars too, but this is a good start.

[1] https://github.com/jbayer/null-buildpack/blob/master/bin/compile

[2] https://gist.github.com/jbayer/11387928

[3] https://gist.github.com/jbayer/11388173

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Ben Hale]

yes jan, if buildpacks start implementing support for the updated compile argument for a env-var directory now, they will be backward compatible when the change switch over to diego occurs if we follow matthew's suggestion to have diego only support opt-in app env-vars.

If Diego only supports opt-in env vars, and Diego is only a couple of weeks away, that means that the "deprecation" period is also only a couple of weeks long.  To me that seems way too short, especially since this is a compatibility breaking change.  It's also the worst kind of compatibility breaking change because it will simply cause a silent failure.  If you haven't updated a custom buildpack in the next couple of weeks (because you don't follow vcap-dev closely), the next time you take a drop (of either the open source or commercial distributions), your buildpack will just stop working and you won't know why.  There will be no indication that environment variables that used to matter no longer do, when your application is staging.

To put this into concrete terms, suppose you've forked the Java Buildpack to customize which version of the JRE to use.  You also depend on the ability to set a JAVA_OPTS variable (with cf set-env) as a flag to manage the security of some endpoint in your application.  If you then upgrade to a Diego-based version, all of a sudden that JAVA_OPTS variable is no longer available and your application is left insecure.  There's no warning that this has happened anywhere in the push lifecycle of your application, so you'd better hope that you've got a pre-production environment and test-suite that perfectly reproduces your production environment and user behavior.

At the moment, there are 200 public forks of the Java Buildpack (and who knows how many more private ones).  If I had to guess, about 190 of them aren't kept up to date and would end up in this silent failure mode after this change.  I believe that the deprecation period for the old environment variable behavior must be significantly longer (months to perhaps a year) and would require a major version change to commercial distributions.  I don't think that we should underestimate the impact of this change to users.

[James Bayer]

thanks for your feedback ben. i believe it's also an important goal to track the way buildpacks work now. the change needed for buildpack authors or forkers is completely trivial [1] so i don't think this type of trivial change makes sense to delay the implementation up to a year. if we hear more vocal community members make a case for a long deprecation period that can influence the length, but i suspect we'll plan on a much shorter deprecation period so we can more closely follow heroku buildpacks.

buildpack authors that expect access to app environment variables should make the change now since it does not break things in the time before the deprecation period is ended.

[1] https://gist.github.com/jbayer/3f6d31f06dcb26f7c325

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Ben Hale]

thanks for your feedback ben. i believe it's also an important goal to track the way buildpacks work now. the change needed for buildpack authors or forkers is completely trivial [1] so i don't think this type of trivial change makes sense to delay the implementation up to a year. if we hear more vocal community members make a case for a long deprecation period that can influence the length, but i suspect we'll plan on a much shorter deprecation period so we can more closely follow heroku buildpacks.

I'm not advocating a delay in implementation for a year, I'm advocating both approaches being available for a year.  By all means, add the Heroku style tomorrow but I don't believe that you should be removing the Cloud Foundry style at the same time.

buildpack authors that expect access to app environment variables should make the change now since it does not break things in the time before the deprecation period is ended.

As you point out, the change is trivial.  My question is, how are you planning on communicating the change to the 205 people who've already forked the Java buildpack?  I'm pretty sure that not all of them follow vcap-dev or read the change logs for each release of the product.

[James Bayer]

ben,

there are multiple ways this could go. if VCAP_SERVICES is an opt-in as matthew kocher is advocating in the story [1], then it's more disruptive to those that are relying on auto configuration, but still not that bad in my view.

however, if VCAP_SERVICES, MEMORY_LIMIT are provided automatically without opt-in, which seems consistent with Heroku's style of providing some system env vars that are not explicitly set by the user, then isn't the java buildpack impact limited to things like log level?

[1] https://www.pivotaltracker.com/story/show/70246426

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Guillaume Berche]

Can you clarify the following point for me :  regardless of whether the compile script exports the env dir files as envs, the env vars will be defined prior to the release script being executed ? So, for a buildpack to black list env vars set in cf-env such as PATH, LD_LIBRARY_PATH, it would have to explicitely do so in the release command or script.

In other words, I understand the impact of this story is mostly buildpacks that lookup the env vars in the compile script to:
- configure the compile script (e.g. alternate location for dependencies download)
- use them in the release command or in the droplet files.

The practice of setting http_proxy and https_proxy in the app with cf-env prior to starting the app is currently a necessary workaround to be able to use buildpack with dependencies fetched on the internet from a Cf instance that don't have direct outgoing internet access, and not require local download of each artefact as a 1st attempt to run the buildpack privately.

Would it make sense to let cf operators extend the list of no-opt-in variables (currently MEMORY_LIMIT, VCAP_SERVICES ...) with some envs specific to the local cf instance that are likely to be necessary on most buildpacks. I'm thinking of:
- HTTP_PROXY HTTPS_PROXY: as described above
- LANG: controls languages in which buildpack produces messages
- TZ: controls the timezone of produced logs

As a first step having these env var names and values in the bosh deployment manifest would be sufficient. Having them configureable per space might be useful in some contexts.

Thanks,

Guillaume.

[James Bayer]

HTTP_PROXY and HTTPS_PROXYboth make sense to me to have as system env variables relevant whenever we have user-provided code that may need to reach out into the world. according to what we’re learning from some users in really locked down environments, even http interactions between system components are required to go through a mandated proxy.

LANG and TZ seem valuable as system env vars too, such that they could be set in a bosh deployment manifest and be available universally if the bosh packaging for a component uses them.

i don’t think we’re at the level of refinement to have a space-wide setting for these and apply overrides, etc. as PM for runtime, these are all mark’s decision to consider all of this input and make/communicate the decision for now and what makes sense for future considerations.

[Guillaume Berche]

thanks James, looking forward to hearing Mark's decisions on this.

Guillaume.

[Guillaume Berche]

Mark, James,

Any update on the decision to allow some system env variables customized in the bosh manifest ?

Looking for related entries in the tracker, I could only find https://www.pivotaltracker.com/story/show/63635056 "Expose http_proxy, https_proxy, no_proxy envvars to the dea so buildpacks can use them" but which does not provide much comments on its status.

Thanks in advance,