Hello vcap-dev!
This email details a proposed change to how Cloud Foundry buildpacks are packaged, with respect to the ever-increasing number of binary dependencies being cached within them.
This proposal's permanent residence is here:
https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4
Feel free to comment there or reply to this email.
Many of you have seen, and possibly been challenged by, the enormous sizes of some of the buildpacks that are currently shipping with cf-release.
Here's the state of the world right now, as of v205:
php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G
These enormous sizes are the result of the current policy of packaging every-version-of-everything-ever-supported ("EVOEES") within the buildpack.
Most recently, this problem was exacerbated by the fact that buildpacks now contain binaries for two rootfses.
If continued, buildpacks will only continue to increase in size, leading to longer and longer build and deploy times, longer test times, slacker feedback loops, and therefore less frequent buildpack releases.
Additionally, this also means that we're shipping versions of interpreters, web servers, and libraries that are deprecated, insecure, or both. Feedback from CF users has made it clear that many companies view this as an unnecessary security risk.
This policy is clearly unsustainable.
There are many things being discussed to ameliorate the impact that buildpack size is having on the operations of CF.
Notably, Onsi has proposed a change to buildpack caching, to improve Diego staging times (link to proposal).
However, there is an immediate solution available, which addresses both the size concerns as well as the security concern: packaging fewer binary dependencies within the buildpack.
I'm proposing that we reduce the binary dependencies in each buildpack in a very specific way.
Aside on terms I'll use below:
I'd like to move forward soon with the following changes:
An example for #1 is that we'll go from packaging 34 versions of node
v0.10.x to only packaging two: 0.10.37 and 0.10.38.
An example for #2 is that we'll go from packaging 3 versions of nginx
1.5 in the PHP buildpack to only packaging one: 1.5.12.
An example for #3 is that we'll discontinue packaging ruby 1.9.3 in the ruby-buildpack, which reached end-of-life in February 2015.
With these changes, the total buildpack size will be reduced greatly. As an example, we expect the ruby-buildpack size to go from 922M to 338M.
We also want to set the expectation that, as new interpreter versions are released, either for new features or (more urgently) for security fixes, we'll release new buildpacks much more quickly than we do today. My hope is that we'll be able to do it within 24 hours of a new release.
These changes will be relatively easy to make, since all the buildpacks are now using a manifest.yml
file to declare what's being packaged. We expect to be able to complete this work within the next two weeks.
Stories are in the Tracker backlog under the Epic named "skinny-buildpacks", which you can see here:
Please let me know how these changes will impact you and your organizations, and let me know of any counter-proposals or variations you'd like to consider.
Thanks,
-mike
--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com.
To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/94f5dfd0-6f3e-4d7f-83ff-c855fd1b2c53%40cloudfoundry.org.
Hi Lucas,Thanks for commenting!I agree completely that if everyone followed semver, it would be easier to approach this systematically. However, since often projects interpret semver differently (and sometimes people make mistakes), I'm worried that if we're too aggressive in removing older-but-still-supported versions from the buildpacks, we'll increase the burden to application developers unnecessarily.In the Ruby buildpack, which I'm most familiar with personally, we definitely *do* need to treat MINOR bumps as possibly-API-incompatible; Ruby 2.0.x and 2.2.x are different enough that we need to support all of them. As an example, Ruby 2.2 removed some deprecated C API calls that were present in 2.1; any library that used these deprecated APIs would stop working in 2.2. Application developers may not want to upgrade to Ruby 2.2.x just yet, and I don't want that upgrade pain to be a blocker to CF adoption.In the PHP buildpack, if we assumed that MINOR bumps were still API/ABI compatible, then we'd be able to drop 5.4.x and 5.5.x completely. I don't have enough experience in the PHP ecosystem to know whether this is desired or not. Do you think it's acceptable to include only 5.6.6 and 5.6.7 in the buildpack? Or would we run into similar issues?
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwpU0%3DM%2BCrJJLaHd%3DhmAfTNxxMhHFzKTymGYCtX%3D7W9Nw%40mail.gmail.com.
On Wed, Apr 15, 2015 at 10:30 AM, Mike Dalessio <mdal...@pivotal.io> wrote:Hi Lucas,Thanks for commenting!I agree completely that if everyone followed semver, it would be easier to approach this systematically. However, since often projects interpret semver differently (and sometimes people make mistakes), I'm worried that if we're too aggressive in removing older-but-still-supported versions from the buildpacks, we'll increase the burden to application developers unnecessarily.In the Ruby buildpack, which I'm most familiar with personally, we definitely *do* need to treat MINOR bumps as possibly-API-incompatible; Ruby 2.0.x and 2.2.x are different enough that we need to support all of them. As an example, Ruby 2.2 removed some deprecated C API calls that were present in 2.1; any library that used these deprecated APIs would stop working in 2.2. Application developers may not want to upgrade to Ruby 2.2.x just yet, and I don't want that upgrade pain to be a blocker to CF adoption.In the PHP buildpack, if we assumed that MINOR bumps were still API/ABI compatible, then we'd be able to drop 5.4.x and 5.5.x completely. I don't have enough experience in the PHP ecosystem to know whether this is desired or not. Do you think it's acceptable to include only 5.6.6 and 5.6.7 in the buildpack? Or would we run into similar issues?I'm curious to hear what others thing, but I don't think this would be acceptable for PHP users. While almost all PHP apps use 5+, usage of the minor versions is still pretty diverse with the majority using older versions [1]. While I think the relevance of these stats could be debated, because I feel like this trend is caused by the fact that most Linux distros don't keep up with the latest versions and people tend to use what's packaged with the distro, I think it shows that most people aren't thinking about using PHP 5.6 yet.My thought for support is that we should continue including binaries for a PHP 5.x version as long as it's being maintained. In other words, support the latest 5.4.x release until the PHP dev's stop publishing them. Since they seem to kill off the oldest release when the next PHP is stable, that generally leaves the build pack with three versions to support.On a slightly different note, I think we can help adoption of newer PHP versions by bumping up the default version of PHP used by the build pack. It's currently 5.4. Perhaps we should consider going to 5.5?
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAAqVxAwHagsfPz997Mb%2BfH6Y4MwvyxdqyR%3DO6bXSOchYXQ6zxg%40mail.gmail.com.
To view this discussion on the web visit https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com.