Relationship between cf command line tool and buildpacks

[Brian DePradine]

Hi all,

I have been working on the Liberty buildpack for a while now, and one thing that surprised me in cloud foundry was that the cf command line tool behaves differently depending on what is pushed. This is highlighted in this issue that was raised, by another member of my team, issue #23. I assume that this behaviour existed before buildpacks were introduced in cloud foundry in order to simplify support for containers like tomcat. Is this correct?

In the Liberty buildpack we would like to add support for additional archive types such as .ear files, however, in order to get consistent behaviour with what currently happens for .war files we would need to update the list in the command line tool in order to recognize the new archive type. So the question in my mind is, wouldn't it be better if the command line tool didn't care about any archive types and simply pushed what it is given, leaving it entirely up to the buildpack to do whatever processing is needed?

Brian

[Ben Hale]

In the Liberty buildpack we would like to add support for additional archive types such as .ear files, however, in order to get consistent behaviour with what currently happens for .war files we would need to update the list in the command line tool in order to recognize the new archive type. So the question in my mind is, wouldn't it be better if the command line tool didn't care about any archive types and simply pushed what it is given, leaving it entirely up to the buildpack to do whatever processing is needed?

I'd say no, I don't think that it should be the buildpack's responsibility to determine how to detect and handle archives.  Since file extensions are a very crude (and often inaccurate) way of determining what type a file is, you'd end up with every buildpack replicating the same detection, extraction, error handling, etc. for dealing with archives.  I much prefer that Cloud Foundry (either the CLI or the core runtime) handle that in a central place.  In fact, I believe that one of the big reasons that the CLI even does this in the first place is so that it can explode the archive and upload only the change files for performance.

That being said, the current way of detecting archives for extraction (via file extensions) isn't great either.  What we should be doing is examine the bytes of each file to determine if it is an archive using each archive type's magic headers.  You can see as an example that the Apache Compress ZipArchiveInputStream has the ability to 'match' a Zip file.  That is to say that it examines its bytes for a magic grouping and that's what it uses to determine if the file is actually a Zip file, not its extension.  Since all of the Java file types are essentially specially formatted Zips, and both Tar and GZIP have magic headers as well, I suspect that this would cover nearly all of the archive types we'd want to upload and as new esoteric archive types (e.g. SARs) are used, it'd require no changes to either the cli, core-runtime, or buildpacks.  If a genuinely new archive type appeared (e.g. bzip2) we could use the same magic header detection for those as well.

-Ben Hale

Cloud Foundry Java Experience

[Scott Truitt]

Brian, I just talked to the engineers on the Go CLI project and they thought that adding support for .ear files was a simple one line fix. I have a story in my icebox for that work today, so I'll pull it into next week's iteration for you. 

Ben, you're right that the CLI explodes archives to check for changes before uploading. There's not a lot of magic there, pun intended, nor have I heard of much pain outside of this issue, but I'm open to exploring that fix if you think it's of value. 

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.

[Brian DePradine]

Thanks Ben & Scott,

I hadn't considered that the change processing would also be applied to the contents of the archive, but I guess that is fair enough.

@Scott, I expect that the existing cf command line tool is unlikely to be changed, but is there also a story to eventually cover the suggestion that Ben made about inspecting the files to determine if they are archives via the magic headers in the Go CLI tool?

Brian

[Scott Truitt]

Correct, there's no plan to update the Ruby cf cli with this functionality. I'll add a story to investigate magic headers in the Go CLI backlog. 

To unsubscribe from this group and stop receiving emails from it, send an email to vcap-dev+u...@cloudfoundry.org.