Proposal for a new project: OCI Image Format Spec

[Brandon Philips]

TOB Members-

Please read this proposal based on our call. We are all very busy. So, at a minimum please reply with one of three options:

- Agree with this proposal
- Disagree with this proposal
- Lets discuss, I need more information

Also, it looks like our next meeting is centering around Mar 4th: http://doodle.com/poll/9gdepaaygtcvidrt

Thank You,

Brandon

OCI Image Format Spec

New project proposal: The TOB will create a new OCI project tasked with creating a software shipping container image format spec (OCI Image Format) with security, naming, and distribution. In addition the OCI TOB will establish a new set of maintainers for this new project with people who have expertise in image formats, transport mechanisms, and package management.

Initial Recommendation: Over the last 16 months we have seen evolution of container formats towards solid technical underpinnings. When the AppC project was introduced in December 2014 its goals were to create an open specification shared between implementations while addressing concerns in the Docker v1 image format around content addressable images, signing, and a federated/delegatable namespace. Today, the Docker v2.2 image format is close to having all of the desirable traits of AppC while having widespread registry implementations in projects/products from Amazon, Google, CoreOS, Docker, Huawei, and JFrog.

This new OCI project would be recommended to start with the Docker v2.2 specification, improve any remaining technical concerns, and create an OCI project and maintainers to develop and shepard an OCI Image Fromat Spec. By starting from this project we intend to standardize and improve the understood properties of a container image format, namely:

• Image that is content addressable

• Security that is based on signing image content address

• Naming that is federated based on DNS (enables ec2 cr, gcr, quay.io, localhost registry, etc) and can be delegated

• Distribution that is based on HTTPS and enables other protocols (e.g. dockyard bittorrent p2p)

Initial Maintainers: to be brainstormed on a separate thread.

Cooperation with OCI Runtime Project: The OCI Runtime Specs project is working diligently to create a specification for the lifecycle of a running container. The OCI Image Format Spec project should work with the OCI Runtime Spec project so that the an image can support the UX that users have come to expect from container engines like Docker and rkt. Namely the ability to pull human names with implied context (OS, arch, etc) and have them run with no additional arguments:

   docker run example.com/org/app:v1.0.0

   rkt run example.com/org/app,version=v1.0.0

This implies that the Software Shipping Container Spec contain sufficient information to launch the application on the target platform e.g. command, arguments, environment variables, etc.

FAQ

Q: Why a new project?

A: The first OCI spec centered around defining the run side of a container. This is generally seen to be an orthogonal concern to the shipping container component. As practical examples of this separation you see many organizations separating these concerns into different teams and organizations: the Docker Distribution project and the Docker containerd project; Amazon ECS and Amazon EC2 Container Registry, etc.

Q: Why start this work now?

A: We are seeing many independent implementations of container image handling including build systems, registries, and image analysis tools. As an organization we would like to encourage this growth and bring people together to ensure a technically correct and open specification continues to evolve reflecting the OCI values.

Q: What happens to AppC or Docker Image Formats?

A: Existing formats can continue to be a proving ground for technologies, as needed. For example, in line with the OCI values, we expect mechanisms like the AppC name delegation will help inspire portions of the OCI Software Shipping Container project. The OCI Image Format project should strive to provide a dependable open specification that can be shared between different tools and be evolved for years or decades of compatibility; as the deb and rpm format have.

Proposed Roadmap

• March ?? v0.0.0

• Import Docker v2.2 format

• April 4th v0.1.0

• Spec factored for top to bottom reading with three audiences in-mind:

• Build system creators

• Image registry creators

• Container engine creators

• Test cases for each of the potential users exercising build, push, and pull verbs

• April 18th v0.2.0

• Release version of spec with improvements from two independent experimental implementations from OCI members e.g. Amazon Container Registry and rkt

• May 16th v1.0.0

• Release initial version of spec with two independent non-experimental implementations from OCI members

[Greg KH]

On Fri, Feb 26, 2016 at 12:33:07AM +0000, Brandon Philips wrote:
> TOB Members-
>
> Please read this proposal based on our call. We are all very busy. So, at a
> minimum please reply with one of three options:
>
> - Agree with this proposal
> - Disagree with this proposal
> - Lets discuss, I need more information
>
> Also, it looks like our next meeting is centering around Mar 4th: http://
> doodle.com/poll/9gdepaaygtcvidrt

This looks very good to me, so I'm replying:
I agree with this proposal.

thanks for writing this all up.

greg k-h

[Jason Bouzane]

I agree with this proposal.

> --
> You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.
>

[Diogo Mónica]

I think we all agreed on the call that a distributable image format was something we should pursue, and that it is already slated to be addressed by spec 1.0.

As stated, your proposal makes it seem like there has been no progress on this front, while in fact both this issue: https://github.com/opencontainers/specs/issues/11 and this PR: https://github.com/opencontainers/specs/pull/293 are currently actively being discussed, and seem to be focused on exactly that goal.

I genuinely don't see the need for yet another group of people, selected from essentially the same pool, to address a problem that is already being addressed.

Moreover, instead of focusing on a distributable format, you also seem to be proposing standardizing naming, security and specific distribution methods. These are mostly orthogonal problems and I strongly believe that they should be addressed at a later time, if the TOB feels the need to create these extra optional layers.

Unfortunately, even though I agree with pursuing the standardization of a distributable image format, I cannot agree with current proposal as stated.

--

You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.

--

Diogo Mónica

[Greg KH]

On Thu, Feb 25, 2016 at 09:05:46PM -0800, Diogo Mónica wrote:
> I think we all agreed on the call that a distributable image format was
> something we should pursue, and that it is already slated to be addressed by
> spec 1.0.
>
> As stated, your proposal makes it seem like there has been no progress on this
> front, while in fact both this issue: https://github.com/opencontainers/specs/
> issues/11 and this PR: https://github.com/opencontainers/specs/pull/293 are
> currently actively being discussed, and seem to be focused on exactly that
> goal.
>
> I genuinely don't see the need for yet another group of people, selected from
> essentially the same pool, to address a problem that is already being
> addressed.

But it isn't being addressed from what I can see. That issue you refer
to and the pr both aren't going anywhere, they are just too big and
hairy to produce any valid end result.

And the roadmap does not have any of this type of information in it, so
I don't see how anyone is thinking that it will be addressed in the
current framework of what we have now.

> Moreover, instead of focusing on a distributable format, you also seem to be
> proposing standardizing naming, security and specific distribution methods.

Naming and security are an intricate part of a usable distributable
format, they can not be separated, if you want a specification that is
actually usable :)

The fact that the image can be distributed is key, not the method in
in which it is distributed, so I don't see why you would worry about
specifics there.

And even if you do worry about that, we have to specify this, our
roadmap says so (last bullet point), so it's not an option.

> These are mostly orthogonal problems and I strongly believe that they should be
> addressed at a later time, if the TOB feels the need to create these extra
> optional layers.

Don't "tack on" security and naming for later, that way is madness, and
a guarantee that you will make your previous versions incompatible. And
a guarantee that everyone will "tack on" their own half-baked security
model, making everything instantly incompatible to start with.

> Unfortunately, even though I agree with pursuing the standardization of a
> distributable image format, I cannot agree with current proposal as stated.

Because you don't want security and naming? Or you feel it is somehow
outside the scope of this organization? Or if it is inside the scope,
that it should just be done within?

In short, what would an acceptable proposal look like to you? Or do you
think that we should just not work on this type of thing at all at this
point in time, despite it being something that it seems we all want?

thanks,

greg k-h

[W. Trevor King]

On Fri, Feb 26, 2016 at 05:41:49AM +0000, Greg KH wrote:
> Don't "tack on" security and naming for later, that way is madness…

If the image format is built on cryptographic hashes for
content-addressable chunks, then tacking on security and naming seems
feasible to me (e.g. Git tacks on security and naming with signed
tags, and the only quibble I've ever heard about that approach is that
SHA-1 isn't strong enough).

Cheers,
Trevor

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[Diogo Mónica]

Like Trevor said: if we have an image format that is content addressable, having the hash is enough to build strong security. 

In fact, Docker Content Trust works exactly that way: we providing a secure mapping between a tag and a hash, and then ensure the content that is downloaded follows the Merkle DAG.

The fact that the image can be distributed is key, not the method in
in which it is distributed, so I don't see why you would worry about
specifics there.

I agree that the image being distributable is key, and that the method in which it is distributed doesn't matter. That is exactly why I'm pushing back on the "Distribution that is based on HTTPS" that is included in the proposal.

Same thing for naming. I'm not opposed to adding this optional layer to the OCI spec, but I don't think a specific mechanism based on DNS is something we should do right now, definitely not before the spec 1.0 comes out and not before we actually have agreement on what the distribution format looks like.

Just my 2c.

--

Diogo Mónica

[Greg KH]

On Thu, Feb 25, 2016 at 10:05:58PM -0800, W. Trevor King wrote:
> On Fri, Feb 26, 2016 at 05:41:49AM +0000, Greg KH wrote:
> > Don't "tack on" security and naming for later, that way is madness…
>
> If the image format is built on cryptographic hashes for
> content-addressable chunks, then tacking on security and naming seems
> feasible to me (e.g. Git tacks on security and naming with signed
> tags, and the only quibble I've ever heard about that approach is that
> SHA-1 isn't strong enough).

Git didn't "tack on" security, it was built into the very first version
with the fact that nothing was ever rewritten and the use of SHA1.
Signed tags were added later, of course, but that has nothing to do with
what we are talking about here. If we don't give a way to have hashes
or something like that, then it will have to be added later, which is
why we need it from the beginning, as you say :)

thanks,

greg k-h

[Jason Bouzane]

On Thu, Feb 25, 2016 at 10:23 PM, Diogo Mónica <diogo....@docker.com> wrote:
> Like Trevor said: if we have an image format that is content addressable,
> having the hash is enough to build strong security.
>
> In fact, Docker Content Trust works exactly that way: we providing a secure
> mapping between a tag and a hash, and then ensure the content that is
> downloaded follows the Merkle DAG.
>
>> The fact that the image can be distributed is key, not the method in
>> in which it is distributed, so I don't see why you would worry about
>> specifics there.
>
>
> I agree that the image being distributable is key, and that the method in
> which it is distributed doesn't matter. That is exactly why I'm pushing back
> on the "Distribution that is based on HTTPS" that is included in the
> proposal.
>
> Same thing for naming. I'm not opposed to adding this optional layer to the
> OCI spec, but I don't think a specific mechanism based on DNS is something
> we should do right now, definitely not before the spec 1.0 comes out and not
> before we actually have agreement on what the distribution format looks
> like.
>
> Just my 2c.

Without naming, bundles are much less useful. Being able to
symbolically name other bundles enables all sorts of use cases (e.g.
being able to reference the stable track of another bundle to include
in your own). Naming brings in distribution and trust, too, since you
need to be able to retrieve the named object and you need to be able
to trust that the object is created by who you expect.

Referencing other bundles can be done without naming, but it's much
harder to do discovery, to write a build rule that constructs a bundle
from other bundles, and so on.

I'd certainly agree that the bundle format is the most important
thing, and that the other things cannot be done without a bundle
format. But naming, security, and distribution are still very
important goals that should be part of the standard.

[Greg KH]

On Thu, Feb 25, 2016 at 10:23:12PM -0800, Diogo Mónica wrote:
> Like Trevor said: if we have an image format that is content addressable,
> having the hash is enough to build strong security. 
>
> In fact, Docker Content Trust works exactly that way: we providing a secure
> mapping between a tag and a hash, and then ensure the content that is
> downloaded follows the Merkle DAG.
>
>
> The fact that the image can be distributed is key, not the method in
> in which it is distributed, so I don't see why you would worry about
> specifics there.
>
>
> I agree that the image being distributable is key, and that the method in which
> it is distributed doesn't matter. That is exactly why I'm pushing back on the
> "Distribution that is based on HTTPS" that is included in the proposal.

Ah, that makes more sense. I don't agree with you (https is a great
place to start), but I understand your objection.

thanks,

greg k-h

[Michael Crosby]

From my POV I don't see how specifying "Distribution that is based on HTTPS and enables other protocols (e.g. dockyard bittorrent p2p)" is very important.

Its like saying that all containers will be shipped by boat, if you use a train then you are using a non-standard shipping service when all that you really care about is the cargo. Just something that i'm trying to understand.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.

--

Michael Crosby

@crosbymichael

[Jason Bouzane]

My reading of that was that HTTPS must be supported to be compliant with the OCI spec, but the bundle format should allow for other protocols to be used if the particular implementation supported them. Brandon: can you clarify whether that's correct?

[Brandon Philips]

On Fri, Feb 26, 2016 at 5:19 PM Jason Bouzane <jbou...@google.com> wrote:

My reading of that was that HTTPS must be supported to be compliant with the OCI spec, but the bundle format should allow for other protocols to be used if the particular implementation supported them. Brandon: can you clarify whether that's correct?

In my mind without specifying something we really aren't creating a shipping container as we have no baseline to test against. We should design flexibility for other schemes as I know we all share a common interest in other methods like p2p, etc. And starting with docker v2.2 spec I know that flexibility exists.

Overall, I think we all acknowledge that nearly all containers are being shipped over HTTPS today and we should start with that design with that in mind. Otherwise, we could end up designing for 0% market share with this spec.

Brandon

[Brandon Philips]

On Fri, Feb 26, 2016 at 5:28 PM Brandon Philips <brandon...@coreos.com> wrote:

On Fri, Feb 26, 2016 at 5:19 PM Jason Bouzane <jbou...@google.com> wrote:

My reading of that was that HTTPS must be supported to be compliant with the OCI spec, but the bundle format should allow for other protocols to be used if the particular implementation supported them. Brandon: can you clarify whether that's correct?

In my mind without specifying something we really aren't creating a shipping container as we have no baseline to test against. We should design flexibility for other schemes as I know we all share a common interest in other methods like p2p, etc. And starting with docker v2.2 spec I know that flexibility exists.

Overall, I think we all acknowledge that nearly all containers are being shipped over HTTPS today and we should start with that design with that in mind. Otherwise, we could end up designing for 0% market share with this spec.

To clarify why I think we need to include some sort of transport in the spec vs just leaving it undefined let's run through this scenario:

After we release this OCI Image Format spec Kubernetes wants to become compliant by implementing all of the naming, download, and extraction functionality itself. So, the Kubernetes team starts to implement a test case that looks something like:

    "image": {"name: "example.com/foo", "type": "oci"}

Next they will configure and run one or more OCI Image Format registries to serve up the images. The compliance test will then need to tell Kubernetes how to find the registry and what protocol to use in configuration files or APIs. And finally Kubernetes will use this configuration to download the image from the registry and run it.

My concern is that if we avoid specifying anything then we really have nothing to test naming, transport, etc end-to-end. Or put another way if we don't have any transport in the spec how do we have projects implementing the spec test interop with each other? Having some initial transport mechanism seems necessary.

I recognize alternative image transport mechanisms are likely to appear but starting with HTTP seems reasonable given nearly 100% of the images today are transported this way and the intention of this spec project is to start from known working systems with an eye towards upgrades.

Brandon

[Vincent Batts]

On 25/02/16 22:23 -0800, Diogo M??nica wrote:
>Like Trevor said: if we have an image format that is content addressable,
>having the hash is enough to build strong security.
>
>In fact, Docker Content Trust works exactly that way: we providing a secure
>mapping between a tag and a hash, and then ensure the content that is
>downloaded follows the Merkle DAG.

This stills relies on pointers or reference. Otherwise creator identity
and federation will be lost to an upper level implementation that is
looking to use the distributable image.

>The fact that the image can be distributed is key, not the method in
>> in which it is distributed, so I don't see why you would worry about
>> specifics there.
>
>
>I agree that the image being distributable is key, and that the method in
>which it is distributed doesn't matter. That is exactly why I'm pushing
>back on the "Distribution that is based on HTTPS" that is included in the
>proposal.
>
>Same thing for naming. I'm not opposed to adding this optional layer to the
>OCI spec, but I don't think a specific mechanism based on DNS is something
>we should do right now, definitely not before the spec 1.0 comes out and
>not before we actually have agreement on what the distribution format looks
>like.
>
>Just my 2c.

DNS and the like (i even group the java package style naming in here)
are needed. This is how upper level implementations will build out
indexes, mirrors, etc. Discovery is not solely limited and provided in a
checksum.

>On Thu, Feb 25, 2016 at 10:05 PM, W. Trevor King <wk...@tremily.us> wrote:
>
>> On Fri, Feb 26, 2016 at 05:41:49AM +0000, Greg KH wrote:
>> > Don't "tack on" security and naming for later, that way is madness…
>>
>> If the image format is built on cryptographic hashes for
>> content-addressable chunks, then tacking on security and naming seems
>> feasible to me (e.g. Git tacks on security and naming with signed
>> tags, and the only quibble I've ever heard about that approach is that
>> SHA-1 isn't strong enough).
>>
>> Cheers,
>> Trevor
>>
>> --
>> This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
>> For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
>>
>
>
>
>--
>Diogo Mónica
>

[Vincent Batts]

tl;dr; I agree with this proposal

It is a concern that a new project could introduce delay, but the
Roadmap provided falls inline with the one we were just able to settle
on for opencontainers/specs that includes having an image format for
1.0. The biggest delays being largely logistical.
I really go back and forth on whether this ought to be in the specs
project, but despite the charter stating values of portability and
decentralization, it's only been the recent face-2-face that we could
get agreement to include anything about a distributable format. That is
tough progress. Having a separation of focus could alleviate this.

I feel that part of this ought to be renaming the project
"opencontainers/specs" to "opencontainers/runtime-spec", as this proposal
for project would be something like "opencontainers/distribution-spec"

Between the direction that
https://github.com/opencontainers/specs/pull/293 is headed, and docker
image format v2.2
https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md,
these two are not so far stretched from each other as is, save having
name/reference/pointer available for each typed hash object.

> -

>
> Image that is content addressable

> -

>
> Security that is based on signing image content address

> -

>
> Naming that is federated based on DNS (enables ec2 cr, gcr, quay.io,
> localhost registry, etc) and can be delegated

> -

> -
>
> March ?? v0.0.0
> -
>
> Import Docker v2.2 format
> -
>
> April 4th v0.1.0
> -

>
> Spec factored for top to bottom reading with three audiences in-mind:

> -
>
> Build system creators
> -
>
> Image registry creators
> -
>
> Container engine creators
> -

>
> Test cases for each of the potential users exercising build, push,
> and pull verbs

> -
>
> April 18th v0.2.0
> -

>
> Release version of spec with improvements from two independent
> experimental implementations from OCI members e.g. Amazon Container
> Registry and rkt

> -
>
> May 16th v1.0.0
> - Release initial version of spec with two independent non-experimental
> implementations from OCI members
>

[W. Trevor King]

On Tue, Mar 01, 2016 at 04:24:38PM -0500, Vincent Batts wrote:
> On 25/02/16 22:23 -0800, Diogo M??nica wrote:
> >> The fact that the image can be distributed is key, not the method
> >> in in which it is distributed, so I don't see why you would worry
> >> about specifics there.
> >
> > I agree that the image being distributable is key, and that the
> > method in which it is distributed doesn't matter. That is exactly
> > why I'm pushing back on the "Distribution that is based on HTTPS"
> > that is included in the proposal.
> >
> > Same thing for naming. I'm not opposed to adding this optional
> > layer to the OCI spec, but I don't think a specific mechanism
> > based on DNS is something we should do right now, definitely not
> > before the spec 1.0 comes out and not before we actually have
> > agreement on what the distribution format looks like.
> >
> > Just my 2c.
>
> DNS and the like (i even group the java package style naming in
> here) are needed. This is how upper level implementations will build
> out indexes, mirrors, etc. Discovery is not solely limited and
> provided in a checksum.

Discovery (going from a human-oriented name like
example.com/org/app:v1.0.0 [1] or from some other metadata to a signed
name/content-addressable-ID association) seems completely orthogonal
to handling the content-addressable image itself. The code behind:

$ docker run example.com/org/app:v1.0.0

would likely hit some user-configured registries (or a well-known
example.com URL) to retrieve a signed tag where Alice asserts
example.com/org/app:v1.0.0 is sha-256:01ba4719…. From that point on,
it's just fetching content-addressable chunks, starting with
sha-256:01ba4719…. So you'd need:

1. An image-format proposal that focuses on the content-addressable
Merkle tree (going between hashes like sha-256:01ba4719… and an
unpacked OCI bundle). In IPFS, this is /ipfs [2].

2. A tag-registry proposal that focuses on going between
human-readable names like example.com/org/app:v1.0.0 and signed
tags. In IPFS, this is /inps [2].

What do you gain by leaking additional information between those two
components?

Cheers,
Trevor

[1]: https://groups.google.com/a/opencontainers.org/forum/#!topic/tob/WXk1uTgfXrs
Subject: Proposal for a new project: OCI Image Format Spec
Date: Fri, 26 Feb 2016 00:33:07 +0000
Message-ID: <CAD2oYtPCZRUXOER-Fz7WKsnNwYA-PoPmCG59jD1i3=xgYn...@mail.gmail.com>
[2]: https://github.com/ipfs/papers/raw/master/ipfs-cap2pfs/ipfs-p2p-file-system.pdf
This is a bit stale, but it covers the content-addressable
vs. mutable split well. See §3.7 for IPNS and mutable data, and
§3.5 for the immutable Merkle DAG.

[Jason Bouzane]

On Tue, Mar 1, 2016 at 2:00 PM, W. Trevor King <wk...@tremily.us> wrote:
> On Tue, Mar 01, 2016 at 04:24:38PM -0500, Vincent Batts wrote:
>> On 25/02/16 22:23 -0800, Diogo M??nica wrote:
>> >> The fact that the image can be distributed is key, not the method
>> >> in in which it is distributed, so I don't see why you would worry
>> >> about specifics there.
>> >
>> > I agree that the image being distributable is key, and that the
>> > method in which it is distributed doesn't matter. That is exactly
>> > why I'm pushing back on the "Distribution that is based on HTTPS"
>> > that is included in the proposal.
>> >
>> > Same thing for naming. I'm not opposed to adding this optional
>> > layer to the OCI spec, but I don't think a specific mechanism
>> > based on DNS is something we should do right now, definitely not
>> > before the spec 1.0 comes out and not before we actually have
>> > agreement on what the distribution format looks like.
>> >
>> > Just my 2c.
>>
>> DNS and the like (i even group the java package style naming in
>> here) are needed. This is how upper level implementations will build
>> out indexes, mirrors, etc. Discovery is not solely limited and
>> provided in a checksum.
>
> Discovery (going from a human-oriented name like
> example.com/org/app:v1.0.0 [1] or from some other metadata to a signed
> name/content-addressable-ID association) seems completely orthogonal
> to handling the content-addressable image itself.

That's true, and it's useful to be able to have a cryptographically
secure attestation that a particular human-readable name is intended
to map to a particular content-addressable digest. However, I don't
see how that in any way obviates the need for DNS (or similar) or
disputes anything Vincent was saying. The DNS address still needs to
be part of the human readable tag, and it would be useful if a tag
specified where the image corresponding to the named digest could be
found (even though the actual location wouldn't be part of any signed
attestation since any location is as good as any other, and the bundle
could move).

[Diogo Mónica]

Again, just like Trevor said: DNS is completely orthogonal to handling the content-addressable image itself.

At this point we should be discussing the content-addressable, distributable format. Trying to tackle everything in one go, even before we even finished the spec 1.0, seems like a losing proposition to me.

--

Diogo Mónica

[Jason Bouzane]

On Tue, Mar 1, 2016 at 6:01 PM, Diogo Mónica <diogo....@docker.com> wrote:
> Again, just like Trevor said: DNS is completely orthogonal to handling the
> content-addressable image itself.

And again, I don't see how that's true. Unless you assume that the
bundle is always available via the same named entity as is present in
the human readable tag. E.g. if I tag something as
www.example.org/pony-server/1.0, and I get back the digest 0xaabbccdd,
where do I go to find the bundle that corresponds to that digest?
www.example.org? The issues only seem orthogonal if you make some
limiting assumptions about how discovery works.

> At this point we should be discussing the content-addressable, distributable
> format. Trying to tackle everything in one go, even before we even finished
> the spec 1.0, seems like a losing proposition to me.

I don't think a distributable format is especially useful if you've
defined no actual method of distribution and discovery. I suppose it
helps in the sense that if I'm wiling to write interop layers to go
grab my bundle and all of its references from one system like EC2 and
then shove them into another, like Azure or GCE, using whatever
proprietary methods of distribution they've defined, I have some
guarantee that they will understand the bundles, even if I have to
code to three different protocols to move a bundle between them. But
it seems to me much more useful if I can send the bundle to each of
them the same way, or, better yet, upload the bundles to a single
repository and reference the bundles from any of the three just by
giving them the DNS name of the service that's storing them.

[Diogo Mónica]

And again, I don't see how that's true. Unless you assume that the
bundle is always available via the same named entity as is present in
the human readable tag. E.g. if I tag something as
www.example.org/pony-server/1.0, and I get back the digest 0xaabbccdd,
where do I go to find the bundle that corresponds to that digest?
www.example.org? The issues only seem orthogonal if you make some
limiting assumptions about how discovery works.

What I think we're defining is: given a content addressable, transportable, cross-platform blob of data and a hash Y, I both know how to verify H(blob) == Y and how to run blob.

The Docker v2.2 manifest format was mentioned multiple times as something we could potentially adopt, and I would like to point out that it has no naming or transport assumptions built into it: https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md

 

I don't think a distributable format is especially useful if you've
defined no actual method of distribution and discovery. I suppose it
helps in the sense that if I'm wiling to write interop layers to go
grab my bundle and all of its references from one system like EC2 and
then shove them into another, like Azure or GCE, using whatever
proprietary methods of distribution they've defined, I have some
guarantee that they will understand the bundles, even if I have to
code to three different protocols to move a bundle between them. But
it seems to me much more useful if I can send the bundle to each of
them the same way, or, better yet, upload the bundles to a single
repository and reference the bundles from any of the three just by
giving them the DNS name of the service that's storing them.

In my view a "guarantee that they will understand the bundles", or phrased differently, the ability to have cross platform bundles, is absolutely something that is useful. Not only that, it seems to be exactly what we've defined as the mission for the OCI. From our charter:

While the mission of OCI is not to create a complete stack, the format and runtime will:

- Provide a container format and runtime specification that enables portability across compliant runtimes

- Provide a robust stand-alone runtime that can directly consume the specification and run a container

--

Diogo Mónica

[W. Trevor King]

On Tue, Mar 01, 2016 at 07:00:23PM -0800, Diogo Mónica wrote:
> > And again, I don't see how that's true. Unless you assume that the
> > bundle is always available via the same named entity as is present
> > in the human readable tag. E.g. if I tag something as
> > www.example.org/pony-server/1.0, and I get back the digest
> > 0xaabbccdd, where do I go to find the bundle that corresponds to
> > that digest? www.example.org? The issues only seem orthogonal if
> > you make some limiting assumptions about how discovery works.

In IPFS, you'd query a global distributed hash table (DHT) to find
nodes carrying a given hash. After successfully retrieving the
associated blob from one of those nodes, you'd optimistically ask that
node for any related objects, going back to the DHT for hashes the
node could not resolve [1].

In Git, you explicitly tell it which remotes to fetch from, and Git
has a number of protocols built in and an extension mechanism for
adding more [2].

So sure, you need to know something about your content-addressable
storage (CAS) to have a working tag ↔ unpacked bundle component (the
(un)bundler?). But I don't see how details of the CAS implementation
would impact the (un)bundler. Why not standardize an API like [3] and
have folks plug in whatever they like? The language doesn't matter.
Git uses separate processes for easy, language-agnostic binding [2],
but there's not much going on here (as far as the API consumer cares)
so it's easy to wrap if the canonical API ends up being Go or C or
some such and your client is using something else. Then you just have
tag registries return:

* A tag (e.g. “example.com/org/app:v1.0.0 is sha-256:01ba4719…”).
* Signatures on that tag (e.g. [“OpenPGP: says Alice”, “X.509: says
Bob”, …]).
* A CAS-options hint (e.g. “oci-v1://www.example.com/cas”) to feed in
here [4], selecting the protocol and seeding the protocol with any
needed non-CAS information). Clients would be free to ignore the
hint (or use it as a fallback) and look in other places (like a
local object store).

> What I think we're defining is: given a content addressable,
> transportable, cross-platform blob of data and a hash Y, I both know
> how to verify H(blob) == Y and how to run blob.
>
> The Docker v2.2 manifest format was mentioned multiple times as
> something we could potentially adopt, and I would like to point out
> that it has no naming or transport assumptions built into it:
> https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md

+1 to all of that. And with a pluggable CAS layer, we can just use a
simple local store (like Git's loose objects) for conformance-testing
the (un)bundler.

Cheers,
Trevor

[1]: Or something like that. I never worked on that code, and it's
been a while since I talked to folks who had.
[2]: https://git.kernel.org/cgit/git/git.git/tree/Documentation/gitremote-helpers.txt?h=v2.7.2
[3]: https://www.npmjs.com/package/content-addressable-blob-store#api
[4]: https://www.npmjs.com/package/content-addressable-blob-store#var-store--blobsopts

[Vincent Batts]

On 01/03/16 21:29 -0800, W. Trevor King wrote:
>On Tue, Mar 01, 2016 at 07:00:23PM -0800, Diogo Mónica wrote:
>> > And again, I don't see how that's true. Unless you assume that the
>> > bundle is always available via the same named entity as is present
>> > in the human readable tag. E.g. if I tag something as
>> > www.example.org/pony-server/1.0, and I get back the digest
>> > 0xaabbccdd, where do I go to find the bundle that corresponds to
>> > that digest? www.example.org? The issues only seem orthogonal if
>> > you make some limiting assumptions about how discovery works.
>
>In IPFS, you'd query a global distributed hash table (DHT) to find
>nodes carrying a given hash. After successfully retrieving the
>associated blob from one of those nodes, you'd optimistically ask that
>node for any related objects, going back to the DHT for hashes the
>node could not resolve [1].

Regardless, even for IPFS you still must resolve the content to a node.
Arguing against HTTP doesn't exclude DNS.

[W. Trevor King]

On Wed, Mar 02, 2016 at 10:02:22AM -0500, Vincent Batts wrote:
> On 01/03/16 21:29 -0800, W. Trevor King wrote:
> >On Tue, Mar 01, 2016 at 07:00:23PM -0800, Diogo Mónica wrote:
> >> > And again, I don't see how that's true. Unless you assume that
> >> > the bundle is always available via the same named entity as is
> >> > present in the human readable tag. E.g. if I tag something as
> >> > www.example.org/pony-server/1.0, and I get back the digest
> >> > 0xaabbccdd, where do I go to find the bundle that corresponds
> >> > to that digest? www.example.org? The issues only seem
> >> > orthogonal if you make some limiting assumptions about how
> >> > discovery works.
> >
> > In IPFS, you'd query a global distributed hash table (DHT) to find
> > nodes carrying a given hash. After successfully retrieving the
> > associated blob from one of those nodes, you'd optimistically ask
> > that node for any related objects, going back to the DHT for
> > hashes the node could not resolve [1].
>
> Regardless, even for IPFS you still must resolve the content to a
> node. Arguing against HTTP doesn't exclude DNS.

I'm not arguing against HTTP, I'm arguing for a pluggable API for the
content-addressable functionality. DNS is one way to resolve content
to a node. A DHT is another way, and I'm sure folks have dreamed up
lots of other ways to do that sort of thing. Why would a the CAS
client care about which ones the CAS engine was using?

Cheers,
Trevor

[Jason Bouzane]

On Wed, Mar 2, 2016 at 9:30 AM, W. Trevor King <wk...@tremily.us> wrote:
> I'm not arguing against HTTP, I'm arguing for a pluggable API for the
> content-addressable functionality. DNS is one way to resolve content
> to a node. A DHT is another way, and I'm sure folks have dreamed up
> lots of other ways to do that sort of thing. Why would a the CAS
> client care about which ones the CAS engine was using?

Nobody is saying it should. What we're saying is that if you know the
target node supports HTTPS, you can fetch the bundle for a particular
digest in a standard, well known way. Having additional methods of
fetching that data, or checking a local cache before going to the
remote server are both things that an implementation could provide.

There's a difference between saying HTTPS access to bundles must be
provided in a particular way and saying that's the only way those
bundles can be fetched.

[W. Trevor King]

The distinction is in where you draw boundaries for compliance and the
project boundaries those compliance boundaries encourage. If you
require all (un)bundler implementations to support a particular
CAS-over-HTTPS protocol, that makes it harder to test (because you
have to mock DNS/HTTPS) and sets folks up to duplicate code (unless
they all decide to collaborate on a CAS-over-HTTPS library). And it
makes plugging a generic (un)bundler into your alternative CAS
implementation a one-off integration.

If, on the other hand, you require all (un)bundler implementations to
consume a particular CAS API, you get easy testing (plug in a simple,
local CAS implementation), easy sharing (you only need a single
CAS-over-HTTP implementation for all (un)bundlers), and efficient
integrations (your alternative CAS implementation automatically works
with all (un)bundlers).

So the final (un)bundler + CAS-over-HTTPS stack is going to be the
same either way, but the individual components are more flexible if
you standardize on the CAS API. I don't see an implementation cost to
that standardization. Do you? If we leave the CAS API up to the
(un)bundler implementations, what sort of efficiencies do you gain to
offset the loss of drop-in composability?

[Jason Bouzane]

On Wed, Mar 2, 2016 at 8:24 PM, W. Trevor King <wk...@tremily.us> wrote:
> The distinction is in where you draw boundaries for compliance and the
> project boundaries those compliance boundaries encourage. If you
> require all (un)bundler implementations to support a particular
> CAS-over-HTTPS protocol, that makes it harder to test (because you
> have to mock DNS/HTTPS) and sets folks up to duplicate code (unless
> they all decide to collaborate on a CAS-over-HTTPS library). And it
> makes plugging a generic (un)bundler into your alternative CAS
> implementation a one-off integration.
>
> If, on the other hand, you require all (un)bundler implementations to
> consume a particular CAS API, you get easy testing (plug in a simple,
> local CAS implementation), easy sharing (you only need a single
> CAS-over-HTTP implementation for all (un)bundlers), and efficient
> integrations (your alternative CAS implementation automatically works
> with all (un)bundlers).
>
> So the final (un)bundler + CAS-over-HTTPS stack is going to be the
> same either way, but the individual components are more flexible if
> you standardize on the CAS API. I don't see an implementation cost to
> that standardization. Do you? If we leave the CAS API up to the
> (un)bundler implementations, what sort of efficiencies do you gain to
> offset the loss of drop-in composability?

This all seems like it's getting very far off topic, and I'm not sure
where you're even going with this. Absolutely none of what you say is
precluded by anything said in Brandon's email or any of the followups.
A standardized CAS API with an HTTPS wrapper is a perfectly good way
of achieving standard compliance (and could, in fact, be part of the
standard). All that's been said in Brandon's short write up is that
HTTPS is one way the CAS data must be exported.

Diogo's objection (as I understand it; please correct me if I'm wrong)
was that any kind of transport standard is not useful enough to
justify the additional scope and effort involved in standardizing it.
I've been suggesting ways that the transport standard might be useful
and you seem to be objecting that you don't like my examples, but
prefer another one where a transport protocol is still required for it
to work.

So in an effort to get this back on topic, I will concede that you
have provided another example where a standardized bundle
transportation mechanism would be useful, and that standardizing a CAS
access API might be a useful thing to do. But that's beside the point
of whether a transportation mechanism should be part of our
standardization efforts at all.

[W. Trevor King]

On Wed, Mar 02, 2016 at 08:58:28PM -0800, Jason Bouzane wrote:
> > So the final (un)bundler + CAS-over-HTTPS stack is going to be the
> > same either way, but the individual components are more flexible
> > if you standardize on the CAS API. I don't see an implementation
> > cost to that standardization. Do you? If we leave the CAS API up
> > to the (un)bundler implementations, what sort of efficiencies do
> > you gain to offset the loss of drop-in composability?
>
> This all seems like it's getting very far off topic, and I'm not
> sure where you're even going with this.

If we standardize on the CAS API, we can punt on the HTTPS API for
now. That lets us specify, write, and compliance-test (un)bundlers
without having to talk about a HTTPS API.

> Absolutely none of what you say is precluded by anything said in
> Brandon's email or any of the followups.

A spec that requires a particular HTTPS CAS protocol precludes a
conformant (un)bundler based on a local CAS engine.

> Diogo's objection (as I understand it; please correct me if I'm
> wrong) was that any kind of transport standard is not useful enough
> to justify the additional scope and effort involved in standardizing
> it.

… as part of the bundler component. I think Diogo and I agree that
it's worth standardizing any CAS protocols, but that there's no
benefit and some cost to biting off all the components (the manifest
format and (un)bundler, the CAS protocol, the tag format and registry,
and supported signature algorithms) in a single OCI Project / spec /
runtime / test-suite [1]. So “let's separate orthogonal concerns, and
here are some clean boundaries”, not “those aren't concerns”.

> I've been suggesting ways that the transport standard might be
> useful and you seem to be objecting that you don't like my examples,
> but prefer another one where a transport protocol is still required
> for it to work.

*A* transport protocol is required for the (un)bundler to work, I just
don't see what the (un)bundler project / spec / runtime / test-suite
gains by requiring a specific one, when it seems like a separable
concern.

> So in an effort to get this back on topic, I will concede that you
> have provided another example where a standardized bundle
> transportation mechanism would be useful, and that standardizing a
> CAS access API might be a useful thing to do. But that's beside the
> point of whether a transportation mechanism should be part of our
> standardization efforts at all.

But it's on point for “how tightly scoped is the OCI Image Format Spec
and its shepherd OCI Project and controlling TDC going to be?”. If a
CAS-over-HTTPS protocol and a tag registry are goals for 1.0, I'd
rather see them spun off as separate specs. They're all distribution
related, so I'm fine having them all live under a distribution
Project:

* Linux Foundation Board of Directors
* OCI TOB
* Runtime TDC
* projects
* runC (opencontainers/runC)
* runtime spec
* Distribution TDC
* projects
* (un)bundler specification (Merkle bundle serialization, CAS
API, (un)bundler API, and test suite)
* (un)bundler implementation
* CAS-over-HTTPS specification and test suite
* CAS-over-HTTPS implementation
* tag format and registry API and test suite
* tag registry implementation
* signature and trust specification and test suite
* signature and trust implementation

With the alternative being Brandon's:

* Linux Foundation Board of Directors
* OCI TOB
* Runtime TDC
* projects
* runC (opencontainers/runC)
* runtime spec
* Distribution TDC
* projects
* OCI Image Format Spec (Merkle bundle serialization,
CAS-over-HTTPS protocol, tag format, registry API,
signature, and trust specification and test suite).
* OCI Image implementation

I think the former gives a number of focused projects where each
component can easily be swapped out to cheaply build alternative
implementations. I think the latter gives a single, vertically
integrated project where customization involves forking an existing
runtime and binding your alternative component (e.g. a CAS engine)
onto unspecified internal APIs.

Cheers,
Trevor

[1]: https://groups.google.com/a/opencontainers.org/d/msg/tob/WXk1uTgfXrs/ynB3v5JDCQAJ
Subject: Re: [oci-tob] Proposal for a new project: OCI Image Format Spec
Date: Tue, 1 Mar 2016 18:01:24 -0800
Message-ID: <CA+q1=fToD37ZE7mcH2ssSS86mD9muS5JjV89SEF=xih9A...@mail.gmail.com>

[DL duglin]

Irrespective of the location of where this work is done (new WG or existing one), I’d like to see what the initial PR would look like that would kick off this work. This discussion has many twists and turns and I find it much easier to discuss things when people are focused on the exact wording is being proposed rather than more abstract ideas. Brandon has suggested we start with Docker’s v2.2.0 image format - so let’s see a branch (or PR in existing wg) with what that would look like plus any additional features that he thinks needs to be there - a complete proposal.

-Doug

[Chris Aniszczyk]

As an LF representative here, I suggest adding the "project structure" to the agenda for tomorrow's TOB meeting. From my understanding, the original intention was to have one TDC with multiple OCI projects. Each project can have their a) own list of maintainers or b) one list of maintainers across all OCI projects. The charter can be modified to support what option people prefer.

Is everyone OK with adding that as a topic?

--
You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.

--

Chris Aniszczyk (@cra) | +1-512-961-6719

[W. Trevor King]

On Thu, Mar 03, 2016 at 11:22:21AM -0600, Chris Aniszczyk wrote:
> As an LF representative here, I suggest adding the "project

> structure" to the agenda for tomorrow's TOB meeting…

I think that's [1]. But yeah, it's hard to figure out how a
distribution project should fit into the organization if we aren't
clear on the organization structure ;).

Cheers,
Trevor

[1]: https://groups.google.com/a/opencontainers.org/forum/#!topic/tob/1pgiEffnu0M
Subject: [oci-tob] Clarification of TDC and TDC Maintainers
Date: Thu, 25 Feb 2016 05:42:17 +0000
Message-ID: <CAD2oYtNMrLxrUmfrNo2LLBgD...@mail.gmail.com>

[Brandon Philips]

On Thu, Mar 3, 2016 at 9:22 AM Chris Aniszczyk <canis...@linuxfoundation.org> wrote:

As an LF representative here, I suggest adding the "project structure" to the agenda for tomorrow's TOB meeting. From my understanding, the original intention was to have one TDC with multiple OCI projects. Each project can have their a) own list of maintainers or b) one list of maintainers across all OCI projects. The charter can be modified to support what option people prefer.

Option b with projects having their own list of maintainers is what we are doing today and that should continue. We had this discussion during formation as well so whatever we need to do to make it clear makes sense to me.

Here is what we are doing today:

https://github.com/opencontainers/runc/blob/master/MAINTAINERS

https://github.com/opencontainers/specs/blob/master/MAINTAINERS

 

Is everyone OK with adding that as a topic?

Discussing the topic is fine but if we can do this async it is better.

I will send out a separate thread to the TOB list with: vote agree/disagree that we create separate TDC maintainers for the different projects (status quo).

Brandon

[Chris Aniszczyk]

We can move that structure discussion here, would be great to get input from the TDC here.

https://github.com/opencontainers/tob/issues/2

In my opinion, approving a project is possible and can be done in parallel with the structure clarifications, but it's all open to discussion :)

[ovzx...@gmail.com]

I agree with the proposal.

The only thing that is still unclear from the description is the connection of runtime-spec with the shipping format.

-- Pavel

пятница, 26 февраля 2016 г., 3:33:17 UTC+3 пользователь Brandon Philips написал:

TOB Members-

Please read this proposal based on our call. We are all very busy. So, at a minimum please reply with one of three options:

- Agree with this proposal
- Disagree with this proposal
- Lets discuss, I need more information

Also, it looks like our next meeting is centering around Mar 4th: http://doodle.com/poll/9gdepaaygtcvidrt

Thank You,

Brandon

OCI Image Format Spec

New project proposal: The TOB will create a new OCI project tasked with creating a software shipping container image format spec (OCI Image Format) with security, naming, and distribution. In addition the OCI TOB will establish a new set of maintainers for this new project with people who have expertise in image formats, transport mechanisms, and package management.

Initial Recommendation: Over the last 16 months we have seen evolution of container formats towards solid technical underpinnings. When the AppC project was introduced in December 2014 its goals were to create an open specification shared between implementations while addressing concerns in the Docker v1 image format around content addressable images, signing, and a federated/delegatable namespace. Today, the Docker v2.2 image format is close to having all of the desirable traits of AppC while having widespread registry implementations in projects/products from Amazon, Google, CoreOS, Docker, Huawei, and JFrog.

This new OCI project would be recommended to start with the Docker v2.2 specification, improve any remaining technical concerns, and create an OCI project and maintainers to develop and shepard an OCI Image Fromat Spec. By starting from this project we intend to standardize and improve the understood properties of a container image format, namely:

• Image that is content addressable

• Security that is based on signing image content address

• Naming that is federated based on DNS (enables ec2 cr, gcr, quay.io, localhost registry, etc) and can be delegated

• Distribution that is based on HTTPS and enables other protocols (e.g. dockyard bittorrent p2p)

Initial Maintainers: to be brainstormed on a separate thread.

Cooperation with OCI Runtime Project: The OCI Runtime Specs project is working diligently to create a specification for the lifecycle of a running container. The OCI Image Format Spec project should work with the OCI Runtime Spec project so that the an image can support the UX that users have come to expect from container engines like Docker and rkt. Namely the ability to pull human names with implied context (OS, arch, etc) and have them run with no additional arguments:

   docker run example.com/org/app:v1.0.0

   rkt run example.com/org/app,version=v1.0.0

This implies that the Software Shipping Container Spec contain sufficient information to launch the application on the target platform e.g. command, arguments, environment variables, etc.

FAQ

Q: Why a new project?

A: The first OCI spec centered around defining the run side of a container. This is generally seen to be an orthogonal concern to the shipping container component. As practical examples of this separation you see many organizations separating these concerns into different teams and organizations: the Docker Distribution project and the Docker containerd project; Amazon ECS and Amazon EC2 Container Registry, etc.

Q: Why start this work now?

A: We are seeing many independent implementations of container image handling including build systems, registries, and image analysis tools. As an organization we would like to encourage this growth and bring people together to ensure a technically correct and open specification continues to evolve reflecting the OCI values.

Q: What happens to AppC or Docker Image Formats?

A: Existing formats can continue to be a proving ground for technologies, as needed. For example, in line with the OCI values, we expect mechanisms like the AppC name delegation will help inspire portions of the OCI Software Shipping Container project. The OCI Image Format project should strive to provide a dependable open specification that can be shared between different tools and be evolved for years or decades of compatibility; as the deb and rpm format have.

Proposed Roadmap

• March ?? v0.0.0

• Import Docker v2.2 format

• April 4th v0.1.0

• Spec factored for top to bottom reading with three audiences in-mind:

• Build system creators

• Image registry creators

• Container engine creators

• Test cases for each of the potential users exercising build, push, and pull verbs

• April 18th v0.2.0

• Release version of spec with improvements from two independent experimental implementations from OCI members e.g. Amazon Container Registry and rkt

• May 16th v1.0.0

[Stephen Walli]

I like Trevor's structural project breakdown as it provides a crisp way for me to think about the discussions. I'm concerned only in that by creating a highly modular approach to the problems the specification working group(s) need to specify, we drift towards a situation where two conforming implementations of the collected specifications will be incapable of supporting the cross-implementation container portability that is the overall goal of the project. (The next logical step standards groups take when they encounter such situations is to take more time and create standardized profiles.) 

From that perspective I lean towards the way Trevor articulated Brandon's proposal. By narrowing the choices to a single guaranteed specification thread through the technology choices, I believe we arrive at a testable specification sooner, while leaving lots of room for product implementation innovation in the market.  By all means use Trevor (and Diogo's) observations to inform and improve the way you build the single specification. 

I offer this commentary as a member of the OCI Trademark Board. In the past, I've built standards, built conforming product implementations of those standards, and survived the certifications and branding programs that were in place around them.  

I may well be over-reading the situation here. 

my two Canadian cents,

stephe

--

You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.

--

Stephen R. Walli
mailto:   stephe...@gmail.com
mobile:   +1 425 785 6102 
blogs:    http://stephesblog.blogs.com  (Once More unto the Breach)
          https://opensource.com/user_articles/16271 (opensource.com)
Flickr:   http://www.flickr.com/photos/stephenrwalli/
LinkedIn: http://www.linkedin.com/in/stephenrwalli
Twitter:  @stephenrwalli

[Chris Wright]

* Brandon Philips (brandon...@coreos.com) wrote:
> TOB Members-
>
> Please read this proposal based on our call. We are all very busy. So, at a
> minimum please reply with one of three options:
>
> - Agree with this proposal

[arnaud....@docker.com]

On Thursday, February 25, 2016 at 4:33:17 PM UTC-8, Brandon Philips wrote:

New project proposal: The TOB will create a new OCI project tasked with creating a software shipping container image format spec (OCI Image Format) with security, naming, and distribution. In addition the OCI TOB will establish a new set of maintainers for this new project with people who have expertise in image formats, transport mechanisms, and package management.

 While I’m not directly involved in OCI, please find below my input as a consumer of the standard.

1. I share the view that a content-addressable image format is missing from the spec. I however disagree that naming, signing, and distribution mechanism need to be defined as part of OCI, for several reasons.
   
   Our experience at Docker is that naming and signing were part of the original image format and removed in recent iterations. We learned the hard way that these aspects could very well be implemented at the upper layers, allowing both for a better separation of concerns, and for exploration of alternative ways of achieving those goals.
   
   I see in OCI a great way to solidify ideas on which everyone can agree on. As far as content distribution goes, the only aspect on which I believe consensus exists today is that a content-addressable image format is the cornerstone. I would have a hard time getting consensus even among my immediate coworkers for anything above this.
   
   

2. As for specifying a content-addressable image format, I do believe that Docker’s v2.2 manifest can be a good starting point.
   
   

3. My team is currently going through the difficult process of rebasing our container engine on top of OCI. This, to my knowledge, will mark the first widespread deployment of an OCI implementation, and I’m sure we’ll learn a lot from it.
   
   We see so much value in this standard that we didn’t want to wait for a 1.0 of the spec, but I’m very concerned about fragmenting the efforts while the runtime spec is incomplete (most notably still lacking Windows support), and while getting changes through is already pretty slow and complex.

I respectfully suggest to give priority to a complete 1.0 runtime spec, and plan designing a content-addressable image format as a next step.

Thank you all for your hard work on OCI.

--

Arnaud Porterie

@icecrime

[W. Trevor King]

On Fri, Feb 26, 2016 at 12:33:07AM +0000, Brandon Philips wrote:
> This new OCI project would be recommended to start with the Docker

> v2.2 specification… By starting from this project we intend to

> standardize and improve the understood properties of a container
> image format, namely:

> …

> Naming that is federated based on DNS (enables ec2 cr, gcr,
> quay.io, localhost registry, etc) and can be delegated

> …

> Q: What happens to AppC or Docker Image Formats?
>
> A: Existing formats can continue to be a proving ground for
> technologies, as needed. For example, in line with the OCI values,
> we expect mechanisms like the AppC name delegation will help inspire
> portions of the OCI Software Shipping Container project.

I'm trying to unpack Brandon's “federated/delegatable namespace”.
When i talked about tag signing (e.g. in [1]), I'd assumed he meant
something like:

Clients should trust an X.509 signature for
example.com/org/app:v1.0.0 if the signer has a certificate chain
rooted in some trusted certificate authority that asserts the signer
can sign for example.com.

After going through the appc spec, I think Brandon meant something
more central to distribution than a trust framework.

The only content-addressable storage (CAS) reference I can find in the
appc spec is along the lines of (paraphrased) “you can use CAS for
de-duplicating exploded images if you like, and we may incorperate
them in a future streaming format” [2]. Appc sets up layered images
with a dependency list [3,4], but instead of using CAS IDs in that
list like Docker's fsLayers [5,6], appc primarily uses DNS-based names
(dependencies[].imageName) [3,4] with a procedure for retrieving
images based on those names [5,7]. There is an optional
dependencies[].imageID [3,8] which could be used for CAS retrieval,
but there's no standardization around how that is expected to work.

I expect Brandon means appc's DNS-based image lookup [7] when he
refers to DNS delegation (also in [9]), and that doesn't seem to have
had much airtime in these threads. John referenced DNS-based
delegation briefly [10], and Vincent responded with words about
mirroring [11].

One of the benefits of a CAS approach based on cryptographic hashes is
that you don't care where the content comes from. I think any scoping
and spec work on CAS protocols should avoid baking stuff like DNS into
the protocol. Without a single, global CAS registry, folks will still
need some hints (e.g. “look in https+dockerv2://example.com/cas/ or
https+ociv2://docker.com/cas/ for sha-256:01ba4719…” [12]), but I
don't think those hints belong in the CAS objects that make up an
image (manifests, layers, etc. [12]). For example, having external
hints (in the tag registry [12]?) will make it easy for folks to
register additional mirrors of popular content.

Anyhow, if DNS is scoped into the CAS portion of this proposal
(vs. the trust portion), I think it would be good to have more clarity
on what is intended and whether it's a TOB-specified requirement or
just a suggestion to the maintainers who will be working on the
project.

Cheers,
Trevor

[1]: https://groups.google.com/a/opencontainers.org/d/msg/tob/WXk1uTgfXrs/TXBP4e1OCQAJ

Subject: Re: [oci-tob] Proposal for a new project: OCI Image Format Spec

Date: Tue, 01 Mar 2016 21:29:47 -0800
Message-ID: <2016030205...@odin.tremily.us>
[2]: https://github.com/appc/spec/blob/v0.7.4/GUIDE.md#storing-exploded-acis
[3]: https://github.com/appc/spec/blob/v0.7.4/spec/aci.md#image-manifest-schema
[4]: https://github.com/appc/spec/blob/v0.7.4/spec/aci.md#dependency-matching
[5]: https://docs.docker.com/registry/spec/api/#pulling-an-image
[6]: https://github.com/docker/docker/issues/8093#issue-43050417
[7]: https://github.com/appc/spec/blob/v0.7.4/spec/discovery.md
[8]: https://github.com/appc/spec/blob/v0.7.4/spec/types.md#image-id-type
[9]: https://groups.google.com/a/opencontainers.org/d/msg/tob/A3JnmI-D-6Y/JiQmtktCAgAJ
Subject: Re: [oci-tob] Proposal for an OCI Distribution Format Spec
Date: Thu, 10 Mar 2016 02:14:49 +0000
Message-ID: <CAD2oYtPevxNWYvg_z1y11aRjrpxnSrDogGfrW=9=c8syw...@mail.gmail.com>
[10]: https://groups.google.com/a/opencontainers.org/d/msg/tob/V2IkZW-JB0w/IXKYRtUPCgAJ
Subject: Re: [oci-tob] TOB meeting tomorrow (3/3/2016)
Date: Fri, 04 Mar 2016 08:24:50 -0800
Message-ID: <e77df7fc-d09f-49e4...@opencontainers.org>
[11]: https://groups.google.com/a/opencontainers.org/d/msg/tob/V2IkZW-JB0w/GCQnHq6_AgAJ
Subject: Re: [oci-tob] TOB meeting tomorrow (3/3/2016)
Date: Fri, 11 Mar 2016 08:32:40 -0800
Message-ID: <82fc8bd4-e44e-44c9...@opencontainers.org>
[12]: https://groups.google.com/a/opencontainers.org/d/msg/tob/A3JnmI-D-6Y/cbrKQGVOAgAJ
Subject: Re: [oci-tob] Proposal for an OCI Distribution Format Spec
Date: Wed, 09 Mar 2016 21:56:40 -0800
Message-ID: <20160310055...@odin.tremily.us>