Proposal for an OCI Distribution Format Spec

[Diogo Mónica]

TOB Members,
As requested on our last call, here is an alternative proposal for a new project around an OCI Distribution Format.

OCI Distribution Format Spec Proposal

The Open Container Initiative has made significant progress towards the definition of a lightweight, universal runtime container format, and is about one month away from a release candidate version of the spec. When the OCI was first established, its goals were to create an open specification shared between implementations that allowed the distribution and execution of containers; and optional layers defining signing and naming. The OCI was established with a clear charter and with extensively discussed scope, and it is our responsibility as a TOB to govern in a manner consistent with that charter and scope.

       Charter:         https://www.opencontainers.org/governance

Scope:         https://www.opencontainers.org/governance/oci-scope-table

The scope table in particular touches on all four aspects of Brandon’s original proposal, and very explicitly establishes both that we should take a layered approach and that certain layers should be optional or out of scope. Attempting to add too much to the base OCI specs during this innovation phase will slow down the current rate of evolution that some of the layers are undergoing.

This proposal recommends that the TOB continues with the initial plans of creating a software shipping container distribution format spec (OCI Distribution Format) that includes a simple, distributable, filesystem bundle format, compatible with most popular storage and distribution mechanisms, in particular those implementing content-addressable storage such as Docker registry and ipfs.

The recommendation is to start with a simplified version of the Docker v2.2 specification, and create a new OCI project to develop and shepherd an OCI Distribution Format Spec. The objectives of this particular initiative are:

• Define a filesystem bundle that can be serialized and hashed to ensure content integrity, ensuring interoperability between implementations.

• Create a reference spec for an optional layer that describes how to use digital signatures to verify the integrity of the content.

• Create a reference spec for an optional layer that allows DNS based naming.

This proposal is fully in-line with the charter of the OCI initiative as defined in https://www.opencontainers.org/governance, and in particular, it follows directly from the OCI Scope Table that all of the members of the OCI initiative agreed to.

Given the close relationship with the OCI Runtime Spec 1.0, some of the work around defining this OCI Distribution Format should either wait for the Runtime spec to be finalized, or at least be done in a way such that the OCI Distribution Format specification is ensured to be compatible.

Proposed Timeline and Roadmap

• March

• Define the maintainers that will responsible for delivering this spec

• April

• Beginning defining a filesystem bundle format based on a simplified version of the Docker v.2.2 specification

• May

• First complete draft of OCI Distribution Format spec

• June

• Release initial version of OCI Distribution Format spec (This aligns with OCI Runtime Container Format spec v1.0 timing)

Thank you,

--

Diogo Mónica

[W. Trevor King]

On Wed, Mar 09, 2016 at 03:58:00PM -0800, Diogo Mónica wrote:
> Define a filesystem bundle that can be serialized and hashed to
> ensure content integrity, ensuring interoperability between
> implementations.

nit: defining the filesystem bundle is in the scope of the runtime
spec [1]. The distribution format spec should define the
serialization for that filesystem format, and not define the
filesystem format itself. Maybe that's what you're getting at here:

> Given the close relationship with the OCI Runtime Spec 1.0, some of
> the work around defining this OCI Distribution Format should either
> wait for the Runtime spec to be finalized, or at least be done in a
> way such that the OCI Distribution Format specification is ensured
> to be compatible. Proposed Timeline and Roadmap

but I think it would be good to make the line between the specs clear.

Splitting serialization, signing, and naming into separate specs
sounds good to me. Does your naming layer also cover a name-discovery
registry API [2] (e.g. “give me names (and sigatures?) of all images
that contain ‘nginx’ in their name”), or is that still out-of-scope
each of your specs? It looks like the CAS API / protocol is
out-of-scope for all your projects as well, which seemed like a
(optional?) component of Brandon's proposal that folks were committed
to. (Un)bundler implementations will have a few interfaces:

* With the filesystem bundle (defined by the serialization format).
* With the CAS storage (defined by a CAS API or specific CAS
protocol).
* With the (un)bundler caller (e.g. a command-line API).

If we don't specify something for each of those interfaces, I expect
compliance testing will be difficult. They don't all have to be
developed in the same spec/repository, but I think we'll need each of
them somewhere.

Cheers,
Trevor

[1]: https://github.com/opencontainers/specs/blob/v0.3.0/bundle.md
[2]: https://groups.google.com/a/opencontainers.org/d/msg/tob/WXk1uTgfXrs/QscoXgWeCQAJ
Subject: Re: [oci-tob] Proposal for a new project: OCI Image Format Spec
Date: Wed, 2 Mar 2016 21:39:10 -0800
Message-ID: <2016030305...@odin.tremily.us>

--
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[Brandon Philips]

On Wed, Mar 9, 2016 at 11:58 PM Diogo Mónica <diogo....@docker.com> wrote:

When the OCI was first established, its goals were to create an open specification shared between implementations that allowed the distribution and execution of containers; and optional layers defining signing and naming. The OCI was established with a clear charter and with extensively discussed scope, and it is our responsibility as a TOB to govern in a manner consistent with that charter and scope.

The TOB has the ability to change or expand this scope table as needed.

If it is necessary to expand the scope of the OCI table in order to have an end-to-end specification that works for real use cases then we should absolutely do that.

The recommendation is to start with a simplified version of the Docker v2.2 specification, and create a new OCI project to develop and shepherd an OCI Distribution Format Spec. The objectives of this particular initiative are:

• Define a filesystem bundle that can be serialized and hashed to ensure content integrity, ensuring interoperability between implementations.

• Create a reference spec for an optional layer that describes how to use digital signatures to verify the integrity of the content.

• Create a reference spec for an optional layer that allows DNS based naming.

I think there are essentially two differences between this proposal and the one that was approved last week[1]. Please let me know if you think this is accurate.

## No Distribution

This proposal skips any mention of a distribution method.

The approved proposal from last week has the complete Docker v2.2 format, including the HTTP distribution method, as its foundation because it is implemented in 6 or more independent systems today. Starting here gives us something that everyone understands and works well for many members. But, has some things we wish to improve like delegation and other distribution methods.

I understand the concern that distribution via HTTP might not be desirable by all systems. However, seeing as Docker already implements this distribution method (thus agreeing with it), and there exists a wide ecosystem of alternative implementations that implement it as well it feels like it should be included. Would adding a tweak to be more inclusive such as "HTTP distribution and an optional method such as ipfs" be sufficient to get Michael Crosby and you to accept the original proposal?

Our acceptance criteria should be that a software engineer should be able to read the spec from end-to-end and be able to download a container unambiguously. If we don't include any layer that specifies distribution I don't know how we get there.

For example, the Docker v2.2 manifest tells you, essentially, get blob-A, blob-B, and blob-C. If we skip on defining a layer of how an HTTP request must be structured to fetch A, B, and C blobs we are sunk.

## No Delegation

Besides removal of any distribution method this proposal also removes mention of name delegation. E.g. example.com delegates downloads to gcr.io. Why is this something you oppose from the original proposal?

Thank You,

Brandon

[1] https://groups.google.com/a/opencontainers.org/d/msg/tob/WXk1uTgfXrs/-xICOdi1BwAJ

[W. Trevor King]

On Thu, Mar 10, 2016 at 02:14:49AM +0000, Brandon Philips wrote:
> I understand the concern that distribution via HTTP might not be
> desirable by all systems. However, seeing as Docker already
> implements this distribution method (thus agreeing with it), and
> there exists a wide ecosystem of alternative implementations that
> implement it as well it feels like it should be included. Would
> adding a tweak to be more inclusive such as "HTTP distribution and
> an optional method such as ipfs" be sufficient to get Michael Crosby
> and you to accept the original proposal?

If someone builds a system that uses a different distribution
mechanism, and they only include the CAS-over-HTTP protocol for
compliance testing, we'll want to make sure the CAS-over-HTTP protocol
is simple enough that they bother coding it up ;). The current Docker
Registry HTTP API v2 [1] is more than a vanilla CAS API, with separate
endpoints for manifests [2], layers [3], and a repository listing [4].
I see no reason for a CAS engine to know about the details of the
content it stores (who cares if the blob is a manifest or a layer?),
and I see poor separation in mixing CAS content (manifests and layers)
and mutable content (repository listings at a well-known location) in
the same engine [5]. So I'm skeptical that the full Docker Registry
HTTP API v2 is a good place to start for a base layer spec.

It's possible that you could trim down the protocol spec to something
that looks more like vanilla CAS, and things like resumable push/pull
[6,7] are certainly worth keeping if the filestystem ↔ Merkle scheme
remains based on possibly-large tarball objects [8,9]. So this is
just “the Docker Registry HTTP API v2” is a heavy lowest common
denominator for standardization”. (Un)bundler engines that want to
support it should be welcome to do so.

But I'm pretty sure “… and an optional method such as IPFS” is a
no-go. For one thing, IFPS hashes are based on the the protobuf
object, not the raw payload, so you can't just push content with a
given hash into IPFS and get it back using the same hash. More
generally, almost all the things this project tries to do (defining a
filesystem ↔ Merkle mapping [10], defining a CAS protocol [11], and
signed mutable tags [12]) already exist in IPFS. I expect overlap, if
any, will be in registering named tags of all types in tag registries,
so folks can discover entries like:

* “example.com/org/app:v1.0.0 is sha-256:01ba4719… on https+dockerv2”
* “example.com/org/app:v1.0.0 is QmVc6zuAneK… on IPFS”

and clients can pick their favorite protocol from the available set.
That would make it easy for folks to buy into the broader OCI
ecosystem even if they're using images where the layered-tarball
approach is a poor fit [14].

Cheers,
Trevor

[1]: https://docs.docker.com/registry/spec/api/
[2]: https://docs.docker.com/registry/spec/api/#pulling-an-image-manifest
[3]: https://docs.docker.com/registry/spec/api/#pulling-a-layer
[4]: https://docs.docker.com/registry/spec/api/#listing-repositories
[5]: https://github.com/docker/docker-registry/issues/704
Subject: Split streaming, content-addressable storage from
transactional, mutable storage
[6]: https://docs.docker.com/registry/spec/api/#resumable-push
[7]: https://docs.docker.com/registry/spec/api/#resumable-pull
[8]: https://github.com/docker/docker/issues/8093
Subject: Proposal: Provenance step 1 - Transform images for
validation and verification
[9]: https://github.com/opencontainers/specs/pull/293
Subject: Create image.md
[10]: https://github.com/ipfs/specs/tree/master/protocol#41-unixfs----representing-traditional-files
[11]: https://github.com/ipfs/specs/tree/master/libp2p
[12]: https://github.com/ipfs/specs/tree/master/records
[13]: https://github.com/docker/docker/issues/3156#issuecomment-30369989
Subject: host-mounted volumes in Dockerfile's VOLUME

[ovzx...@gmail.com]

I think there are essentially two differences between this proposal and the one that was approved last week[1]. Please let me know if you think this is accurate.

## No Distribution

[snip]

 

For example, the Docker v2.2 manifest tells you, essentially, get blob-A, blob-B, and blob-C. If we skip on defining a layer of how an HTTP request must be structured to fetch A, B, and C blobs we are sunk.

Should the HTTP(s) be enforced in the spec at all? I would expect that the distribution format is protocol-agnostic and allows for any way to actually fetch a blob. Or do you mean, that the HTTP part would be like "IF you chose http as download method THEN the request must be structured this and this"?

-- Pavel

[gossm...@gmail.com]

I have the same question. It seems like different folks have different things in mind when talking about what distribution mechanism means.

[Michael Crosby]

Exmul,

I don't think it should be enforced at all and if we add it to the base spec then that means for anyone to be compliant they have the burden of running some hosted service, with HTTPS, gettings certs, configuring them, etcc..

As users that just want to build and run containers they should be free to fetch the distributed format anyway they feel like without having to stand up infrastructure to be compliant.  Besides it explicitly saying this was out of scope in our charter it does not make sense to enforce something like this.  Users should be free to rsync, scp, HTTP(S), or NFS these containers however they want without this group telling them how to do so. 

We are defining the "what" you move not how to move it.

--
You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.

--

Michael Crosby

@crosbymichael

[Greg KH]

On Thu, Mar 10, 2016 at 10:11:01AM -0800, Michael Crosby wrote:
> Exmul,
>
> I don't think it should be enforced at all and if we add it to the base spec
> then that means for anyone to be compliant they have the burden of running some
> hosted service, with HTTPS, gettings certs, configuring them, etcc..

What? That's crazy talk, where are you getting this from?

If I want to write a program to fetch a container, I need to know how to
fetch the container, and I don't have to run any service.

An example of this would be all of the different implementations out
there today that fetch existing containers, not all of them have any
hosting service at all (systemd is one such example.)

Like Brandon has said, the goal here is to have someone be able to take
the spec and be able to implement everything that is needed to fetch and
run a container. Yes, that implies knowing how to talk to some random
server that hosts the containers, and find the container, but it does
not require that same person to go and buy a server to host them on.

thanks,

greg k-h

[Michael Crosby]

Greg,

What if I'm running my own datacenter/ internal network and I'm not going to use a third party service that hosts containers? Now, I have a lot of infrastructure to setup to just fetch containers.  

--

Michael Crosby

@crosbymichael

[Greg KH]

On Thu, Mar 10, 2016 at 10:49:26AM -0800, Michael Crosby wrote:
> Greg,
>
>
> What if I'm running my own datacenter/ internal network and I'm not going to
> use a third party service that hosts containers? Now, I have a lot of
> infrastructure to setup to just fetch containers.  

If you want to run your own datacenter, wonderful, run one. But don't
somehow claim that all implementors of this spec have to run one, as
that's just wrong, and you know it :)

thanks,

greg k-h

[Michael Crosby]

Greg,

I'm sorry but I think we confused each other.  Let me explain better.

If we say that to be OCI compliant you have to fetch containers via HTTP(S) then people wanting to integrate with the OCI ecosystem ( i.e. use our tools ) are then forced to setup up infrastructure to distribute containers that they produce.  Not all users will be comfortable hosting their containers on external services but the tools being built for the OCI ecosystem will expect containers to be available via HTTP(S).  

So for people wanting to consume the tools that we produce as part of OCI and build their own containers, host them internally, for their own deployments they would then have a hard time because dictating how you distribute something introduces dependencies on required infrastructure, especially with HTTP(S).

When we just spec and define the format/artifact then interop with tool we produce stays without forcing any type of restrictions on the infrastructure a user has to be running to just fetch their own containers.  

--

Michael Crosby

@crosbymichael

[Diogo Mónica]

My proposal is a subset of the proposal that was sent last week, and it is completely in line with the current charter of the OCI. We agreed that that was the objective of this counter-proposal: achieving a distributable format spec w/ consensus from the TOB.

Given that I don't see any consensus, I propose that we delay tomorrow's meeting to next week to let people take a look/discuss it async. I'm afraid I might have sent it too close to Friday to guarantee that all members of the TOB have time to take a look and comment on it.

Cheers,

--

Diogo Mónica

[Greg KH]

On Thu, Mar 10, 2016 at 12:12:05PM -0800, Diogo Mónica wrote:
> My proposal is a subset of the proposal that was sent last week, and it is
> completely in line with the current charter of the OCI. We agreed that that was
> the objective of this counter-proposal: achieving a distributable format spec w
> / consensus from the TOB.
>
> Given that I don't see any consensus, I propose that we delay tomorrow's
> meeting to next week to let people take a look/discuss it async. I'm afraid I
> might have sent it too close to Friday to guarantee that all members of the TOB
> have time to take a look and comment on it.

No, let's not delay, I think we have enough here already, I doubt one
more week is going to change much of anything.

thanks,

greg k-h

[Greg KH]

Ugh, it turns out that a bunch of people are in London this week at
KubeCon so how about we talk it out tomorrow with the people that can
make it, and make something "final" next week when everyone can be
there?

thanks,

greg k-h

[Chris Aniszczyk]

I have a panel/talk that conflicts with this at KubeCon also.

--
You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.

--

Chris Aniszczyk (@cra) | +1-512-961-6719

[Vincent Batts]

On Thu, Mar 10, 2016 at 2:11 PM, Michael Crosby
<michael...@docker.com> wrote:
> Greg,
>
> I'm sorry but I think we confused each other. Let me explain better.
>
> If we say that to be OCI compliant you have to fetch containers via HTTP(S)
> then people wanting to integrate with the OCI ecosystem ( i.e. use our tools
> ) are then forced to setup up infrastructure to distribute containers that
> they produce. Not all users will be comfortable hosting their containers on
> external services but the tools being built for the OCI ecosystem will
> expect containers to be available via HTTP(S).
>
> So for people wanting to consume the tools that we produce as part of OCI
> and build their own containers, host them internally, for their own
> deployments they would then have a hard time because dictating how you
> distribute something introduces dependencies on required infrastructure,
> especially with HTTP(S).
>
> When we just spec and define the format/artifact then interop with tool we
> produce stays without forcing any type of restrictions on the infrastructure
> a user has to be running to just fetch their own containers.

On the contrary, providing an means to distribute, albeit starting
with HTTP(S) a widely known and embraced method, then tool can be more
compatible, there for lowering the overall complexity and need for
folks to stand up custom frameworks (which could be nasty, fragmented
risk).
This does not even enter to datacenter level complexity yet, but that
could be a boundless issue.

> On Thu, Mar 10, 2016 at 11:00 AM, Greg KH <gre...@linuxfoundation.org>
> wrote:
>>
>> On Thu, Mar 10, 2016 at 10:49:26AM -0800, Michael Crosby wrote:
>> > Greg,
>> >
>> >
>> > What if I'm running my own datacenter/ internal network and I'm not
>> > going to
>> > use a third party service that hosts containers? Now, I have a lot of
>> > infrastructure to setup to just fetch containers.
>>
>> If you want to run your own datacenter, wonderful, run one. But don't
>> somehow claim that all implementors of this spec have to run one, as
>> that's just wrong, and you know it :)
>>
>> thanks,
>>
>> greg k-h
>
>
>
>
> --
> Michael Crosby
> @crosbymichael
>

[Vincent Batts]

I don't think that this need delay any decision. It's already been
said that known methods like HTTP(S) would not be exclusive methods.
But there must be an initial method that is common.

vb

>
> thanks,
>
> greg k-h

[Chris Aniszczyk]

Also, we have to be cognizant of (6)(l) in the charter regarding at least two weeks for review and time to explore compromise solutions.

Issues should be referred to the TOB sufficiently in advance of a meeting to provide an appropriate time for TOB members to evaluate the issues, and the positions of the TDC, the positions of users, as well as sufficient time to explore compromise solutions. It is expected an appropriate review should require at least a two-week review period, though it is recognized some timecritical circumstances may call for a shorter review (e.g. security issues).

Brandon also added a note about voting for the TOB's review.
https://github.com/opencontainers/tob/pull/3

My suggestion is to have a discussion tomorrow to further explore consensus.

Is this OK Brandon?

[DL duglin]

hmm if only there were some some kind of string that was able to encode the transport mechanism as part of it.....

-Doug

[Brandon Philips]

On Thu, Mar 10, 2016 at 9:42 PM Chris Aniszczyk <canis...@linuxfoundation.org> wrote:

Also, we have to be cognizant of (6)(l) in the charter regarding at least two weeks for review and time to explore compromise solutions.

Issues should be referred to the TOB sufficiently in advance of a meeting to provide an appropriate time for TOB members to evaluate the issues, and the positions of the TDC, the positions of users, as well as sufficient time to explore compromise solutions. It is expected an appropriate review should require at least a two-week review period, though it is recognized some timecritical circumstances may call for a shorter review (e.g. security issues).

Brandon also added a note about voting for the TOB's review.
https://github.com/opencontainers/tob/pull/3

My suggestion is to have a discussion tomorrow to further explore consensus.

Is this OK Brandon?

I am OK with having a discussion tomorrow to further explore consensus. 

But, as a reminder: two weeks ago today the approved OCI Image Format project proposal[1] was first sent to the TOB after our phone discussion. By our meeting on March 18th we will have been discussing this issue for four weeks[2], and the approved proposal for three.

So, during our upcoming March 18th meeting I believe we have dedicated sufficient time to discuss this topic; particularly in light of having an approved project proposal. My expectation is that on March 18th we should begin the logistical work to start the new OCI Image Format project based on either: a) the approved proposal or b) a compromise proposal that evolves over the coming week.

This Image Format work is critical to the OCI. We should have a robust discussion but also be cognizant that we have a responsibility to make timely progress. Does any TOB member disagree with this timeline?

Thank You,

Brandon

[1] https://groups.google.com/a/opencontainers.org/d/msg/tob/WXk1uTgfXrs/-xICOdi1BwAJ

[2] https://groups.google.com/a/opencontainers.org/d/msg/tob/_MIc_QKHaII/ErPUTCXSAgAJ

[W. Trevor King]

On Thu, Mar 10, 2016 at 10:19:18PM +0000, Brandon Philips wrote:
> My expectation is that on March 18th we should begin the logistical
> work to start the new OCI Image Format project based on either: a)
> the approved proposal or b) a compromise proposal that evolves over
> the coming week.

This logistical work is:

* Create some repositories.
* Document the maintainers of those repositories in the repositories
themselves.
* Document the scope of those repositories in the repositories
themselves.
* Document the new OCI project (and maybe new TDC?) wherever those are
supposed to get documented (opencontainers/tob?) [1,2].

It's easy enough to go through that for the adopted proposal [3], and
make adjustments later as the TOB adjusts the scope (or not). The
bulk of the early work will likely be on the filesystem ↔ CAS mapping,
which everyone seems to agree on. And if DNS naming or some such ends
up being worked on early, it should be easy enough to split it out
into a separate project if/when the TOB decides that's a good idea.

Cheers,
Trevor

[1]: https://groups.google.com/a/opencontainers.org/d/msg/tob/1pgiEffnu0M/RFu1lyKwBwAJ
Subject: Re: [oci-tob] Clarification of TDC and TDC Maintainers
Date: Thu, 25 Feb 2016 14:48:35 -0800
Message-ID: <2016022522...@odin.tremily.us>

[2]: https://groups.google.com/a/opencontainers.org/d/msg/tob/WXk1uTgfXrs/QscoXgWeCQAJ
Subject: Re: [oci-tob] Proposal for a new project: OCI Image Format Spec
Date: Wed, 2 Mar 2016 21:39:10 -0800
Message-ID: <2016030305...@odin.tremily.us>

[3]: https://groups.google.com/a/opencontainers.org/forum/#!msg/tob/BHaVoBWV-5Y/GIKabA0ZCgAJ
Subject: Next Steps: OCI Image Format Project
Date: Fri, 04 Mar 2016 19:13:36 +0000
Message-ID: <CAD2oYtMroMrbM9vcUxMvRqED...@mail.gmail.com>

[Michael Crosby]

How many people are able to attend the meeting tomorrow?  

--

Michael Crosby

@crosbymichael

[Brandon Philips]

I can.

[Sarah Saul]

Chris and I are out tomorrow. 

I can.

--

You received this message because you are subscribed to the Google Groups "Technical Oversight Board" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tob+uns...@opencontainers.org.

--

Sarah Saul

Client Services Manager

The Linux Foundation

(M) 520-245-5185

Skype: srsaul

[Greg KH]

On Thu, Mar 10, 2016 at 03:14:52PM -0800, Michael Crosby wrote:
> How many people are able to attend the meeting tomorrow?  

I will be there.

greg k-h

[Chris Wright]

* Michael Crosby (michael...@docker.com) wrote:
> How many people are able to attend the meeting tomorrow?

I will be there.

[Pavel Emelyanov]

I can't, sorry :( But I don't insist in blocking/removing the https
part from the proposal.

[Vincent Batts]

Schedule change today, for me. I will be able to make the call.

vb

[Brandon Philips]

Did we cancel the call? I had some wifi problem; then I tried to join and no one is on the call.

[Chris Wright]

Yes. W/out one or both of you and Diogo, we weren't making useful
progress. Expecting to try again next week.

[Vincent Batts]

On Fri, Mar 11, 2016 at 12:31 PM, Chris Wright <chr...@redhat.com> wrote:
> Yes. W/out one or both of you and Diogo, we weren't making useful
> progress. Expecting to try again next week.

Or on the email thread :-)

[Brandon Philips]

On Fri, Mar 11, 2016 at 5:34 PM Vincent Batts <vba...@redhat.com> wrote:

On Fri, Mar 11, 2016 at 12:31 PM, Chris Wright <chr...@redhat.com> wrote:
> Yes.  W/out one or both of you and Diogo, we weren't making useful
> progress.  Expecting to try again next week.

Understood. Sorry about that everyone.  

Or on the email thread :-)

I would like to hear from Diogo about the two questions here: https://groups.google.com/a/opencontainers.org/d/msg/tob/A3JnmI-D-6Y/JiQmtktCAgAJ

Thank You,

Brandon

[Diogo Mónica]

Brandon,

You're referring to last week's proposal as the proposal that was approved, but I honestly don't understand how a proposal that includes something that goes explicitly against our charter can even be approved.

From our charter:

If we even want to bring up the possibility of a proposal that allows us to standardize an optional distribution method, then we will have to get our charter changed to allow that optional layer to exist.

As you mentioned, a change requires two thirds vote of the TOB, but it also requires thirty days notice to all OCI members and is subject to veto by the Linux Foundation, before taking effect:

12. Amendments and Notice
a. This Charter may be amended by a two thirds vote of the Technical Oversight Board, subject to veto by The Linux Foundation Board of Directors for reasonable cause, with thirty (30) days’ notice to the OCI Members before taking effect.

I don't think I will personally will ever agree that a distribution in the base layer is a good idea, but if all TOB and OCI Members are ok with a charter change that adds optional HTTP transport as an optional layer, I don't think I would be opposed to that.

As far as naming goes, I included naming in my proposal "Create a reference spec for an optional layer that allows DNS based naming.". The implementation details will probably belong to the TDC that is running with the project.

I hope this answers both your questions.

Cheers,

Diogo Mónica

[Brandon Philips]

On Fri, Mar 11, 2016 at 11:04 AM Diogo Mónica <diogo....@docker.com> wrote:

I don't think I will personally will ever agree that a distribution in the base layer is a good idea, but if all TOB and OCI Members are ok with a charter change that adds optional HTTP transport as an optional layer, I don't think I would be opposed to that.

OK, lets start with the OCI Image Format project not having distribution and then we can work from there.

I will send out an updated version of the proposal later today that simply removes distribution and adds an FAQ items based on your suggestion and then we can work from there on the call tomorrow.

Cheers,

Brandon

[Diogo Mónica]

Great, thank you Brandon.

It would also be good if we adjusted the language of the proposal slightly, to match more closely what we currently have on our governance docs. In particular, we should explicitly point out that the bundle format should be an OCI base-layer.

Thank you for all your work on getting these proposals together and moving forward.

Cheers,

Diogo

--

Diogo Mónica

[Brandon Philips]

On Thu, Mar 17, 2016 at 3:12 PM Diogo Mónica <diogo....@docker.com> wrote:

It would also be good if we adjusted the language of the proposal slightly, to match more closely what we currently have on our governance docs. In particular, we should explicitly point out that the bundle format should be an OCI base-layer.

Diogo- here is an update to the original proposal. Changes include: adding an FAQ about distribution, removal of distribution, and specifying optional/base based on the projects table. LMK how it looks to you and I will start a new thread that we can vote on and discuss if it looks OK.

Thanks,

Brandon

OCI Image Format Spec

New project proposal: The TOB will create a new OCI project tasked with creating a software shipping container image format spec (OCI Image Format) with security and naming as components. In addition the OCI TOB will establish a new set of maintainers for this new project with people who have expertise in image formats and package management.

Initial Recommendation: Over the last 16 months we have seen evolution of container formats towards solid technical underpinnings. When the AppC project was introduced in December 2014 its goals were to create an open specification shared between implementations while addressing concerns in the Docker v1 image format around content addressable images, signing, and a federated/delegatable namespace. Today, the Docker v2.2 image format is close to having all of the desirable traits of AppC while having widespread registry implementations in projects/products from Amazon, Google, CoreOS, Docker, Huawei, and JFrog.

This new OCI project would be recommended to start with the Docker v2.2 specification, improve any remaining technical concerns, and create an OCI project and maintainers to develop and shepherd an OCI Image Format Spec. By starting from this project we intend to standardize and improve the understood properties of a container image format. This new project will have the objectives of specifying the following components:

• A serialized image format (base layer)

• A process of hashing the image format for integrity and content-addressing (base layer)

• Signatures that are based on signing image content address (optional layer)

• Naming that is federated based on DNS and can be delegated (optional layer)

Initial Maintainers: to be brainstormed on a separate thread.

Cooperation with OCI Runtime Project: The OCI Runtime Specs project is working diligently to create a specification for the lifecycle of a running container. The OCI Image Format Spec project should work with the OCI Runtime Spec project so that an image can support the UX that users have come to expect from container engines like Docker and rkt. Primarily the ability to run an image with no additional arguments:

  docker run example.com/org/app:v1.0.0

  rkt run example.com/org/app,version=v1.0.0

This implies that the OCI Image Format contain sufficient information to launch the application on the target platform e.g. command, arguments, environment variables, etc.

FAQ

Q: Why doesn't this project mention distribution?

A: Distribution, for example using HTTP as both Docker v2.2 and AppC do today, is not part of this project initially. This is because the OCI scope table says this is out of scope; but largely the TOB discussion has deemed it is necessary to an optional layer that defines this. To make a practical system that can work end-to-end the TOB is going to have to work quickly and diligently to fix the scope table. See this thread https://groups.google.com/a/opencontainers.org/d/msg/tob/A3JnmI-D-6Y/tLuptPDHAgAJ

Q: Why a new project?

A: The first OCI spec centered around defining the run side of a container. This is generally seen to be an orthogonal concern to the shipping container component. As practical examples of this separation you see many organizations separating these concerns into different teams and organizations: the Docker Distribution project and the Docker containerd project; Amazon ECS and Amazon EC2 Container Registry, etc.

Q: Why start this work now?

A: We are seeing many independent implementations of container image handling including build systems, registries, and image analysis tools. As an organization we would like to encourage this growth and bring people together to ensure a technically correct and open specification continues to evolve reflecting the OCI values.

Q: What happens to AppC or Docker Image Formats?

A: Existing formats can continue to be a proving ground for technologies, as needed. For example, in line with the OCI values, we expect mechanisms like the AppC name delegation will help inspire portions of the OCI Software Shipping Container project. The OCI Image Format project should strive to provide a dependable open specification that can be shared between different tools and be evolved for years or decades of compatibility; as the deb and rpm format have.

Proposed Roadmap

• March ?? v0.0.0

• Import Docker v2.2 format

• April 18th v0.1.0

• Spec factored for top to bottom reading with three audiences in-mind:

• Build system creators

• Image registry creators

• Container engine creators

• Test cases for each of the potential users exercising build, push, and pull verbs

• May 16th v0.2.0

• Release version of spec with improvements from two independent experimental implementations from OCI members e.g. Amazon Container Registry and rkt

• June 13th v1.0.0

• Release initial version of spec with two independent non-experimental implementations from OCI members

[W. Trevor King]

On Thu, Mar 17, 2016 at 10:59:56PM +0000, Brandon Philips wrote:
> … but largely the TOB discussion has deemed it is necessary to an

> optional layer that defines this.

This fails to parse for me. A narrow fix might be:

… but largely the TOB discussion has deemed it is necessary to
define this in an optional layer.

And a broader fix might be:

… but mostly the TOB is still seeking consensus on the right project
groupings to support flexibility while encouraging interoperability.

Cheers,
Trevor

[Brandon Philips]

On Thu, Mar 17, 2016 at 5:54 PM W. Trevor King <wk...@tremily.us> wrote:

On Thu, Mar 17, 2016 at 10:59:56PM +0000, Brandon Philips wrote:
> … but largely the TOB discussion has deemed it is necessary to an
> optional layer that defines this.

This fails to parse for me.  A narrow fix might be:

  … but largely the TOB discussion has deemed it is necessary to
  define this in an optional layer.

Oops, missing a word, tried to rewrite it, how does this hit you:

But, the TOB discussion has deemed it is necessary to add this as an optional layer in the scope table in the near future. To make a practical system that can work end-to-end the TOB is going to have to work quickly and diligently to make this addition to the scope table.

Brandon

[W. Trevor King]

On Fri, Mar 18, 2016 at 01:10:03AM +0000, Brandon Philips wrote:
> But, the TOB discussion has deemed it is necessary to add this as an
> optional layer in the scope table in the near future. To make a
> practical system that can work end-to-end the TOB is going to have
> to work quickly and diligently to make this addition to the scope
> table.

That sounds fine to me, and it seems to match Diogo's acceptable
optional HTTP layer wording in [1].

Cheers,
Trevor

[1]: https://groups.google.com/a/opencontainers.org/d/msg/tob/A3JnmI-D-6Y/tLuptPDHAgAJ
Subject: Re: [oci-tob] Proposal for an OCI Distribution Format Spec
Date: Fri, 11 Mar 2016 11:03:42 -0800
Message-ID: <CA+q1=fTUrVQYuiR6qemhpsUuSQd...@mail.gmail.com>

[ovzx...@gmail.com]

Sound OK to me. Thanks, Brandon!

четверг, 10 марта 2016 г., 2:58:21 UTC+3 пользователь diogo.monica написал:

TOB Members,
As requested on our last call, here is an alternative proposal for a new project around an OCI Distribution Format.

OCI Distribution Format Spec Proposal

The Open Container Initiative has made significant progress towards the definition of a lightweight, universal runtime container format, and is about one month away from a release candidate version of the spec. When the OCI was first established, its goals were to create an open specification shared between implementations that allowed the distribution and execution of containers; and optional layers defining signing and naming. The OCI was established with a clear charter and with extensively discussed scope, and it is our responsibility as a TOB to govern in a manner consistent with that charter and scope.

       Charter:         https://www.opencontainers.org/governance

Scope:         https://www.opencontainers.org/governance/oci-scope-table

The scope table in particular touches on all four aspects of Brandon’s original proposal, and very explicitly establishes both that we should take a layered approach and that certain layers should be optional or out of scope. Attempting to add too much to the base OCI specs during this innovation phase will slow down the current rate of evolution that some of the layers are undergoing.

This proposal recommends that the TOB continues with the initial plans of creating a software shipping container distribution format spec (OCI Distribution Format) that includes a simple, distributable, filesystem bundle format, compatible with most popular storage and distribution mechanisms, in particular those implementing content-addressable storage such as Docker registry and ipfs.

The recommendation is to start with a simplified version of the Docker v2.2 specification, and create a new OCI project to develop and shepherd an OCI Distribution Format Spec. The objectives of this particular initiative are:

• Define a filesystem bundle that can be serialized and hashed to ensure content integrity, ensuring interoperability between implementations.

• Create a reference spec for an optional layer that describes how to use digital signatures to verify the integrity of the content.

• Create a reference spec for an optional layer that allows DNS based naming.

This proposal is fully in-line with the charter of the OCI initiative as defined in https://www.opencontainers.org/governance, and in particular, it follows directly from the OCI Scope Table that all of the members of the OCI initiative agreed to.

Given the close relationship with the OCI Runtime Spec 1.0, some of the work around defining this OCI Distribution Format should either wait for the Runtime spec to be finalized, or at least be done in a way such that the OCI Distribution Format specification is ensured to be compatible.

Proposed Timeline and Roadmap

• March

• Define the maintainers that will responsible for delivering this spec

• April

• Beginning defining a filesystem bundle format based on a simplified version of the Docker v.2.2 specification

• May

• First complete draft of OCI Distribution Format spec

• June

• Release initial version of OCI Distribution Format spec (This aligns with OCI Runtime Container Format spec v1.0 timing)

Thank you,

--

Diogo Mónica