[gentoo-dev] Next council meeting on 7 Dec 2009 at 1900UTC
Denis Dupeyron
us to discuss things please let us know in reply to this email. What
is already known is we'll talk about mtime preservation and prefix.
You can find threads about those at:
http://archives.gentoo.org/gentoo-dev/msg_a9e26414f2278275bdfa08baf839704f.xml
http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml
Denis.
Brian Harring
I'd like
http://archives.gentoo.org/gentoo-dev/msg_6b3e00049a1bf35fbf7a5e66d1449553.xml
to be discussed, specifically zacs form of forced mtime updating of
/var/db/pkg on vdb modifications
Reiterating the reasoning, it'll be used for enabling the managers to
manage caches, specifically invalidating said caches if an alternative
manager modifies the vdb.
End result, if we can build better caches for vdb, vdb access can be
heavily sped up.
At this point, pkgcore/portage both have it implemented. Not sure if
portage has released it yet, but >=pkgcore-0.5.2 is in the tree w/
said updating support.
~harring
Zac Medico
> At this point, pkgcore/portage both have it implemented. Not sure if
> portage has released it yet, but >=pkgcore-0.5.2 is in the tree w/
> said updating support.
It's in >=portage-2.1.7.2 (and 2.1.7.x should be going stable in a
month or so).
--
Thanks,
Zac
Ciaran McCreesh
Brian Harring <ferr...@gmail.com> wrote:
> I'd like
> http://archives.gentoo.org/gentoo-dev/msg_6b3e00049a1bf35fbf7a5e66d1449553.xml
> to be discussed, specifically zacs form of forced mtime updating of
> /var/db/pkg on vdb modifications
I've still not had an answer to:
"Provide proof that all existing and future caches that would rely upon
this validation mechanism are functions purely and exclusively
dependent upon the VDB content, and I shall be happy to make the
change."
Without that proof, all introducing that change would do is provide
another unreliable mechanism, so it's of no use at all.
--
Ciaran McCreesh
Brian Harring
> On Wed, 25 Nov 2009 17:34:38 -0800
> Brian Harring <ferr...@gmail.com> wrote:
> > I'd like
> > http://archives.gentoo.org/gentoo-dev/msg_6b3e00049a1bf35fbf7a5e66d1449553.xml
> > to be discussed, specifically zacs form of forced mtime updating of
> > /var/db/pkg on vdb modifications
>
> I've still not had an answer to:
>
> "Provide proof that all existing and future caches that would rely upon
> this validation mechanism are functions purely and exclusively
> dependent upon the VDB content, and I shall be happy to make the
> change."
First I've seen this question actually or at least this particular
interesting phrasing. That said, "no" comes to mind, since the
requirement you set is daft.
The timestamp updating is for whenever the vdb content (addition of a
pkg, pkgmoves being applied, removal of a pkg, modification of
metadata, etc) is changed. That's all that timestamp is for. Vdb
content.
In light of what the timestamp is, your demand for proof is pretty off
the mark. If you still consider it to be a valid question, please
rephrase it and clarify why exactly proof must be provided that people
reading that timestamp (which is for vdb content only) will only be
using that timestamp for vdb content.
Your request is akin to demanding proof that a hammer only be used as
a hammer. It's a fricking hammer- it has one use, one way of being
used. If someone goes out of their way to be an idiot, they're an
idiot, not the specs problem.
Seriously, if you're actually worrying about some specific usage case,
state it- on the face of it, your request for proof right now makes
zero sense. Kindly provide a scenario or elucidation.
~harring
Ciaran McCreesh
Brian Harring <ferr...@gmail.com> wrote:
> > "Provide proof that all existing and future caches that would rely
> > upon this validation mechanism are functions purely and exclusively
> > dependent upon the VDB content, and I shall be happy to make the
> > change."
>
> First I've seen this question actually or at least this particular
> interesting phrasing. That said, "no" comes to mind, since the
> requirement you set is daft.
It's clipboarded from the bug.
> The timestamp updating is for whenever the vdb content (addition of a
> pkg, pkgmoves being applied, removal of a pkg, modification of
> metadata, etc) is changed. That's all that timestamp is for. Vdb
> content.
>
> In light of what the timestamp is, your demand for proof is pretty
> off the mark. If you still consider it to be a valid question,
> please rephrase it and clarify why exactly proof must be provided
> that people reading that timestamp (which is for vdb content only)
> will only be using that timestamp for vdb content.
>
> Your request is akin to demanding proof that a hammer only be used as
> a hammer. It's a fricking hammer- it has one use, one way of being
> used. If someone goes out of their way to be an idiot, they're an
> idiot, not the specs problem.
>
> Seriously, if you're actually worrying about some specific usage
> case, state it- on the face of it, your request for proof right now
> makes zero sense. Kindly provide a scenario or elucidation.
You're asking for a cache validation mechanism that's based not upon
what it logically invalidates, but upon something you assume to be
equivalent. Specifically, you assume that "VDB has changed" and "the
installed packages have changed" are exactly the same thing, and you're
wanting to build a cache based upon that highly questionable
assumption. There are at least two logically different sets of
'changes' that might apply to VDB (metadata changes, and package
version changes), and you're attempting to confuse the two, along with
any others that people come up with later on.
What's wrong with, instead, having cache files for "something has
changed", "the set of installed packages has potentially changed", "the
metadata for installed packages has potentially changed" and "the set of
installable packages has potentially changed"? That way you can write
your cache checks to depend explicitly upon the thing upon which they
depend (along with a global catch-all to make future new validation
mechanisms easier), not upon something you assume is equivalent but
probably isn't.
--
Ciaran McCreesh
Brian Harring
> On Thu, 26 Nov 2009 08:33:03 -0800
> Brian Harring <ferr...@gmail.com> wrote:
> > > "Provide proof that all existing and future caches that would rely
> > > upon this validation mechanism are functions purely and exclusively
> > > dependent upon the VDB content, and I shall be happy to make the
> > > change."
Ah... there we go, you're again asking for specific timestamps such
that specific caches can be invalidated. Same as I said in the bug,
you want it, propose it. As you stated above, *still* a global
timestamp of some sort is needed.
Seriously- if you want some specific cache timestamp, go nuts. The
global timestamp is still needed and that's the only one I give a damn
about at this juncture- I'm not as much interested in layering in new
hacky caches on the vdb to try and make it performant as I'm
interested in building flat out, a new vdb that is designed from the
ground up for efficiency/performance.
When the old vdb format has the timestamp requirements, I can use that
to keep the two in sync (maintaining compatibility while being free
to start developing a better vdb, or alternate implementations- say
remote). Beyond that, for people less ambitious it serves as a
timestamp they can use for updating their own caches- not the most
fine grained admittedly, but it's also a rare scenario.
Either way, you want specific cache timestamps, it's an addition to
this proposal- for example I'm specifically disinterested in adding a
pkg names cache because the gain from it isn't particularly high for
the scenarios I'm looking at (provides/use/iuse/depends/rdepends per
node being higher cost in my profilings). Admittedly it speeds up
simple lookups in the vdb of "what version do I have installed?", but
most folk do a pmerge/emerge -Dup for that (meaning full metadata
access still).
~harring
Antoni Grzymala
Hi!
How about getting back to GLEP-57 [1]? Robin Hugh Johnson made an effort
a year ago to summarize the then-current state of things regarding tree
and package signing, however the matter seems to have lain idle and
untouched for more than a year since.
I reckon that missing GPG infrastructure is one of the greatest problems
of the Gentoo distribution esp. regarding serious corporate and academic
deployments.
I can devote some time to helping with the matter.
Regards,
Antoni Grzymała
--
[a]
Antoni Grzymala
> Denis Dupeyron dixit (2009-11-25, 14:50):
>
> > The next council meeting will be on 7 Dec 2009 at 1900UTC. If you want
> > us to discuss things please let us know in reply to this email. What
> > is already known is we'll talk about mtime preservation and prefix.
> > You can find threads about those at:
> > http://archives.gentoo.org/gentoo-dev/msg_a9e26414f2278275bdfa08baf839704f.xml
> > http://archives.gentoo.org/gentoo-dev/msg_2a62689c71f95e4de5699a330b8b5524.xml
>
> Hi!
>
> How about getting back to GLEP-57 [1]? Robin Hugh Johnson made an effort
[...]
Forgot the URL, of course:
[1] http://www.gentoo.org/proj/en/glep/glep-0057.html
--
[a]
Thomas Sachau
Hm, i requested the discussion of real multilib support for portage 2 months ago, i requested it for
the following council meeting, i requested it for the last council meeting and i requested it for
the next council meeting. I hope, it will finally get a place on 7 Dec 2009.
I have a git branch with a modified 2.2_rc* version of portage with included multilib support. I
already wrote about basic implementation 2 months ago in the conversation on this list with vapier.
Since zmedico wants a council-ok before accepting any patches for multilib-support in portage, i
request this. My main idea behind this request is, that more people will have a look and there are
potentially more people involved in improving it. In addition it allows more people to easily get
required 32bit libs the way they want them (specific version, specified USE flags, self-compiled, so
up-to-date unlike the emul-linux-x86-* packages).
Since the code may change in the future, my idea is to restrict it currently only to 2.2_rc*
versions and in addition a required feature (e.g. FEATURES="multilib"). Once everyone is ok with the
code and the way it works, it could be proposed as an PMS-update. If other PM-mantainers are
interested in improving it before, they are of course free to also help improving the code and the
way it works.
--
Thomas Sachau
Gentoo Linux Developer
Richard Freeman
> How about getting back to GLEP-57 [1]? Robin Hugh Johnson made an effort
> a year ago to summarize the then-current state of things regarding tree
> and package signing, however the matter seems to have lain idle and
> untouched for more than a year since.
>
One concern I have with the GLEP-57 is that it is a bit hazy on some of
the implementation details, and the current implementation has some
weaknesses.
I go ahead and sign my commits. However, when I do this I'm signing the
WHOLE manifest. So, if I stabilize foo-1.23-r5 on my arch, at best I've
tested that one particular version of that package works fine for me.
My signature applies to ALL versions of the package even though I
haven't tested those.
Now, if we had an unbroken chain of custody then that wouldn't be a
problem. However, repoman commit doesn't enforce this and the manifest
file doesn't really contain any indication of what packages are assured
to what level of confidence.
If we want to sign manifests then the only way I see it actually
providing real security benefits is if either:
1. The distro does this in the background in some way in a secure
manner (ensuring it happens 100% of the time).
2. Every developer signs everything 100% of the time (make it a QA
check).
The instant you have a break in the signature chain you can potentially
have a modification. If somebody cares enough to check signatures, then
they're going to care that the signature means something. Otherwise it
only protects against accidental modifications, and the hashes already
provide pretty good protection against this.
Dawid Węgliński
> Antoni Grzymala wrote:
> > How about getting back to GLEP-57 [1]? Robin Hugh Johnson made an effort
> > a year ago to summarize the then-current state of things regarding tree
> > and package signing, however the matter seems to have lain idle and
> > untouched for more than a year since.
>
> One concern I have with the GLEP-57 is that it is a bit hazy on some of
> the implementation details, and the current implementation has some
> weaknesses.
>
> I go ahead and sign my commits. However, when I do this I'm signing the
> WHOLE manifest. So, if I stabilize foo-1.23-r5 on my arch, at best I've
> tested that one particular version of that package works fine for me.
> My signature applies to ALL versions of the package even though I
> haven't tested those.
>
I may be wrong - then please correct me. You don't sign every package versions
but Manifest. Thus you somehow prove every file checksum is correct. If there
were any changes made on server side, those checksums would be incorrect
according to your signed Manifest. Currently any change may be fixed by whoever
it is by the same command ebuild foo digest.
> Now, if we had an unbroken chain of custody then that wouldn't be a
> problem. However, repoman commit doesn't enforce this and the manifest
> file doesn't really contain any indication of what packages are assured
> to what level of confidence.
That's what should be discussed - forcing developers to sign their commits and
implementing this support in package managers.
>
> If we want to sign manifests then the only way I see it actually
> providing real security benefits is if either:
>
> 1. The distro does this in the background in some way in a secure
> manner (ensuring it happens 100% of the time).
>
> 2. Every developer signs everything 100% of the time (make it a QA
> check).
>
> The instant you have a break in the signature chain you can potentially
> have a modification. If somebody cares enough to check signatures, then
> they're going to care that the signature means something. Otherwise it
> only protects against accidental modifications, and the hashes already
> provide pretty good protection against this.
>
That's not really true. I see tips like "if you have digest incorrect, run
ebuild foo.ebuild digest" very often. Really small group of people care about
broken digests. :(
--
Cheers
Dawid Węgliński
Robin H. Johnson
> I reckon that missing GPG infrastructure is one of the greatest problems
> of the Gentoo distribution esp. regarding serious corporate and academic
> deployments.
>
> I can devote some time to helping with the matter.
There are still two GLEPs in the series that have not yet made it to
draft status:
http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/02-developer-process-security
http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/03-gnupg-policies-and-handling
However the main content of GLEPS 58-61 IS ready for the council to
approve, and are NOT blocking on the above two items.
As such, I would like to present GLEPS 58,59,60,61 for final review, and
for the council to vote on their approval during the January meeting.
I'm going to summarize them here:
GLEP58: Security of distribution ... MetaManifest
-------------------------------------------------
- covers all Manifests with a infra-generated parent Manifest.
- required for end-to-end validation.
- prevents certain package manager attacks.
- NO day-to-day developer actions required.
GLEP59: Manifest2 hash policies and security implications
---------------------------------------------------------
- Add SHA512 to all Manifest files.
- Schedule removal of SHA1, MD5, RMD160 for 6-18 months after SHA512
addition.
- Be prepared to add the NIST hash contest candidates/winner.
GLEP60: Manifest2 filetypes
---------------------------
(Has one TODO that needs clarification).
- Breaks down the Manifest2 filetypes into INFOrmational and CRITical.
- If the package manager is being strict, then INFO filetypes are
treated as CRIT filetypes.
- INFO filetypes merely cause a warning on absence.
- CRIT filetypes may trigger a delayed OR immediate failure of absence.
GLEP61: Manifest2 compression
-----------------------------
- Disk space optimization for MetaManifest from GLEP58.
There is a prototype of the MetaManifest code here:
http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/prototype/
It worked on Portage 2 years ago, but I haven't run it since then.
--
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : rob...@gentoo.org
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
Robin H. Johnson
> Antoni Grzymala wrote:
> >How about getting back to GLEP-57 [1]? Robin Hugh Johnson made an effort
> >a year ago to summarize the then-current state of things regarding tree
> >and package signing, however the matter seems to have lain idle and
> >untouched for more than a year since.
> One concern I have with the GLEP-57 is that it is a bit hazy on some
> of the implementation details, and the current implementation has
> some weaknesses.
The GLEP on Individual developer signing has not made it into a Draft
yet.
But you can view the very brief version here:
http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/02-developer-process-security?view=markup
> I go ahead and sign my commits. However, when I do this I'm signing
> the WHOLE manifest. So, if I stabilize foo-1.23-r5 on my arch, at
> best I've tested that one particular version of that package works
> fine for me. My signature applies to ALL versions of the package
> even though I haven't tested those.
This was covered in the draft linked above.
A larger discussion on it is welcome, as while both competing options
exist, neither has a clear advantage over the other.
> Now, if we had an unbroken chain of custody then that wouldn't be a
> problem. However, repoman commit doesn't enforce this and the
> manifest file doesn't really contain any indication of what packages
> are assured to what level of confidence.
Chain of custody from infrastructure to user is covered in GLEP58
(MetaManifest).
> If we want to sign manifests then the only way I see it actually
> providing real security benefits is if either:
>
> 1. The distro does this in the background in some way in a secure
> manner (ensuring it happens 100% of the time).
See GLEP58.
> 2. Every developer signs everything 100% of the time (make it a QA
> check).
+1 on this.
> The instant you have a break in the signature chain you can
> potentially have a modification. If somebody cares enough to check
> signatures, then they're going to care that the signature means
> something. Otherwise it only protects against accidental
> modifications, and the hashes already provide pretty good protection
> against this.
GLEP60 covers the Manifest2 filetypes and better logic on which
missing/mismatches should be considered as fatal.
Torsten Veller
> The GLEP on Individual developer signing has not made it into a Draft
> yet.
>
> But you can view the very brief version here:
> http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/02-developer-process-security?view=markup
[...]
> > 2. Every developer signs everything 100% of the time (make it a QA
> > check).
> +1 on this.
In the GLEPs i missed the point where the signatures of Manifests are verified.
Only the MetaManifest gets verified.
So what's the advantage of individually signed Manifests?
The only thing we can check: Is the key used for signing listed in ldap
(and thus in "the keyring of automated Gentoo keys")? Are the keys in ldap
really mine?
Do I miss anything?
BTW: About a third of the Manifests are signed [1]. We didn't improve
since 2005/2006 [2]. The two parties are working hard against each other [3].
55 Manifests are signed by revoked keys [4].
[1] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/Manifest.png
[2] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/ratio_2005.png
[3] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/Manifest2.png
[4] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/signatures_by_revoked_keys.txt
Thilo Bangert
if we really want to get there, maybe repoman should give a _small_
warning, starting now.
i dont sign my commits and have seen how my commits removed signatures of
others. i am not proud of it - but given that these are apparently never
checked in any way, then no harm is done... or what?
Thilo
Robin H. Johnson
> * "Robin H. Johnson" <rob...@gentoo.org>:
> > The GLEP on Individual developer signing has not made it into a Draft
> > yet.
> >
> > But you can view the very brief version here:
> > http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/02-developer-process-security?view=markup
>
> [...]
>
> > > 2. Every developer signs everything 100% of the time (make it a QA
> > > check).
> > +1 on this.
>
> In the GLEPs i missed the point where the signatures of Manifests are verified.
> Only the MetaManifest gets verified.
under "Procedure for verifying an item in the MetaManifest"
4.2: "M2-verifying the contents of the Manifest."
Where "M2-verify" is the verb describing the verification of a Manifest.
It _may_ include signature validation.
> So what's the advantage of individually signed Manifests?
Basically making sure that your SSH keys weren't stolen.
They explicitly protect the commit from the developer to infrastructure.
MetaManifest protects the integrity of the contents from infrastructure
out to the user. It does NOT validate the functionality of the tree or
any prior injection.
> The only thing we can check: Is the key used for signing listed in ldap
> (and thus in "the keyring of automated Gentoo keys")? Are the keys in ldap
> really mine?
> Do I miss anything?
Later on I'd like to REJECT unsigned commits.
> BTW: About a third of the Manifests are signed [1]. We didn't improve
> since 2005/2006 [2]. The two parties are working hard against each other [3].
> 55 Manifests are signed by revoked keys [4].
> [1] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/Manifest.png
> [2] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/ratio_2005.png
> [3] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/Manifest2.png
> [4] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/signatures_by_revoked_keys.txt
Nice graphs. Can you show them over a larger timespan?
Torsten Veller
> > [2] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/ratio_2005.png
> > [3] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/Manifest2.png
Yes, I recreated them from cvs and the keys available now.
[1] and [3] show the progress for the last year and [4] the history since May
2004.
- In Jan 2008 the transition to Manifest2 was finished and all
signatures were dropped.
- I guess [2] didn't "require-cross-certification" while gnupg now
defaults to "require-cross-certification".
So the number of valid signatures in [4] is lower than in [2].
After the "Manifest2"-break the level is lower.
[4] http://dev.gentoo.org/~tove/stats/gentoo-x86/Manifest/Manifest-all.png