DEP 12: Per-package machine-readable metadata about Upstream

[Charles Plessy]

Dear all,

I am starting the Debian Enahncement Proposal (DEP) number 12 together with
Andreas Tille who kindly accepted to co-drive this work.

For general information about DEPs, please see http://dep.debian.net/deps/dep0/

The puropose of this proposal is to store in source packages some metadata
related to the usptream project, in a file called debian/upstream. We are
currently using it in some Debian Pure Blends projects to feed some summary
pages with the bibliographic references of manuscripts describing the packaged
works. See for instance Debian Med's "biology" task page.
http://debian-med.alioth.debian.org/tasks/bio

I think that debian/upstream can be more broadly useful, and therefore propose
this DEP, to finalise the format and standardise the most frequent fields of
this file.

Your comments, suggestions or questions are all very welcome; please send them
to the debian-qa mailing list, where the discussion will take place.

Here is the full information about this DEP:

Title: Per-package machine-readable metadata about Upstream
DEP: 12
State: DRAFT
Date: 2013-01-03
Drivers: Charles Plessy <ple...@debian.org>, Andreas Tille <ti...@debian.org>
URL: http://dep.debian.net/deps/dep12
Source: http://anonscm.debian.org/viewvc/dep/web/deps/dep12.mdwn
Abstract:
The source package control files and some of their derivatives are currently
used to document the URL of the home page of the work that is packaged
("upstream"). However, this approach is hard to extend to other information
describing upstream, because the size of the control files has to be limited
according to the limited power of some systems running Debian.
.
We propose a new file in the source packages, debian/upstream, formatted
in YAML to hold arbitrary information about upstream, for instance the
URL to a donation page, bibliographic information about publications
describing the upstream work, the URL to the upstream sources, etc.
.
The scope of this DEP is the format of the debian/upstream file, and the
specification of the most frequent fields. The flow of information from the
source packages to data providers such as the UDD are is specified here.
Please refer to http://wiki.debian.org/UpstreamMetadata for more information
on that subject.
.
Discussion takes place on the debian-qa mailing list.

Have a nice day,

--
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan


--
To UNSUBSCRIBE, email to debian-proj...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2013010301...@falafel.plessy.net

[Paul Wise]

The idea in general (providing more metadata about our upstreams) is a good one.

On Thu, Jan 3, 2013 at 9:11 AM, Charles Plessy wrote:

> The source package control files and some of their derivatives are currently
> used to document the URL of the home page of the work that is packaged
> ("upstream"). However, this approach is hard to extend to other information
> describing upstream, because the size of the control files has to be limited
> according to the limited power of some systems running Debian.

This DEP-12 seems to exist because of this assumption. Is it true that
everything from debian/control must end up in the Packages files?

> We propose a new file in the source packages, debian/upstream, formatted
> in YAML to hold arbitrary information about upstream, for instance the

Is there any reason you didn't choose a deb822 based format?

> The scope of this DEP is the format of the debian/upstream file, and the
> specification of the most frequent fields. The flow of information from the
> source packages to data providers such as the UDD are is specified here.
> Please refer to http://wiki.debian.org/UpstreamMetadata for more information
> on that subject.

Some further thoughts:

debian/upstream is to contain a Homepage, should we move the Homepage
from debian/control to debian/upstream and thus out of Packages?

Can we have the Debian mirrors contain an Upstreams file in addition
to Packages/Sources?

DEP-11 is also about upstream metadata, but about upstream metadata
that is extracted from binary packages. It might be interesting to
harmonise these two DEPs and the effect they will have on the archive.
For example they both introduce another mechanism to point at the
upstream homepage.

http://wiki.debian.org/AppStreamDebianProposal

--
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to debian-q...@lists.debian.org

with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CAKTje6FJON=rN3mR2chvzY+Jk4H+Ck...@mail.gmail.com

[Raphael Hertzog]

On Thu, 03 Jan 2013, Paul Wise wrote:
> > The source package control files and some of their derivatives are currently
> > used to document the URL of the home page of the work that is packaged
> > ("upstream"). However, this approach is hard to extend to other information
> > describing upstream, because the size of the control files has to be limited
> > according to the limited power of some systems running Debian.
>
> This DEP-12 seems to exist because of this assumption. Is it true that
> everything from debian/control must end up in the Packages files?

No. Currently all fields ends up somewhere (either in the .dsc and thus in
Sources, or in the .deb and thus in Packages, or in .changes) but this
doesn't have to be that way.

Where fields ends up is controlled by Dpkg/Control/Fields.pm (in
libdpkg-perl), more precisely by the "allowed" key in the huge list of
hash in that file.

It should be possibleto have only CTRL_INFO_(PKG|SRC) in that value and
not have it copied anywhere.

> debian/upstream is to contain a Homepage, should we move the Homepage
> from debian/control to debian/upstream and thus out of Packages?

No, Homepage is an important information that can be used as identifier as
well. I'd like to keep it in .deb files and thus in Packages to be
coherent.

Cheers,
--
Raphaël Hertzog ◈ Debian Developer

Get the Debian Administrator's Handbook:
→ http://debian-handbook.info/get/

--
To UNSUBSCRIBE, email to debian-q...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130103072...@x230-buxy.home.ouaza.com

[Guillem Jover]

On Thu, 2013-01-03 at 10:29:13 +0800, Paul Wise wrote:
> On Thu, Jan 3, 2013 at 9:11 AM, Charles Plessy wrote:
> > The source package control files and some of their derivatives are currently
> > used to document the URL of the home page of the work that is packaged
> > ("upstream"). However, this approach is hard to extend to other information
> > describing upstream, because the size of the control files has to be limited
> > according to the limited power of some systems running Debian.
>
> This DEP-12 seems to exist because of this assumption. Is it true that
> everything from debian/control must end up in the Packages files?

Not really, there are different stages where this gets decided. There's
the user defined fields in debian/control with their explicit export
rules, there's dpkg-gencontrol which has its own defaults, then there's
the archive generation software (dak in Debian's case). So a field could
be shipped in a binary(.deb)/source(.dsc) package but still be excluded
from a repository index (Packages/Sources), or placed in a different
index file (Upstreams) on the archive side, for example.

We could also modify for example where dpkg-gencontrol retrieves its
information, to add as a source debian/upstreams in addition to
debian/control, but first we'd need to see if this is desriable or if
it makes sense at all (but definitely not if that file is in YAML).

> > We propose a new file in the source packages, debian/upstream, formatted
> > in YAML to hold arbitrary information about upstream, for instance the
>
> Is there any reason you didn't choose a deb822 based format?

I already mentioned my dissatisfaction with that on the last iteration
of this proposal, and I still think it's a very poor choice of a format
given our current tools and “unified metadata format”. But no specific
reasons were given AFAIR.

> > The scope of this DEP is the format of the debian/upstream file, and the
> > specification of the most frequent fields. The flow of information from the
> > source packages to data providers such as the UDD are is specified here.
> > Please refer to http://wiki.debian.org/UpstreamMetadata for more information
> > on that subject.
>
> Some further thoughts:
>
> debian/upstream is to contain a Homepage, should we move the Homepage
> from debian/control to debian/upstream and thus out of Packages?

I guess we could have a discussion about the possibility/advantages of
having one index file (ex. Packages) containing exclusively machine
consumable metadata (for package managers), to favour small/embedded
systems, and another one with user consumable metadata. Because these
could be recombined by high-level package managers if desired, think
the distinction between apt and aptitude for example, or apt with an
option. The first fields that come to mind could be Homepage,
Maintainer and Description (and for the latter this has already happened
somewhat). But then this does not require a split in the source format,
but only in the archive indices, and possibly in the package managers
frontends.

Thanks,
Guillem

--
To UNSUBSCRIBE, email to debian-q...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130103155...@gaara.hadrons.org

[Enrico Zini]

On Thu, Jan 03, 2013 at 04:57:56PM +0100, Guillem Jover wrote:

> option. The first fields that come to mind could be Homepage,
> Maintainer and Description (and for the latter this has already happened
> somewhat). But then this does not require a split in the source format,
> but only in the archive indices, and possibly in the package managers
> frontends.

I'll throw in a few more data points.

Other such existing fields that come to mind are Tags, and Vcs-*

Possible new fields that haven't been introduced so far are: Popcon,
Rating, upstream BTS type and URL, extra keywords for full text search
indexing, date+time the package has first entered the archive, and/or
the release; datetime that version of the package has been
built/uploaded.


Ciao,

Enrico

--
GPG key: 4096R/E7AD5568 2009-05-08 Enrico Zini <enr...@enricozini.org>

[Russ Allbery]

Enrico Zini <enr...@enricozini.org> writes:
> On Thu, Jan 03, 2013 at 04:57:56PM +0100, Guillem Jover wrote:

>> option. The first fields that come to mind could be Homepage,
>> Maintainer and Description (and for the latter this has already happened
>> somewhat). But then this does not require a split in the source format,
>> but only in the archive indices, and possibly in the package managers
>> frontends.

> I'll throw in a few more data points.

> Other such existing fields that come to mind are Tags, and Vcs-*

> Possible new fields that haven't been introduced so far are: Popcon,
> Rating, upstream BTS type and URL, extra keywords for full text search
> indexing, date+time the package has first entered the archive, and/or
> the release; datetime that version of the package has been
> built/uploaded.

Upstream VCS is another frequently-requested field that we've held off
from adding largely due to the metadata bloat.

Of course, the other issue that this DEP raises is how much sense it makes
to put all this stuff in the source package, either in debian/control or
in a new file, given that most of these fields (even including Maintainer,
although moving that probably isn't a good idea) can change independent of
any functional change in the source package.

BTW, if we're going to create a new file to hold (in part) upstream
information, it would be really nice if it could replace debian/watch, and
in the long run it would be nice if it would take over some of the
metadata requirements of debian/copyright (all the stuff that isn't actual
legal notices, basically). We currently have that information spread
across multiple files; if we're going to go to the work of introducing a
new file, I think we should get consolidation for our trouble.

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

--
To UNSUBSCRIBE, email to debian-q...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87k3ruy...@windlord.stanford.edu

[Paul Wise]

On Thu, Jan 3, 2013 at 11:57 PM, Guillem Jover wrote:

> I guess we could have a discussion about the possibility/advantages of
> having one index file (ex. Packages) containing exclusively machine
> consumable metadata (for package managers), to favour small/embedded
> systems, and another one with user consumable metadata. Because these
> could be recombined by high-level package managers if desired, think
> the distinction between apt and aptitude for example, or apt with an
> option. The first fields that come to mind could be Homepage,
> Maintainer and Description (and for the latter this has already happened
> somewhat). But then this does not require a split in the source format,
> but only in the archive indices, and possibly in the package managers
> frontends.

Splitting up Packages and Sources into stuff needed by different
audiences (dpkg/apt, PackageKit, buildds, users, developers etc) is
something that I have wanted to see for years. Having additional
metadata from popcon, debian/upstream and other sources would be
excellent too.

I would encourage the authors of DEP-11 and DEP-12 to join forces and
come up with a comprensive proposal covering Packages/Source audience
splitup and metadata extraction from various sources.

--
bye,
pabs

http://wiki.debian.org/PaulWise

--
To UNSUBSCRIBE, email to debian-q...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CAKTje6FN7ER11e7Q8hSPzyVWn6_H6q2dEM2rp=SDhMh5...@mail.gmail.com

[Paul Wise]

On Fri, Jan 4, 2013 at 12:59 AM, Russ Allbery wrote:

> Of course, the other issue that this DEP raises is how much sense it makes
> to put all this stuff in the source package, either in debian/control or
> in a new file, given that most of these fields (even including Maintainer,
> although moving that probably isn't a good idea) can change independent of
> any functional change in the source package.

That sounds like you maybe want DEP-2:

http://dep.debian.net/deps/dep2/

--
bye,
pabs

http://wiki.debian.org/PaulWise

--
To UNSUBSCRIBE, email to debian-q...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CAKTje6F9b4babC3HDobRKYW=-mooGL1AznRMd...@mail.gmail.com

[Russ Allbery]

Paul Wise <pa...@debian.org> writes:
> On Fri, Jan 4, 2013 at 12:59 AM, Russ Allbery wrote:

>> Of course, the other issue that this DEP raises is how much sense it
>> makes to put all this stuff in the source package, either in
>> debian/control or in a new file, given that most of these fields (even
>> including Maintainer, although moving that probably isn't a good idea)
>> can change independent of any functional change in the source package.

> That sounds like you maybe want DEP-2:

> http://dep.debian.net/deps/dep2/

Not really. DEP-2 only addresses one part of my point, and proposes a
very specific solution to that problem.

I'd be happy to see DEP-2 happen, certainly. But I wouldn't want to block
this discussion on the implementation of a comprehensive infrastructure
that does all the things discussed in DEP-2.

My point, rather, is that a bunch of the stuff that's being discussed as
relevant to debian/upstream can change independent of any functional
change in the package, and therefore the proposal raises the question of
whether we want to put even more non-functional metadata directly into the
source package instead of somewhere with lighter-weight update processes.
I think this is particularly relevant to any information that isn't
specific to a particular upstream version of the package.

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

--
To UNSUBSCRIBE, email to debian-q...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87a9sps...@windlord.stanford.edu

[Enrico Zini]

On Thu, Jan 03, 2013 at 04:51:02PM -0800, Russ Allbery wrote:

> My point, rather, is that a bunch of the stuff that's being discussed as
> relevant to debian/upstream can change independent of any functional
> change in the package, and therefore the proposal raises the question of
> whether we want to put even more non-functional metadata directly into the
> source package instead of somewhere with lighter-weight update processes.
> I think this is particularly relevant to any information that isn't
> specific to a particular upstream version of the package.

This currently happens with Debtags, since tags are changed at
debtags.debian.net and not in package source files: "Tag" fields are
injected by dak directly into Packages through an override file.

It's a bit of a hack and we already discussed some ways of improving it;
afaik it's the only existing example of a field that can change
independently of functional changes in the packages.

[Charles Plessy]

Many thanks everybody for your comments.

I have answered in three separate messages on the debian-qa mailign list.

* [DEP 12] Why chosing YAML.
https://lists.debian.org/20130105071...@falafel.plessy.net

* [DEP 12]: Why debian/upstream and not debian/control.
https://lists.debian.org/20130105074...@falafel.plessy.net

* [DEP 12] Relation with DEP 11.
https://lists.debian.org/20130105081...@falafel.plessy.net

Have a nice week-end,

--
Charles Plessy
Tsurumi, Kanagawa, Japan


--
To UNSUBSCRIBE, email to debian-proj...@lists.debian.org

with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130105082...@falafel.plessy.net