NEW processing during freezes (Was: R 3.0.0 and required rebuilds of all reverse Depends: of R)

[Luca Falavigna]

[ This is just my personal point of view, not necessarily the one of the FTP Team ]

2013/4/1 Rene Engelhard <re...@debian.org>

>> Even if you think there are a few days between the time taken to process

>> NEW for experimental vs NEW for unstable, I've seen no evidence of that

>> and it's not as if a few days are really going to matter. (If it's that

>> critical, find a webhost running Debian and install reprepro.)

>

> A few days? There's stuff there *for months*?

True. But most of the packages that currently are on top of the NEW queue would have introduced transitions if FTP Team blindly had accepted them, and we agreed with Release Team to keep them in the queue to avoid potential breakages, given that at the time we just entered the Wheezy freeze. We sent emails to maintainers to inform them about the reasons behind the delay, and we offered to accept packages targeted to experimental instead. I think this is a good approach. Some other packages are stuck in the queue pending an answer from their maintainers about some concerns FTP Team raised. There's little FTP Team can do other than wait for actions from maintainers.

Since July 1st (first day of Wheezy freeze), we have the following figures:

* 2085 NEW packages received (7.694 per day)

* 1379 were accepted (5.089 per day)

* 213 were rejected (0.786 per day)

* 130 generated comments from FTP Team (0.480 per day)

During the freeze, the number of NEW packages received dropped by a half if compared to the average during active development (about 14,85 packages each day), so the number of actions by the FTP Team (about 13.5 accepts, 1.2 rejects, 0.5 comments each day). The above figures are normal during freezes, both maintainers and FTP Team members are focused on other tasks (releasing Wheezy is, of course, one of them!). Maintainers upload packages more often than FTP Team is willing to process them (about 1,8 packages every day), that explains the recent NEW queue growth.

Just for the record, FTP Team managed to keep the NEW queue around ten packages for more than one year and a half, average processing time was less than two days. Also, FTP Team processed almost two hundred packages just before the freeze to give maintainers a chance to make their packages into Wheezy (and most of them did!).

On the other hand, FTP Team is willing to fast-track NEW packages anytime, if needed. Asking for a pacakge acceptance in #debian-ftp is always worth it, and rarely these requests are not taken into consideration (as it happened for some gcc/clang packages, or GNOME ones). If you need actions from FTP Team, feel free to talk to us and we will be happy to help you.

Cheers,

Luca

[Rene Engelhard]

Hi,

On Mon, Apr 01, 2013 at 05:06:03PM +0200, Luca Falavigna wrote:
> True. But most of the packages that currently are on top of the NEW queue
> would have introduced transitions if FTP Team blindly had accepted them,
> and we agreed with Release Team to keep them in the queue to avoid
> potential breakages, given that at the time we just entered the Wheezy
> freeze. We sent emails to maintainers to inform them about the reasons

True for unstable, not for experimental, Because stuff uploading to experimental
can cause a transition if uploaded to unstable, yes - for *jessie*.
Of course if mallicous or careless maintainers uploaded to unstable.. *shrugs*

> behind the delay, and we offered to accept packages targeted to
> experimental instead. I think this is a good approach. Some other packages
> are stuck in the queue pending an answer from their maintainers about some
> concerns FTP Team raised. There's little FTP Team can do other than wait
> for actions from maintainers.

I understand that, *if* comments were needed. But that's not the case always.

Some were stuck there and gor rejedted after 2 months. That specific
exmple was needed to make a transition which *will* happen in jessie
_less painless_.

And this isn't a explanation for *completely new*, (and thus no r-deps,
thus no transition) packages.

(And no, I didn't get a comment.)

> On the other hand, FTP Team is willing to fast-track NEW packages anytime,
> if needed. Asking for a pacakge acceptance in #debian-ftp is always worth
> it, and rarely these requests are not taken into consideration (as it
> happened for some gcc/clang packages, or GNOME ones). If you need actions
> from FTP Team, feel free to talk to us and we will be happy to help you.
> Cheers,

I know, and afaicr I went this way (or /query ansgar), too and I am very
grateful for that.

It's more cumbersome than it needs to be, though.

Regards,

Rene


--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20130401153...@rene-engelhard.de

[Michael Biebl]

Am 01.04.2013 17:06, schrieb Luca Falavigna:
> [ This is just my personal point of view, not necessarily the one of the
> FTP Team ]

Luca, I think you and the FTP team are doing an awesome job! Even during
freezes. If you don't believe me, just have a look at the amount of
packages that were quickly accepted for preparing GNOME 3.8 in experimental.

Michael

/ who is very happy how NEW works nowadays.

--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[Luca Falavigna]

2013/4/1 Rene Engelhard <re...@debian.org>:

> True for unstable, not for experimental, Because stuff uploading to experimental
> can cause a transition if uploaded to unstable, yes - for *jessie*.

Most of the packages introducing new transitions were accepted, if
targeted experimental. A lot are still in the queue, though. The
"rationale" will follow shortly.

> Of course if mallicous or careless maintainers uploaded to unstable.. *shrugs*

True, and currently it's not possible to block those. But TTBOMK, only
a very few cases happened, and fortunately it was not to crack the
game.

> I understand that, *if* comments were needed. But that's not the case always.
>
> Some were stuck there and gor rejedted after 2 months. That specific
> exmple was needed to make a transition which *will* happen in jessie
> _less painless_.

Comments and rejects are issued during package review. Most of the
packages are accepted without remarks, some of them have a longer
path. I know the frustration of waiting for two months or more, and
then having your package rejected for just a few comments, but that is
why the NEW queue is in place, and maintainers sometimes have to face
a "try again".

> And this isn't a explanation for *completely new*, (and thus no r-deps,
> thus no transition) packages.
>
> (And no, I didn't get a comment.)

As I outlined in my previous message, we're in a phase in which we
receive more packages than we're able to process, due to internal
factors (our team is made of five members, while active developers are
about five hundred), or external (some of us have been busy elsewhere,
rationales are better explained in other mailinglists). binary-NEW
packages are usually processed first due to dak sorting, and the NEW
queue is not generally processed as FIFO, so it can happen a package
is processed way later even if it has been uploaded earlier.

>> On the other hand, FTP Team is willing to fast-track NEW packages anytime,
>

> I know, and afaicr I went this way (or /query ansgar), too and I am very
> grateful for that.
>
> It's more cumbersome than it needs to be, though.

In a perfect world there wouldn't be any need for a NEW queue at all.
But we have to face with the reality.
We try to do our best to improve things where we can. From the FTP
Team side, we always try to be quick and helpful with our fellow
developers, and are happy to hear about suggestions how to improve
further. On the other hand, please bear with us a little more when
packages are not processed so quickly. FTP Team is not just pressing a
button.

Cheers,
Luca

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CADk7b0PsPid5gDpbKikUHa4S...@mail.gmail.com

[Wookey]

+++ Luca Falavigna [2013-04-01 17:06 +0200]:

> Since July 1st (first day of Wheezy freeze), we have the following
> figures:
> * 2085 NEW packages received (7.694 per day)
> * 1379 were accepted (5.089 per day)
> * 213 were rejected (0.786 per day)
> * 130 generated comments from FTP Team (0.480 per day)

Wow you guys are busy. I had no idea we generated new packages at such
a rate, even in freeze time.

> During the freeze, the number of NEW packages received dropped by a half
> if compared to the average during active development (about 14,85 packages
> each day), so the number of actions by the FTP Team (about�13.5 accepts,
> 1.2 rejects, 0.5 comments each day).

> Just for the record, FTP Team managed to keep the NEW queue around ten
> packages for more than one year and a half, average processing time was
> less than two days.

Yes, and that excellent record was certainly appreciated by me. You
(the FTP team as a whole) probably don't get told often enough so:
'Thanks for doing a somewhat dull job very well'.

Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013040116...@stoneboat.aleph1.co.uk

[Joachim Breitner]

Hi,

Am Montag, den 01.04.2013, 17:06 +0200 schrieb Luca Falavigna:
> Just for the record, FTP Team managed to keep the NEW queue around ten
> packages for more than one year and a half, average processing time
> was less than two days.

I did a few uploads during that time and really enjoyed the speed; a lot
of kudos for that.

The (luxury) problem is that I got used to it and began uploading the
new (and NEW) dependency bar of package foo along with the new version
of foo (instead of uploading bar first, wait for NEW processing and only
then bump foo’s version), because it was more convenient for me and the
packaging team and worked fine.

Then suddenly NEW processing slowed down (for understandable reasons
that Luca outlined) considerably without me realizing it, or realizing
that it is more than just a temporary ditch, so now everytime I do that,
foo is uninstallable for a prolonged period of time.

So it is not so much the processing speed of NEW that sometimes
irritates me, but rather its unpredictability.

(I do not offer solutions nor blaming anyone here, just sharing my
experiences.)

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
nom...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nom...@joachim-breitner.de | http://people.debian.org/~nomeata

[Thomas Goirand]

On 04/01/2013 11:06 PM, Luca Falavigna wrote:
> On the other hand, FTP Team is willing to fast-track NEW packages
> anytime, if needed.

That's simply not truth. I can't let you say that and not reply.
And I'm happy we come to this topic.

I've sent a mail to the FTP masters last January (IIRC) about it,
because I'm working on Openstack, which has release cycles of 6
months. I've been working on these packages days and nights,
hoping that I would make it for the next release of Openstack.

As it stand, I still have some python packages in the NEW queue,
from the *last* version of Openstack, released nearly 6 months ago.
I did the upload nearly 3 months ago for Cinder. I'm still waiting.
And I'm not even talking about the python dependencies that
I need for the next release of Openstack, due in 3 days. Absolutely
all of them were uploaded for Experimental. There is absolutely
no transition problem with my packages.

The result is a real disaster for my development. There was talks
about doing the packaging of Openstack together, with Ubuntu
and Debian. But as it stands, I don't think I'm in a good position
to ask for that, since it can take a random time for my packages
to get reviewed and finally accepted. Now, if anyone tells me
that Debian isn't adapted for such a fast development as
Openstack, I will be forced to agree. My only option (which I am
already doing) is using a private non-official repository, which
is really not a satisfying solution.

I even proposed my help for the review process (of other packages,
not mine, of course...). This was a no-go refusal. I haven't seen
either that the FTP team asked for help and new members, if
I am seen as not qualified and the team is understaffed!

I by the way agree with Joachim that the unpredictability of the
processing time is the most irritating part of it all.

I really don't understand why all development has to be
completely stalled during the freeze. The only answer I had
from the FTP-masters was "we are frozen, have you noticed?"...

...

Thomas Goirand (zigo)

BTW: http://ftp-master.debian.org/stat.html shows that
everything started to get stuck around mid-december.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/515B40D0...@debian.org

[Thomas Goirand]

On 04/02/2013 12:16 AM, Luca Falavigna wrote:
> In a perfect world there wouldn't be any need for a NEW queue at all.
> But we have to face with the reality.
> We try to do our best to improve things where we can. From the FTP
> Team side, we always try to be quick and helpful with our fellow
> developers, and are happy to hear about suggestions how to improve
> further.

I got a bunch of suggestions...

Suggestion #1: if a package stays more than a month in the NEW
queue, then it gets automatically approved, and may be
reviewed later on. My reasoning is that more than a month,
it can become really problematic and blocks development.

Suggestion #2: get rid of the new binary queue (not new source
package, that's different). There's no reason why the licensing
of a package changes just because the maintainer decides to add
a new binary, so it makes no sense. This would save a lot of
time for the FTP team.

Suggestion #3: have a system where any other DD can review
a package in the NEW queue, not only the FTP masters or the
FTP assistants.

Suggestion #4: recognized that the FTP team needs to work faster,
and get more people in the FTP team.

Suggestion #5: make it so that a bunch of packages can be
reviewed together, as they might depend on each other, and we
would like to avoid cases where some packages are accepted, but
can't be installed because their dependencies are in NEW.

Suggestion #6: get rid of the NEW queue completely. I'm not
the only one that thinks it should be like that, and that the
licensing review process could happen after packages are
accepted. Maybe though, I'll be the only one saying it out
loud (but I'm getting used to it...).

Thomas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/515B4418...@debian.org

[Charles Plessy]

Le Wed, Apr 03, 2013 at 04:34:24AM +0800, Thomas Goirand a écrit :
>
> I even proposed my help for the review process (of other packages,
> not mine, of course...). This was a no-go refusal. I haven't seen
> either that the FTP team asked for help and new members, if
> I am seen as not qualified and the team is understaffed!

Hi Thomas,

it is still worthwile doing reviews even if the FTP team does not consider
them. When doing so, I have sometimes found problems, which caused the
packager to retract or correct his upload, thus saving the time of the FTP team
and accelerating the processing of the queue. Even when the processing of the
queue is fast, it would be worthwile to do it before uploading to NEW. I have
documented my approach on our wiki.

http://wiki.debian.org/CopyrightReview

Thanks to the widespread use of VCSes and platforms such as Debian Mentors, the
source of the uploaded packages is often easy to find.

Cheers,

--
Charles Plessy
Tsurumi, Kanagawa, Japan

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130403095...@falafel.plessy.net

[Clint Adams]

On Wed, Apr 03, 2013 at 03:44:48PM +0100, Jonathan Dowland wrote:
> The NEW queue is not just for double-checking licenses.

But it should be.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130403144...@scru.org

[Jonathan Dowland]

The NEW queue is not just for double-checking licenses.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130403144448.GB11273@debian

[Thomas Goirand]

On 04/03/2013 04:34 AM, Thomas Goirand wrote:
> On 04/01/2013 11:06 PM, Luca Falavigna wrote:
>> On the other hand, FTP Team is willing to fast-track NEW packages
>> anytime, if needed.
> That's simply not truth. I can't let you say that and not reply.

Hi,

I would like to publicly thanks Luca for all the FTP Master assistant
work that he did on the Openstack packages recently. Nearly all of
the Openstack packages have been approved, and now I'm down
to python-pecan and websockify, which have been rejected, for
very valid reasons, with 2 or 3 files that are sourceless. I'll work on
them, and create a DFSG version, hoping that I can finish the work
before the next week Openstack summit. Saying "well, all is done
but it's waiting FTP masters approval" is really not the same as
saying "well, yeah, everything is now in Debian" !!! This was a
very frustrating situation a week ago, and what just happened
fills me with a lot of satisfaction. I am really convince that your
work will really make a difference next week, when we will discuss
with Ubuntu guys, and try to convince everyone that Debian is
also a good platform for Openstack.

So again, thanks so much Luca!

Thomas

P.S: I'm unsure if I'll upload all of Grizzly this week to Experimental
or what, if I can fix python-pecan and websockify...

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/51641476...@debian.org

[Mathieu Malaterre]

On Tue, Apr 9, 2013 at 3:15 PM, Thomas Goirand <zi...@debian.org> wrote:
> On 04/03/2013 04:34 AM, Thomas Goirand wrote:
>> On 04/01/2013 11:06 PM, Luca Falavigna wrote:
>>> On the other hand, FTP Team is willing to fast-track NEW packages
>>> anytime, if needed.
>> That's simply not truth. I can't let you say that and not reply.

> So again, thanks so much Luca!

+1
Thanks Luca for your careful review, you manage to still catch glitch
in packages while under pressure !

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CA+7wUsz2gqwY2sa2=823cN6EqC5c052rU...@mail.gmail.com

[Bernd Zeimetz]

On 02.04.2013 22:48, Thomas Goirand wrote:
> On 04/02/2013 12:16 AM, Luca Falavigna wrote:
>> In a perfect world there wouldn't be any need for a NEW queue at
>> all.
>> But we have to face with the reality.
>> We try to do our best to improve things where we can. From the FTP
>> Team side, we always try to be quick and helpful with our fellow
>> developers, and are happy to hear about suggestions how to improve
>> further.
> I got a bunch of suggestions...
>
> Suggestion #1: if a package stays more than a month in the NEW
> queue, then it gets automatically approved, and may be
> reviewed later on. My reasoning is that more than a month,
> it can become really problematic and blocks development.

No. Go back to start and learn why there is a NEW queue.

> Suggestion #2: get rid of the new binary queue (not new source
> package, that's different). There's no reason why the licensing
> of a package changes just because the maintainer decides to add
> a new binary, so it makes no sense. This would save a lot of
> time for the FTP team.

No. Go back to start and learn why there is a NEW queue.

> Suggestion #3: have a system where any other DD can review
> a package in the NEW queue, not only the FTP masters or the
> FTP assistants.

That would include publishing the contents of the NEW queue,
at least to all Debian Developers - so we might violate
licenses already.

> Suggestion #4: recognized that the FTP team needs to work faster,
> and get more people in the FTP team.

When did you read the last announce mail from the FTP team?
They always look for people to join. But it is a lot of
work, so rarely people like to join. Or they don't get into
the team because they fail to understand what they have to
take care of.
So when did you offer yourself to join the FTP team?

> Suggestion #5: make it so that a bunch of packages can be
> reviewed together, as they might depend on each other, and we
> would like to avoid cases where some packages are accepted, but
> can't be installed because their dependencies are in NEW.

And that breaks exactly what? Such a package will never migrate
to testing. No harm done. Also you might want to avoid to depend
on packages not yet in Debian as they might never end up in
Debian at all.

> Suggestion #6: get rid of the NEW queue completely. I'm not
> the only one that thinks it should be like that, and that the
> licensing review process could happen after packages are
> accepted. Maybe though, I'll be the only one saying it out
> loud (but I'm getting used to it...).

No. Go back to start and learn why there is a NEW queue.



--
Bernd Zeimetz Debian GNU/Linux Developer
http://bzed.de http://www.debian.org
GPG Fingerprint: ECA1 E3F2 8E11 2432 D485 DD95 EB36 171A 6FF9 435F

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/3f9836d48fe7ac1b...@mail.recluse.de

[Thomas Goirand]

On 04/09/2013 11:54 PM, Bernd Zeimetz wrote:
> So when did you offer yourself to join the FTP team?

I didn't offer to completely join forever, but I offered my help,
few months ago. Though considering the mistakes I did in
the past (and still do from time to time, despite my (probably
wrong) feeling to do double-checks), I do understand why
they didn't feel comfortable with me checking for licenses.

On 04/09/2013 11:54 PM, Bernd Zeimetz wrote:
>> Suggestion #2: get rid of the new binary queue (not new source
>> package, that's different). There's no reason why the licensing
>> of a package changes just because the maintainer decides to add
>> a new binary, so it makes no sense. This would save a lot of
>> time for the FTP team.
>
> No. Go back to start and learn why there is a NEW queue.

No what? To which part of the above?

Would you care to explain, since I'm so dumb and I should learn?
In what way adding lines in debian/control changes the licensing
of upstream source?

>
>> Suggestion #5: make it so that a bunch of packages can be
>> reviewed together, as they might depend on each other, and we
>> would like to avoid cases where some packages are accepted, but
>> can't be installed because their dependencies are in NEW.
>
> And that breaks exactly what? Such a package will never migrate
> to testing. No harm done. Also you might want to avoid to depend
> on packages not yet in Debian as they might never end up in
> Debian at all.

If I upload new packages A and B, that A depends and B, and
that A gets approved, but B doesn't, then we end up with
package A being in Debian, but never installable.

Now, if what you are suggesting that I should wait for B
to be approved before uploading A, I think you aren't
being realistic when the NEW queue has a 3 months
waiting time. This might work with small projects, but
if you have to maintain a complex set of packages, with
lots of dependencies, it just doesn't work. Been there,
tried that ...

Also, thinking only about testing, when we have a 10 months
period of freeze, is quite crazy. So yes, harm done, even in
Experimental (during the freeze)!

> No. Go back to start and learn why there is a NEW queue.

You didn't need to repeat this sentence 3 times.

I believe I know why we have it, never the less, I feel
like there would be better ways to handle the problem.
I'm only the vocal person here, I know I'm not the only
one thinking this way. Others probably fear the reaction
of the FTP masters, I personally think (and hope) they
are smarter than this and accept constructive critics.

Thomas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/51646364...@debian.org

[Daniel Pocock]

On 09/04/13 17:54, Bernd Zeimetz wrote:
> On 02.04.2013 22:48, Thomas Goirand wrote:
>> On 04/02/2013 12:16 AM, Luca Falavigna wrote:
>>> In a perfect world there wouldn't be any need for a NEW queue at all.
>>> But we have to face with the reality.
>>> We try to do our best to improve things where we can. From the FTP
>>> Team side, we always try to be quick and helpful with our fellow
>>> developers, and are happy to hear about suggestions how to improve
>>> further.
>> I got a bunch of suggestions...
>>
>> Suggestion #1: if a package stays more than a month in the NEW
>> queue, then it gets automatically approved, and may be
>> reviewed later on. My reasoning is that more than a month,
>> it can become really problematic and blocks development.
>
> No. Go back to start and learn why there is a NEW queue.
>
>> Suggestion #2: get rid of the new binary queue (not new source
>> package, that's different). There's no reason why the licensing
>> of a package changes just because the maintainer decides to add
>> a new binary, so it makes no sense. This would save a lot of
>> time for the FTP team.
>
> No. Go back to start and learn why there is a NEW queue.

That answer is not so clear

Plenty of packages have evolved with new upstream releases over many
years without any ongoing review by the FTP masters. I'm sure I could
find one that has subsequently and inadvertently become non-free if I
really looked hard enough.

Why should review only take place on those packages that the maintainer
chooses to modularise?

Isn't it the content of the source package that needs review? Maybe the
review should be triggered by some other factor? For example, every
time a new upstream major release number increment occurs, the upload
goes into NEW?

>> Suggestion #3: have a system where any other DD can review
>> a package in the NEW queue, not only the FTP masters or the
>> FTP assistants.
>
> That would include publishing the contents of the NEW queue,
> at least to all Debian Developers - so we might violate
> licenses already.

That is not a watertight argument either - it would be quite feasible to
publicize the source package without making the upstream tarball public.
Just make sure that other DDs can see a link to the upstream tarball on
the upstream web site, and the hash from the changes file

This would actually be a very good way of helping to distribute the
workload of FTP masters as all DDs could presumably practice rejecting
things, while the decision to accept something would remain with the FTP
master.

I also value the work of the FTP masters and everybody who scrutinizes
packages to make sure they really are free and open. Just look at the
Android market for an example of what would evolve without such efforts.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/516465B6...@pocock.com.au

[Charles Plessy]

Le Tue, Apr 09, 2013 at 05:54:14PM +0200, Bernd Zeimetz a �crit :

>
> >Suggestion #3: have a system where any other DD can review
> >a package in the NEW queue, not only the FTP masters or the
> >FTP assistants.
>
> That would include publishing the contents of the NEW queue,
> at least to all Debian Developers - so we might violate
> licenses already.

I have not read any convincing argument in favor of our current practice, not
to mention that most arguments are guesses on the reasons of the persons in
charge rather than a clear statement from the persons in charge themselves.

We do not have much measures in place to ensure that our archive does not
contain packages that start to violate licenses after their first upload. In
parallel, we have a lot of download points that are not subjected to copyright
and license review. I do not see a reason why the NEW queue must be more
perfect than both our archive and the rest of the non-aptable files we
distribute.

Conversely, the existence of sites such as Ubuntu's PPA, SourceForge, GitHub
and many others show that a large number of software providers are confident
that a policy of a posteriori removals is sufficient. I do not understand why
we do not reach the same conclusion for the NEW queue, which is not even a
software distribution in the sense of the Debian archive or the sites
mentionned above.

Fedora for instance publicly reviews the new packages in a bugtracker, with
download links that sometimes are pointing to Fedora-hosted machines. I think
that reaching that level of transparency would have a positive impact on our
capacity to keep on attracting new contributors.

Cheers,

--
Charles Plessy
Tsurumi, Kanagawa, Japan

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130409225...@falafel.plessy.net

[Thomas Goirand]

On 04/10/2013 06:56 AM, Charles Plessy wrote:

Exactly. Very well said!

Thomas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/5164BE8D...@debian.org

[Joey Hess]

Charles Plessy wrote:
> Conversely, the existence of sites such as Ubuntu's PPA, SourceForge, GitHub
> and many others show that a large number of software providers are confident
> that a policy of a posteriori removals is sufficient. I do not understand why
> we do not reach the same conclusion for the NEW queue, which is not even a
> software distribution in the sense of the Debian archive or the sites
> mentionned above.

One significant difference between those sites and the Debian NEW queue,
or Debian in general is that sites that allow anyone register and upload
content probably operate under the DMCA safe harbor provisions that only
require they take down infringing material when informed of it.

--
see shy jo

[Luca Falavigna]

2013/4/9 Thomas Goirand <zi...@debian.org>:

> If I upload new packages A and B, that A depends and B, and
> that A gets approved, but B doesn't, then we end up with
> package A being in Debian, but never installable.

That is unlikely to happen: dak has a colour scheme to identify
missing packages. It's also nice to identify packages who belong in
main, contrib, and non-free, just to avoid component mismatches.

> Now, if what you are suggesting that I should wait for B
> to be approved before uploading A, I think you aren't
> being realistic when the NEW queue has a 3 months
> waiting time. This might work with small projects, but
> if you have to maintain a complex set of packages, with
> lots of dependencies, it just doesn't work. Been there,
> tried that ...

Uploading packages in NEW which depend on other packages in NEW is
fine, as explained above. Dependencies will be reviewed first, and
when accepted, the other packages will be processed as well. The major
difficulty happens when the dependency chain is very complex (e.g. A
-> B -> C -> D -> E -> A), in that case it would help if maintainers
suggested the order in which to review packages.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CADk7b0Of088FeO66CYpO-8cR...@mail.gmail.com

[Wouter Verhelst]

Hi Charles,

On 10-04-13 00:56, Charles Plessy wrote:
> Le Tue, Apr 09, 2013 at 05:54:14PM +0200, Bernd Zeimetz a �crit :
>>
>>> Suggestion #3: have a system where any other DD can review
>>> a package in the NEW queue, not only the FTP masters or the
>>> FTP assistants.
>>
>> That would include publishing the contents of the NEW queue,
>> at least to all Debian Developers - so we might violate
>> licenses already.
>
> I have not read any convincing argument in favor of our current practice, not
> to mention that most arguments are guesses on the reasons of the persons in
> charge rather than a clear statement from the persons in charge themselves.
>
> We do not have much measures in place to ensure that our archive does not
> contain packages that start to violate licenses after their first upload. In
> parallel, we have a lot of download points that are not subjected to copyright
> and license review. I do not see a reason why the NEW queue must be more
> perfect than both our archive and the rest of the non-aptable files we
> distribute.

It is a mistake to believe that NEW queue handling only exists for the
benefit of license compliance checking. Yes, that is a big part of it,
but AIUI, the ftp-masters need to do a lot more than that for packages
in NEW.

--
Copyshops should do vouchers. So that next time some bureaucracy
requires you to mail a form in triplicate, you can mail it just once,
add a voucher, and save on postage.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/51650EC7...@debian.org

[Joachim Breitner]

Hi,

Am Dienstag, den 09.04.2013, 17:54 +0200 schrieb Bernd Zeimetz:
> > Suggestion #3: have a system where any other DD can review
> > a package in the NEW queue, not only the FTP masters or the
> > FTP assistants.
>
> That would include publishing the contents of the NEW queue,
> at least to all Debian Developers - so we might violate
> licenses already.

I have long stopped buying this argument, with things like
alioth.debian.org, people.debian.org and mentors.debian.net¹ full of
software without license review.

Greetings,
Joachim

¹ ok, it’s .net... but still.

[Jonathan Dowland]

On Wed, Apr 10, 2013 at 02:52:20AM +0800, Thomas Goirand wrote:
> If I upload new packages A and B, that A depends and B, and
> that A gets approved, but B doesn't, then we end up with
> package A being in Debian, but never installable.

Has this ever happened? I believe the FTP masters do look at dependencies
of packages in NEW to prevent this situation.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130410084319.GA3103@debian

[Nicolas Dandrimont]

* Joachim Breitner <nom...@debian.org> [2013-04-10 10:13:25 +0200]:

> Hi,
>
> Am Dienstag, den 09.04.2013, 17:54 +0200 schrieb Bernd Zeimetz:
> > > Suggestion #3: have a system where any other DD can review
> > > a package in the NEW queue, not only the FTP masters or the
> > > FTP assistants.
> >
> > That would include publishing the contents of the NEW queue,
> > at least to all Debian Developers - so we might violate
> > licenses already.
>
> I have long stopped buying this argument, with things like
> alioth.debian.org, people.debian.org and mentors.debian.net¹ full of
> software without license review.
>

> ¹ ok, it’s .net... but still.

<mentors.d.n admin hat on>

For mentors.debian.net, there are two main blockers for a .org transition:
- Seeking an answer to this redistribution without verification problem
- Making the codebase acceptable for DSA administration

The first point has been handled by zack, and we have on hand a legal document,
vetted by SFLC lawyers, that makes the mentors platform a "DMCA safe harbor".

Basically, everyone is still allowed to upload packages, and those packages are
distributed directly, the admins need to leave the copyright owners a way to
claim that a package infringes on their copyright and act swiftly to hide such
packages, pending a possible counterclaim from the uploader.

We need to publish that policy, and then we should be compliant with DMCA safe
harbor policies.

For the second point... well... we're working on it, albeit slowly. People are
welcome to join :)
<mentors.d.n admin hat off>

Cheers,
--
Nicolas Dandrimont

"I once witnessed a long-winded, month-long flamewar over the use of
mice vs. trackballs...It was very silly."
(By Matt Welsh)

[Thomas Goirand]

On 04/10/2013 05:36 PM, Nicolas Dandrimont wrote:
> The first point has been handled by zack, and we have on hand a legal document,
> vetted by SFLC lawyers, that makes the mentors platform a "DMCA safe harbor".

Does it mean that it is mandatory that mentors is hosted in USA?

Thomas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/516548D1...@goirand.fr

[Jonathan Dowland]

On Wed, Apr 10, 2013 at 07:11:13PM +0800, Thomas Goirand wrote:
> On 04/10/2013 05:36 PM, Nicolas Dandrimont wrote:
> > The first point has been handled by zack, and we have on hand a legal document,
> > vetted by SFLC lawyers, that makes the mentors platform a "DMCA safe harbor".
>
> Does it mean that it is mandatory that mentors is hosted in USA?

I thought to ask the opposite: could this work be avoided simply by hosting m.d.o
in a country which does not honour the DMCA? Assuming we have some DSA assets in
such countries. But I presume the mentors admins and/or DSA have thought of that…

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130410125235.GA31605@debian

[Tollef Fog Heen]

]] Jonathan Dowland

> On Wed, Apr 10, 2013 at 07:11:13PM +0800, Thomas Goirand wrote:
> > On 04/10/2013 05:36 PM, Nicolas Dandrimont wrote:
> > > The first point has been handled by zack, and we have on hand a legal document,
> > > vetted by SFLC lawyers, that makes the mentors platform a "DMCA safe harbor".
> >
> > Does it mean that it is mandatory that mentors is hosted in USA?
>
> I thought to ask the opposite: could this work be avoided simply by hosting m.d.o
> in a country which does not honour the DMCA?

It would mean possible interesting legal challenges for people uploading
crypto software from the US, since there would have been an export with
no corresponding BXA declaration. IANAL, but my understanding is that
it could land the person doing the export in quite a bit of trouble.

> Assuming we have some DSA assets in such countries. But I presume the
> mentors admins and/or DSA have thought of that…

We have machines outside the US, yes, the biggest ones are/will be in
.ca, .de, .uk and .gr.

--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/m2vc7u3...@rahvafeir.err.no

[Stefano Zacchiroli]

On Wed, Apr 10, 2013 at 07:11:13PM +0800, Thomas Goirand wrote:

> On 04/10/2013 05:36 PM, Nicolas Dandrimont wrote:
> > The first point has been handled by zack, and we have on hand a legal document,
> > vetted by SFLC lawyers, that makes the mentors platform a "DMCA safe harbor".
>
> Does it mean that it is mandatory that mentors is hosted in USA?

Of course not. It means that if we decide to host in the US, where DMCA
is in effect, then we have the needed legal advice to go forward there.
Hosting it elsewhere means learning about similar legal challenges that
exist in the country of choice [1] and possibly seeking similar advice
*if* there are DMCA-like worries.

FWIW, as a project we have very good access to high quality, pro bono,
US lawyers at SFLC, but nothing equivalent (all factors considered) for
other countries.

Cheers.

[1] unfortunately, DMCA is not the only "bad" draconian law that exists
around the world, many other countries have similar laws
--
Stefano Zacchiroli . . . . . . . za...@upsilon.cc . . . . o . . . o . o
Maître de conférences . . . . . http://upsilon.cc/zack . . . o . . . o o
Debian Project Leader . . . . . . @zack on identi.ca . . o o o . . . o .
« the first rule of tautology club is the first rule of tautology club »

[Charles Plessy]

Le Wed, Apr 10, 2013 at 11:36:03AM +0200, Nicolas Dandrimont a �crit :

>
> For mentors.debian.net, there are two main blockers for a .org transition:
> - Seeking an answer to this redistribution without verification problem
> - Making the codebase acceptable for DSA administration
>
> The first point has been handled by zack, and we have on hand a legal document,
> vetted by SFLC lawyers, that makes the mentors platform a "DMCA safe harbor".
>
> Basically, everyone is still allowed to upload packages, and those packages are
> distributed directly, the admins need to leave the copyright owners a way to
> claim that a package infringes on their copyright and act swiftly to hide such
> packages, pending a possible counterclaim from the uploader.
>
> We need to publish that policy, and then we should be compliant with DMCA safe
> harbor policies.

Hi,

I do not understand the following:

- If mentors.debian.org needs to follow the DMCA, why would
mentors.debian.net be exempt of it ? Also, how do the safer harbor
procedures differ from your current practices ? Surely, if a copyright holder
reports an infringement to sup...@mentors.debian.net, you will remove the
package, isn't it ?

- If mentors.debian.org can distribute unreviewed packages by becomming a
DMCA safe harbor, wouldn't it be possible for ftp-master.debian.org/NEW.html ?

- Bonus question: since mentors.debian.net seems to be hosted in Germany,
does it mean that developers living in the US should refrain from uploading
crypto to it ? How do other distributions solve that problem ?

Cheers,

--
Charles Plessy
Tsurumi, Kanagawa, Japan

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130410135...@falafel.plessy.net

[Jonathan Dowland]

On Wed, Apr 10, 2013 at 10:51:39PM +0900, Charles Plessy wrote:
> - If mentors.debian.org needs to follow the DMCA, why would
> mentors.debian.net be exempt of it ?

It's not, but Debian is not hosting mentors, the .net domain is a forwarding service
of sorts, so to take on the responsibility for hosting, Debian also needs to address
the DMCA issue.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130410143907.GA4314@debian

[Ben Hutchings]

On Wed, Apr 10, 2013 at 03:37:39PM +0200, Stefano Zacchiroli wrote:
> On Wed, Apr 10, 2013 at 07:11:13PM +0800, Thomas Goirand wrote:
> > On 04/10/2013 05:36 PM, Nicolas Dandrimont wrote:
> > > The first point has been handled by zack, and we have on hand a legal document,
> > > vetted by SFLC lawyers, that makes the mentors platform a "DMCA safe harbor".
> >
> > Does it mean that it is mandatory that mentors is hosted in USA?
>
> Of course not. It means that if we decide to host in the US, where DMCA
> is in effect, then we have the needed legal advice to go forward there.
> Hosting it elsewhere means learning about similar legal challenges that
> exist in the country of choice [1] and possibly seeking similar advice
> *if* there are DMCA-like worries.
>
> FWIW, as a project we have very good access to high quality, pro bono,
> US lawyers at SFLC, but nothing equivalent (all factors considered) for
> other countries.
>
> Cheers.
>
> [1] unfortunately, DMCA is not the only "bad" draconian law that exists
> around the world, many other countries have similar laws

The DMCA 'safe harbor' rules are comparatively *good* for service
providers, though not so much for service users that have enemies.

Ben.

--
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
- Albert Camus

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013041015...@decadent.org.uk

[Peter Samuelson]

[Charles Plessy]

> - If mentors.debian.org needs to follow the DMCA, why would
> mentors.debian.net be exempt of it ?

It's not exempt, but it's also not Debian's problem.

> - If mentors.debian.org can distribute unreviewed packages by becomming a
> DMCA safe harbor, wouldn't it be possible for ftp-master.debian.org/NEW.html ?

The difference is that one is open to the public and the other is not.
If a service is open to the public without any control over who can
post content, then basically you have grounds to claim you do not and
cannot reasonably police the content.

> - Bonus question: since mentors.debian.net seems to be hosted in
> Germany, does it mean that developers living in the US should
> refrain from uploading crypto to it ? How do other distributions
> solve that problem ?

Correct, it means developers living in the US need to follow US laws.

I suspect other distributions solve the problem by ignoring it, thus
leaving individuals responsible for obeying their local laws. Which is
a fine principle, but in practice it probably means some individuals
violate US law without really noticing. (The US government harrassment
of Phil Zimmermann was a long time ago, so I suspect that object lesson
has been mostly lost.)

Peter

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013041017...@p12n.org

[Charles Plessy]

Le Wed, Apr 10, 2013 at 12:09:16PM -0500, Peter Samuelson a �crit :

>
> > - If mentors.debian.org can distribute unreviewed packages by becomming a
> > DMCA safe harbor, wouldn't it be possible for ftp-master.debian.org/NEW.html ?
>
> The difference is that one is open to the public and the other is not.
> If a service is open to the public without any control over who can
> post content, then basically you have grounds to claim you do not and
> cannot reasonably police the content.

Is there a legal ground that disqualifies Debian as service provider is the
sense of the DMCA ? I can not upload to Youtube without authentifcating
myself, how different is it from the impossibility to upload to Debian
without signing my packages ?

Alternatively, if there is no safe harbor for the NEW queue because it is
private to Debian, why its contents can not be open privately to the Debian
developers ?

> > - Bonus question: since mentors.debian.net seems to be hosted in
> > Germany, does it mean that developers living in the US should
> > refrain from uploading crypto to it ? How do other distributions
> > solve that problem ?
>
> Correct, it means developers living in the US need to follow US laws.
>
> I suspect other distributions solve the problem by ignoring it, thus
> leaving individuals responsible for obeying their local laws. Which is
> a fine principle, but in practice it probably means some individuals
> violate US law without really noticing. (The US government harrassment
> of Phil Zimmermann was a long time ago, so I suspect that object lesson
> has been mostly lost.)

I am still puzzled: if we host a service in the US, this helps the US
developers, but this still leaves the other developers living in other
countries under the threat of export restrictions from their local law. Does
that mean that we chose US because it minimises the total number of developers
who have to care about export restrictions, or does that mean that in the end,
if only considering cryptograhpy, the servers could be hosted in other
countries, because anyway there will always be a majority of developers
who need to cross a border ?

Alternatively, doesn't the fact that we seem to be the only ones to
self-inflict so many procedures suggest that we are the ones overinterpreting
or misinterpreting the laws ?

Cheers

--
Charles Plessy
Tsurumi, Kanagawa, Japan

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013041023...@falafel.plessy.net

[Nikolaus Rath]

Charles Plessy <ple...@debian.org> writes:

> Le Wed, Apr 10, 2013 at 11:36:03AM +0200, Nicolas Dandrimont a écrit :
> - Bonus question: since mentors.debian.net seems to be hosted in Germany,
> does it mean that developers living in the US should refrain from uploading
> crypto to it ? How do other distributions solve that problem ?

More interesting (in my opinion): if US developers can safely upload
crypto packages to US hosted debian servers, but Debian then makes these
packages available to everyone in the world for download, why isn't this
export of cryptographic packages a problem for Debian?

Best,

-Nikolaus

--
»Time flies like an arrow, fruit flies like a Banana.«

PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87a9p51x...@vostro.rath.org

[Russ Allbery]

Nikolaus Rath <Niko...@rath.org> writes:

> More interesting (in my opinion): if US developers can safely upload
> crypto packages to US hosted debian servers, but Debian then makes these
> packages available to everyone in the world for download, why isn't this
> export of cryptographic packages a problem for Debian?

Because Debian registers that export with the US government and used to
inform the government of every package until the government asked us to
stop.

US developers would otherwise have to do the same thing to export
software with cryptographic code, at least in theory, to satisfy the law.

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87a9p54...@windlord.stanford.edu

[Joerg Jaspert]

On 13178 March 1977, Nikolaus Rath wrote:
> Charles Plessy <ple...@debian.org> writes:

>> Le Wed, Apr 10, 2013 at 11:36:03AM +0200, Nicolas Dandrimont a �crit :

>> - Bonus question: since mentors.debian.net seems to be hosted in Germany,
>> does it mean that developers living in the US should refrain from uploading
>> crypto to it ? How do other distributions solve that problem ?
> More interesting (in my opinion): if US developers can safely upload
> crypto packages to US hosted debian servers, but Debian then makes these
> packages available to everyone in the world for download, why isn't this
> export of cryptographic packages a problem for Debian?

https://ftp-master.debian.org/crypto-in-main/

Plus one mail for *every* NEW accepted package. Each and every time.
Send to them.[1] See the dak git repo for the bxa stuff.


[1] Nowadays only stored in a mailbox from us, at request from them, but
available if requested.

--
bye, Joerg
Mr. Scorpio says productivity is up 2%, and it's all because of
my motivational techniques -- like donuts and the possibility of more
donuts to come.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/8738uxm...@gkar.ganneff.de

[Alberto Fuentes]

On 04/11/2013 08:27 AM, Joerg Jaspert wrote:
>
> https://ftp-master.debian.org/crypto-in-main/
>
> Plus one mail for *every* NEW accepted package. Each and every time.
> Send to them.[1] See the dak git repo for the bxa stuff.
>
>
> [1] Nowadays only stored in a mailbox from us, at request from them, but
> available if requested.

Okay. *that* is crazy.

Crazy as in insane. Crazy as in coocoo

For so many reasons

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/51668C1C...@qindel.com

[Charles Plessy]

Le Thu, Apr 11, 2013 at 08:27:16AM +0200, Joerg Jaspert a �crit :

>
> https://ftp-master.debian.org/crypto-in-main/
>
> Plus one mail for *every* NEW accepted package. Each and every time.
> Send to them.[1] See the dak git repo for the bxa stuff.
>
>
> [1] Nowadays only stored in a mailbox from us, at request from them, but
> available if requested.

Hi Joerg,

since we are not sending the emails anymore, what would prevent us from
automatically adding to the mailbox one email per package entering the NEW
queue, and opening the contents of the NEW queue to the public ?

Cheers,

--
Charles Plessy
Tsurumi, Kanagawa, Japan

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130411105...@falafel.plessy.net

[Thorsten Glaser]

Joachim Breitner <nomeata <at> debian.org> writes:

> The (luxury) problem is that I got used to it and began uploading the
> new (and NEW) dependency bar of package foo along with the new version
> of foo (instead of uploading bar first, wait for NEW processing and only

I think you shouldn’t do that anyway.

After all, to do that, you’d have to manually install the new version
of bar in the chroot you used to build foo, which is a violation of the
“clean, minimal chroot” rule if read strictly (and, from experience with
bad non-buildd uploads, I tend to read it strictly).

So, normally, you can only upload a new version of foo after bar gets
available on your regular mirror. (Making it temporarily available
locally for development is, of course, okay – just don’t upload that.)

bye,
//mirabilos (who sees the m68k buildds build many NEW packages now)

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/loom.2013042...@post.gmane.org

[Joachim Breitner]

Hi,

Am Dienstag, den 23.04.2013, 15:15 +0000 schrieb Thorsten Glaser:
> So, normally, you can only upload a new version of foo after bar gets
> available on your regular mirror. (Making it temporarily available
> locally for development is, of course, okay – just don’t upload that.)

sounds good in theory, but in practice, when I want to upgrade foo,
which happens to have a new dependency on bar, which needs baz, which
needs qux, then I really want to get this done in one rush, and not wait
for three NEW processings in between.

Luckily, a permanent reject in NEW is rare enough that this process is,
at least after NEW processing, fine.

Greetings,
Joachim

[Russ Allbery]

Joachim Breitner <nom...@debian.org> writes:

> sounds good in theory, but in practice, when I want to upgrade foo,
> which happens to have a new dependency on bar, which needs baz, which
> needs qux, then I really want to get this done in one rush, and not wait
> for three NEW processings in between.

I go through this process with Shibboleth routinely, and it really doesn't
seem that bad to me. There were some delays with NEW recently during the
work on the release, but normally the turnaround for a simple SONAME
change in an existing library is <2 days, which doesn't hold me up that
much. I prepare all the packages in advance, and then just build and
upload the next one as I get the acceptance for the previous one.

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/874new7...@windlord.stanford.edu

[Chow Loong Jin]

On 23/04/2013 23:15, Thorsten Glaser wrote:
> Joachim Breitner <nomeata <at> debian.org> writes:
>
>> The (luxury) problem is that I got used to it and began uploading the
>> new (and NEW) dependency bar of package foo along with the new version
>> of foo (instead of uploading bar first, wait for NEW processing and only
>
> I think you shouldn’t do that anyway.
>
> After all, to do that, you’d have to manually install the new version
> of bar in the chroot you used to build foo, which is a violation of the
> “clean, minimal chroot” rule if read strictly (and, from experience with
> bad non-buildd uploads, I tend to read it strictly).

If you use a local APT repository containing "bar", then it still fulfils the
"clean, minimal chroot" rule, since the debs you have uploaded aren't rebuilt
for the particular architecture you're building "foo" on anyway (assuming that
you use the same debs you uploaded to the Debian archive).

--
Kind regards,
Loong Jin

[Goswin von Brederlow]

On Thu, Apr 25, 2013 at 09:44:58AM +0800, Chow Loong Jin wrote:
> On 23/04/2013 23:15, Thorsten Glaser wrote:
> > Joachim Breitner <nomeata <at> debian.org> writes:
> >
> >> The (luxury) problem is that I got used to it and began uploading the
> >> new (and NEW) dependency bar of package foo along with the new version
> >> of foo (instead of uploading bar first, wait for NEW processing and only
> >

> > I think you shouldn???t do that anyway.
> >
> > After all, to do that, you???d have to manually install the new version

> > of bar in the chroot you used to build foo, which is a violation of the

> > ???clean, minimal chroot??? rule if read strictly (and, from experience with

> > bad non-buildd uploads, I tend to read it strictly).
>
> If you use a local APT repository containing "bar", then it still fulfils the
> "clean, minimal chroot" rule, since the debs you have uploaded aren't rebuilt
> for the particular architecture you're building "foo" on anyway (assuming that
> you use the same debs you uploaded to the Debian archive).

By uploading foo with a dependency on a NEW package you make foo
uninstallable for a time. Which on its own is already a bad thing.

Worse it only works if bar is a new source package or you have a
versioned Build-Depends on the new version. Otherwise all buildds will
simply compile the new foo against the old bar and then you have one
arch where foo is uninstallable while all others work.

Maybe this calls for an upload manager or a dependency based delayed
upload queue. You can prepare and sign the upload at any time after
all. Only the upload itself needs to be timed to the NEW processing.

MfG
Goswin

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130502090646.GC29888@frosties

[Andreas Beckmann]

On 2013-05-02 11:06, Goswin von Brederlow wrote:
> versioned Build-Depends on the new version. Otherwise all buildds will
> simply compile the new foo against the old bar and then you have one
> arch where foo is uninstallable while all others work.

This is quickly "fixed" by doing a binNMU on the uploaded-arch.
People do this all the time: upload packages built against local packages,
experimental or even on Ubuntu to Debian sid.

Rebuilding against the NEW package can be done by binNMUs (with proper
dep-waits), but that very much smells like a transition that is to be
handled by the release team.

> Maybe this calls for an upload manager or a dependency based delayed
> upload queue. You can prepare and sign the upload at any time after
> all. Only the upload itself needs to be timed to the NEW processing.

That will only work if the "delayed" package adds a versioned B-D on the
NEW package (so it comes with an explicit dep-wait), otherwise it will
be uploaded after NEW processing, but before the NEW package was built
everywhere. And rebuilding the "old" package may have higher precedence
than building something completely NEW on architecture $slow.


Andreas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/5182328A...@debian.org

[Holger Levsen]

Hi,

On Donnerstag, 2. Mai 2013, Andreas Beckmann wrote:
> People do this all the time: upload packages built against local packages,
> experimental or even on Ubuntu to Debian sid.

/me shivers. This hurts. There is no reason not to rebuild in sane
environments. Can we please fix this for the next release?!


cheers,
Holger

[Neil Williams]

After Wheezy is released, we can talk about throwing away all binary
uploads again... if we can't prevent people doing the wrong thing, we
might have to send bits of what gets uploaded to /dev/null.

--


Neil Williams
=============
http://www.linux.codehelp.co.uk/

[Cyril Brulebois]

Holger Levsen <hol...@layer-acht.org> (02/05/2013):

Can we really fix people?

Mraw,
KiBi.

[Goswin von Brederlow]

On Thu, May 02, 2013 at 11:31:54AM +0200, Andreas Beckmann wrote:
> On 2013-05-02 11:06, Goswin von Brederlow wrote:
> > versioned Build-Depends on the new version. Otherwise all buildds will
> > simply compile the new foo against the old bar and then you have one
> > arch where foo is uninstallable while all others work.
>
> This is quickly "fixed" by doing a binNMU on the uploaded-arch.
> People do this all the time: upload packages built against local packages,
> experimental or even on Ubuntu to Debian sid.
>
> Rebuilding against the NEW package can be done by binNMUs (with proper
> dep-waits), but that very much smells like a transition that is to be
> handled by the release team.

*shiver* Some people are realy broken.

> > Maybe this calls for an upload manager or a dependency based delayed
> > upload queue. You can prepare and sign the upload at any time after
> > all. Only the upload itself needs to be timed to the NEW processing.
>
> That will only work if the "delayed" package adds a versioned B-D on the
> NEW package (so it comes with an explicit dep-wait), otherwise it will
> be uploaded after NEW processing, but before the NEW package was built
> everywhere. And rebuilding the "old" package may have higher precedence
> than building something completely NEW on architecture $slow.
>
>
> Andreas

No, if the package has a versioned B-D then the buildds already do the
right thing and the delayed queue would be pointless.

I was thinking more of something that detects the depends on the
uploaded binary and figures out you build against a too new version of
a lib or of uploading a command file along with the package that says
what to wait for.

The queue would have to scan the Packages.gz files (hopefully
including incoming to minimize delays) to decide when to upload a
package. Going just by Sources.gz would defeat the purpose again.

Another usefull feature would then be for the queue to notify the
uploader when a package being waited on failed to build. Otherwise the
upload would wait forever.

MfG
Goswin

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130502145926.GA2080@frosties

[Russ Allbery]

We can by rebuilding all packages from source, including on the uploaded
architecture. Then at worst they will be consistently broken across all
architectures. :)

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87ip31q...@windlord.stanford.edu

[Andrey Rahmatullin]

On Thu, May 02, 2013 at 11:00:09AM -0700, Russ Allbery wrote:
> >>> People do this all the time: upload packages built against local
> >>> packages, experimental or even on Ubuntu to Debian sid.
>
> >> /me shivers. This hurts. There is no reason not to rebuild in sane
> >> environments. Can we please fix this for the next release?!
>
> > Can we really fix people?
>
> We can by rebuilding all packages from source, including on the uploaded
> architecture. Then at worst they will be consistently broken across all
> architectures. :)

Don't forget arch:all

--
WBR, wRAR

[Russ Allbery]

Yes, speaking as someone who has, on several occasions, uploaded arch: all
binary packages with source package problems and not discovered that until
months later via a FTBFS bug from an archive rebuild, I think we should
rebuild all arch: all packages on upload as well.

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/8761z1q...@windlord.stanford.edu

[Chow Loong Jin]

On 02/05/2013 18:48, Neil Williams wrote:
> After Wheezy is released, we can talk about throwing away all binary
> uploads again... if we can't prevent people doing the wrong thing, we
> might have to send bits of what gets uploaded to /dev/null.

While we're at it, can we also have source-only uploads? Uploading potentially
huge binary packages that just go to /dev/null seems like a pointless waste of
bandwidth to me, and the only for argument I've heard (which I don't buy) is "so
that we know maintainers have test-built their packages."

[Emilio Pozuelo Monfort]

Isn't that already possible?

Emilio

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/518360DF...@debian.org

[Chow Loong Jin]

On 03/05/2013 15:01, Emilio Pozuelo Monfort wrote:
> Isn't that already possible?

It is? I should try that out with my next upload.

[Chow Loong Jin]

On 03/05/2013 16:12, Arto Jantunen wrote:

> Chow Loong Jin <hype...@debian.org> writes:
>
>> On 03/05/2013 15:01, Emilio Pozuelo Monfort wrote:
>>> Isn't that already possible?
>>
>> It is? I should try that out with my next upload.
>

> No, it's not. Source only uploads were banned many years ago, mainly due
> to problems with maintainers not even build testing their packages.

Weird, why doesn't Ubuntu have the same issues?

[Julien Cristau]

On Fri, May 3, 2013 at 09:01:51 +0200, Emilio Pozuelo Monfort wrote:

> On 05/03/2013 03:18 AM, Chow Loong Jin wrote:
> > While we're at it, can we also have source-only uploads? Uploading potentially
> > huge binary packages that just go to /dev/null seems like a pointless waste of
> > bandwidth to me, and the only for argument I've heard (which I don't buy) is "so
> > that we know maintainers have test-built their packages."
>
> Isn't that already possible?
>

No.

Cheers,
Julien

[Andrey Rahmatullin]

On Fri, May 03, 2013 at 09:01:51AM +0200, Emilio Pozuelo Monfort wrote:
> >> After Wheezy is released, we can talk about throwing away all binary
> >> uploads again... if we can't prevent people doing the wrong thing, we
> >> might have to send bits of what gets uploaded to /dev/null.
> >
> > While we're at it, can we also have source-only uploads? Uploading potentially
> > huge binary packages that just go to /dev/null seems like a pointless waste of
> > bandwidth to me, and the only for argument I've heard (which I don't buy) is "so
> > that we know maintainers have test-built their packages."
>
> Isn't that already possible?

Of course no.

--
WBR, wRAR

[Arto Jantunen]

Chow Loong Jin <hype...@debian.org> writes:

> On 03/05/2013 15:01, Emilio Pozuelo Monfort wrote:
>> Isn't that already possible?
>
> It is? I should try that out with my next upload.

No, it's not. Source only uploads were banned many years ago, mainly due
to problems with maintainers not even build testing their packages.

--
Arto Jantunen

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87wqrg8...@kirika.int.wmdata.fi

[Thomas Goirand]

On 05/03/2013 02:12 AM, Russ Allbery wrote:
>
> Yes, speaking as someone who has, on several occasions, uploaded arch: all
> binary packages with source package problems and not discovered that until
> months later via a FTBFS bug from an archive rebuild, I think we should
> rebuild all arch: all packages on upload as well.

Same experience over here. Same opinion therefore.

I think there's a consensus, the problem is who's going
to do the work for automating dropping of binaries and
rebuild.

Thomas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/51837B2...@debian.org

[Wouter Verhelst]

They do. They just ignore the issue; they can do that because it's a
scalability issue that, ultimately, can be fixed by throwing large
amounts of hardware (for more build hosts) at it. Debian does not have
that luxury.

--
Copyshops should do vouchers. So that next time some bureaucracy
requires you to mail a form in triplicate, you can mail it just once,
add a voucher, and save on postage.

[Thijs Kinkhorst]

On Fri, May 3, 2013 15:09, Wouter Verhelst wrote:
>> > No, it's not. Source only uploads were banned many years ago, mainly
>> due
>> > to problems with maintainers not even build testing their packages.

> They do. They just ignore the issue; they can do that because it's a
> scalability issue that, ultimately, can be fixed by throwing large
> amounts of hardware (for more build hosts) at it. Debian does not have
> that luxury.

I'm not sure how we can know the amount of unneeded build failures would
be caused by such a strategy. Why do you think the current infrastructure
can't handle these cases? How many do you think there will be, as a
percentage?

I would think that the only way to know that is to try and see actually
how much of these uploads would happen as a percentage of the total number
of uploads. This extra cost in wasted cycles would then have to be offset
to the benefit to DD's, especially those with huge packages and/or slow
connections.

In general I believe that within Debian's available sets of resources,
compute cycles is probably the one we have the most of (and can be
bought). Compare that with DD time and effort, which we probably have the
least of. So if we can save DD time and effort while spending more cycles,
I'm all for it.


Thijs

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/11ee417402734629401c...@aphrodite.kinkhorst.nl

[Josselin Mouette]

Le vendredi 03 mai 2013 à 09:18 +0800, Chow Loong Jin a écrit :
> While we're at it, can we also have source-only uploads? Uploading potentially
> huge binary packages that just go to /dev/null seems like a pointless waste of
> bandwidth to me, and the only for argument I've heard (which I don't buy) is "so
> that we know maintainers have test-built their packages."

There is a solution to both the upload bandwidth problem and the the
problem that buildd binaries are untested, but I’m afraid it implies
changes to dak.

This means configuring dak to accepting only two types of uploads:
- source-only uploads
They are pushed to the buildds, and the produced binaries
(including arch:all) are put in a staging area (much like
incoming.d.o). These binaries can be downloaded, but
the .changes cannot (to forbid skipping the second step).
- binary-changes-only uploads, without binaries
The developer uploads a sole .changes referencing the set of
binaries he has downloaded (and tested, although it is hard to
force that step). Anything referencing binaries not built on the
buildds is ditched.

This way, you ensure that the actual binaries ending up in the archive
have been tested, which is neither the case with just source-only
uploads (no binaries tested) nor with ditched-binary uploads (the binary
might be built in a different environment).

Cheers,
--
.''`. Josselin Mouette
: :' :
`. `'
`-

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/1367591920.3585.796.camel@pi0307572

[Bernhard R. Link]

* Holger Levsen <hol...@layer-acht.org> [130502 12:28]:

> > People do this all the time: upload packages built against local packages,
> > experimental or even on Ubuntu to Debian sid.
>
> /me shivers. This hurts. There is no reason not to rebuild in sane
> environments. Can we please fix this for the next release?!

I'm not sure the cure is not worse than the problem.

Apart from the big problem of making it even easier for people to
upload trash without testing it (both wasting buildd resources and
lettings users install broken packages which any trivial testing would
have catched, from which I've seen far too many), reducing the
buildability of packages is a cruical problem for freedom.

If we step towards rebuilding everything in a highly artifical
environment, it should be made clear that a package having missing
build-conflicts or any other bug preventing it from being built on
a real system should still be important bugs afterwards.

Once we drop that and only give people the right to modify the
software we distribute but no longer the possiblity to do so
on their own, the "Free" we are so proud on gets mood.

Also build systems tend to degrade quite heavily over time and
get more and more specific. In some years we might not be able
to switch to some other builder tools as too many packages depend
on the specifics of the ones we currently have.


Bernhard R. Link

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013050317...@client.brlink.eu

[Nick Andrik]

2013/5/3 Josselin Mouette <jo...@debian.org>:

> There is a solution to both the upload bandwidth problem and the the
> problem that buildd binaries are untested, but I’m afraid it implies
> changes to dak.
>
> This means configuring dak to accepting only two types of uploads:
> - source-only uploads
> They are pushed to the buildds, and the produced binaries
> (including arch:all) are put in a staging area (much like
> incoming.d.o). These binaries can be downloaded, but
> the .changes cannot (to forbid skipping the second step).
> - binary-changes-only uploads, without binaries
> The developer uploads a sole .changes referencing the set of
> binaries he has downloaded (and tested, although it is hard to
> force that step). Anything referencing binaries not built on the
> buildds is ditched.
>
> This way, you ensure that the actual binaries ending up in the archive
> have been tested, which is neither the case with just source-only
> uploads (no binaries tested) nor with ditched-binary uploads (the binary
> might be built in a different environment).

Isn't this solution like a "P"PA where the (tested) packages are
forwarded to the main archive?
I remember reading about an effort to have something like PPAs in
debian (I cannot remember how it is called exactly), so why not
combine the two things?

Nick
--
=Do-
N.AND

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CANn5kOtQLyXEnuQThvAiqQg...@mail.gmail.com

[Timo Juhani Lindfors]

"Bernhard R. Link" <brl...@debian.org> writes:
> Once we drop that and only give people the right to modify the
> software we distribute but no longer the possiblity to do so
> on their own, the "Free" we are so proud on gets mood.

Doesn't pbuilder make it easy enough for anyone to modify and build the
software on their own?

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/84li7vb...@sauna.l.org

[Joachim Breitner]

Hi,

Am Samstag, den 04.05.2013, 00:10 +0300 schrieb Timo Juhani Lindfors:
> "Bernhard R. Link" <brl...@debian.org> writes:
> > Once we drop that and only give people the right to modify the
> > software we distribute but no longer the possiblity to do so
> > on their own, the "Free" we are so proud on gets mood.
>
> Doesn't pbuilder make it easy enough for anyone to modify and build the
> software on their own?

easier, but not as easy as
sudo apt-get build-dep foo; apt-get source foo; #hack; dpkg-buildpackage; sudo debi

As a Debian user, I rely on and enjoy being able to build Debian
packages as they are, on the system that I want to run them, without
configuring a build environment or changeroots. I agree with Bernhard
that we should pay attention not to lose that feature.

Greetings,
Joachim

--
Joachim "nomeata" Breitner
Debian Developer
nom...@debian.org | ICQ# 74513189 | GPG-Keyid: 4743206C
JID: nom...@joachim-breitner.de | http://people.debian.org/~nomeata

[brian m. carlson]

On Sat, May 04, 2013 at 12:10:25AM +0300, Timo Juhani Lindfors wrote:
> "Bernhard R. Link" <brl...@debian.org> writes:
> > Once we drop that and only give people the right to modify the
> > software we distribute but no longer the possiblity to do so
> > on their own, the "Free" we are so proud on gets mood.
>
> Doesn't pbuilder make it easy enough for anyone to modify and build the
> software on their own?

The issue with sterile build environments is not just for building
packages for normal use. If I'm fixing a bug in a package, I may need
to build that package several times, testing different fixes. If
everyone assumes that packages will be built in a sterile environment,
nobody will care that their packages don't build twice in a row, but
that's exactly the situation that I have when trying to test a patch.
Also, I'm not going to set up an entire chroot or sbuild or pbuilder
environment just to test a patch. Packages should definitely be built
for the archive in a clean environment, but they should still build
correctly in an unclean one.

--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[Russ Allbery]

"brian m. carlson" <san...@crustytoothpaste.net> writes:

> The issue with sterile build environments is not just for building
> packages for normal use. If I'm fixing a bug in a package, I may need
> to build that package several times, testing different fixes. If
> everyone assumes that packages will be built in a sterile environment,
> nobody will care that their packages don't build twice in a row, but
> that's exactly the situation that I have when trying to test a patch.
> Also, I'm not going to set up an entire chroot or sbuild or pbuilder
> environment just to test a patch. Packages should definitely be built
> for the archive in a clean environment, but they should still build
> correctly in an unclean one.

It's not far from the truth to say that the only features in any piece of
software that will work reliably are those that are tested. Currently,
the initial architecture upload is produced by a wide variety of different
techniques; some of us build directly on the host system, some build in
pbuilder, some in cowbuilder, some in sbuild, etc. This does not result
in any reliable testing of builds in unclean environments. Any testing of
that done via the initial architecture build is accidental, when viewed
across the archive as a whole.

The way to ensure that builds in non-clean environments work properly is
to devise a method for testing them, and to do those tests on a regular
basis and turn test failures into bugs.

Trying to get at this testing indirectly by putting conditions on initial
archive uploads doesn't really accomplish the goal. It's a very random
and scattershot way of testing that already doesn't work for any of us who
use pbuilder and cowbuilder already.

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87vc6z1...@windlord.stanford.edu

[Bernhard R. Link]

* Russ Allbery <r...@debian.org> [130504 00:32]:

> The way to ensure that builds in non-clean environments work properly is
> to devise a method for testing them, and to do those tests on a regular
> basis and turn test failures into bugs.

Noone is speaking about non-clean environments, but only about
non-minimal, non-artifical ones.

> Trying to get at this testing indirectly by putting conditions on initial
> archive uploads doesn't really accomplish the goal. It's a very random
> and scattershot way of testing that already doesn't work for any of us who
> use pbuilder and cowbuilder already.

That's why I think maintainers should not only build in pbuilders and
cowbuilders, but give their packages some actual testing.

Bernhard R. Link

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130503230...@client.brlink.eu

[Jakub Wilk]

* Arto Jantunen <vi...@debian.org>, 2013-05-03, 11:12:

>Source only uploads were banned many years ago, mainly due to problems
>with maintainers not even build testing their packages.

[citation needed]

--
Jakub Wilk

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013050323...@jwilk.net

[Russ Allbery]

"Bernhard R. Link" <brl...@debian.org> writes:

> * Russ Allbery <r...@debian.org> [130504 00:32]:

>> The way to ensure that builds in non-clean environments work properly
>> is to devise a method for testing them, and to do those tests on a
>> regular basis and turn test failures into bugs.

> Noone is speaking about non-clean environments, but only about
> non-minimal, non-artifical ones.

That's what I mean by "non-clean."

>> Trying to get at this testing indirectly by putting conditions on
>> initial archive uploads doesn't really accomplish the goal. It's a
>> very random and scattershot way of testing that already doesn't work
>> for any of us who use pbuilder and cowbuilder already.

> That's why I think maintainers should not only build in pbuilders and
> cowbuilders, but give their packages some actual testing.

We can think maintainers "should" do a lot of things. The reality of
software development is that if you don't test it, it doesn't happen. We
can ask every maintainer to set up a test procedure for this, and some of
them even will. But many of them won't. If we want to support this
feature across the project, and I agree that I would like us to, then we
need to test it across the project and turn the failures into bugs,
rejects, or some other sort of feedback.

This is similar to the argument for continuous integration systems. Yes,
the benefit is marginal if the developers always run the unit tests before
every commit (and there aren't a lot of environment- or architecture-
sensitive properties in the code). If you have developers that always do
that, you may not need continuous integration. If your developers are
like most of us and are not actually that good at maintaining that level
of discipline, continuous integration takes care of that for them.

Remembering a checklist of things to do with each upload is the sort of
thing computers are much better at doing than people.

--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87mwsb1...@windlord.stanford.edu

[Ben Hutchings]

On Sat, May 04, 2013 at 01:05:06AM +0200, Bernhard R. Link wrote:
> * Russ Allbery <r...@debian.org> [130504 00:32]:
> > The way to ensure that builds in non-clean environments work properly is
> > to devise a method for testing them, and to do those tests on a regular
> > basis and turn test failures into bugs.
>
> Noone is speaking about non-clean environments, but only about
> non-minimal, non-artifical ones.

A minimal environment is the correct way to test that nothing is
missing from Build-Depends.

I assume you're concerned that there may be undeclared build-
conflicts. But testing in the maintainer's development system is not
a particularly good way to find those. Testing in a maximal
environment (everything with priority <= optional, minus declared
Build-Conflicts) might be. I think someone has even tried that
before.

> > Trying to get at this testing indirectly by putting conditions on initial
> > archive uploads doesn't really accomplish the goal. It's a very random
> > and scattershot way of testing that already doesn't work for any of us who
> > use pbuilder and cowbuilder already.
>
> That's why I think maintainers should not only build in pbuilders and
> cowbuilders, but give their packages some actual testing.

Actual testing of Debian means auto-building and testing the resulting
binaries. Anything else is secondary, though it is certainly very
irritating when packages cannot be built and incrementally rebuilt in
a normal installation.

Ben.

--
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
- Albert Camus

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013050323...@decadent.org.uk

[Michael Gilbert]

On Fri, May 3, 2013 at 9:25 AM, Thijs Kinkhorst wrote:
> On Fri, May 3, 2013 15:09, Wouter Verhelst wrote:
>>> > No, it's not. Source only uploads were banned many years ago, mainly
>>> due
>>> > to problems with maintainers not even build testing their packages.
>
>> They do. They just ignore the issue; they can do that because it's a
>> scalability issue that, ultimately, can be fixed by throwing large
>> amounts of hardware (for more build hosts) at it. Debian does not have
>> that luxury.
>
> I'm not sure how we can know the amount of unneeded build failures would
> be caused by such a strategy. Why do you think the current infrastructure
> can't handle these cases? How many do you think there will be, as a
> percentage?

I wonder if a somewhat straightforward solution would be for dak to
delay its usual processing (for source-only uploads only) until the
buildds produce binary packages on at least one arch? Then, in the
case of a build failure on all archs, the upload would of course be
automatically rejected.

This kind of approach would at least keep fully unbuildable packages
out of unstable (it doesn't prevent arch-specific build failures),
which is what I think people really want. This of course would
require some additional communication paths between dak and the
buildds.

Best wishes,
Mike

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CANTw=MPyq=f=Bw2-2K8_ZnQroOGA851N3g-r=h2V=GWtS...@mail.gmail.com

[Arto Jantunen]

Jakub Wilk <jw...@debian.org> writes:

> * Arto Jantunen <vi...@debian.org>, 2013-05-03, 11:12:
>> Source only uploads were banned many years ago, mainly due to problems with
>> maintainers not even build testing their packages.
>
> [citation needed]

Indeed. I was fairly certain that a policy decision about this was made
at some point around ten years ago, but after a bit of searching it
seems to me that this isn't actually the case.

In 2001 AJ listed a bunch of technical reasons why source only uploads
can't currently be done (
https://lists.debian.org/debian-devel-announce/2001/01/msg00000.html
). After that the next relevant mail seems to be the fairly recent
https://lists.debian.org/debian-devel-announce/2009/11/msg00001.html
from Ganneff. My own (grep-friendly) d-d-a archive only goes back to
2002 so it's possible that there was another relevant mail in between,
but my google-fu simply isn't strong enough to find it..

I find it hilarious that we seem to have been waiting for a couple of
rather minor technical fixes for the past 12 years or so.

--
Arto Jantunen

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87sj23q...@iki.fi

[Wookey]

+++ brian m. carlson [2013-05-03 21:39 +0000]:

> On Sat, May 04, 2013 at 12:10:25AM +0300, Timo Juhani Lindfors wrote:
> > "Bernhard R. Link" <brl...@debian.org> writes:
> > > Once we drop that and only give people the right to modify the
> > > software we distribute but no longer the possiblity to do so
> > > on their own, the "Free" we are so proud on gets mood.
> >
> > Doesn't pbuilder make it easy enough for anyone to modify and build the
> > software on their own?
>
> The issue with sterile build environments is not just for building
> packages for normal use. If I'm fixing a bug in a package, I may need
> to build that package several times, testing different fixes. If
> everyone assumes that packages will be built in a sterile environment,
> nobody will care that their packages don't build twice in a row,

This is a good point. I've been doing a lot of building of stuff
(mostly the core/base ~200 packages) with small fixes recently and
it's clear that 'doesn't build a second time' is increasingly common
(and very annoying). This is a result of maintainer's workflows never
doing this, I presume. As Russ said - if it's not tested it probably
doesn't work.

I am huge fan of both building in clean environments _and_ being able
to build twice. I don't think there is any solution to this other than
testing it in an automated fashion. An sbuild or pbuilder option for
--build-twice would make testing a very simple matter.

Does policy say anything about this? Can it in fact be deemed FTBFS if
it fails the second time? I think it should.

Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013050409...@stoneboat.aleph1.co.uk

[Julian Taylor]

On 04.05.2013 11:10, Wookey wrote:
>
> I am huge fan of both building in clean environments _and_ being able
> to build twice. I don't think there is any solution to this other than
> testing it in an automated fashion. An sbuild or pbuilder option for
> --build-twice would make testing a very simple matter.
>

pbuilder and cowbuilder already support the --twice option since a while.

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/5184D981...@googlemail.com

[Jakub Wilk]

* Wookey <woo...@wookware.org>, 2013-05-04, 10:10:

>I am huge fan of both building in clean environments _and_ being able
>to build twice. I don't think there is any solution to this other than
>testing it in an automated fashion. An sbuild or pbuilder option for
>--build-twice would make testing a very simple matter.

Or a dpkg-buildpackage option, so that you don't have to implement in
each and every dpkg-buildpackage wrapper. Unfortunately #671074 wasn't
well-received by dpkg maintainers.

>Does policy say anything about this?

It says that the "clean" target should work. :)

>Can it in fact be deemed FTBFS if it fails the second time? I think it
>should.

We've always treated "FTBFS if built twice in a row" bugs as important:
http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debi...@lists.debian.org;tag=qa-doublebuild

--
Jakub Wilk

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013050410...@jwilk.net

[Xavier Roche]

Le 02/05/2013 20:12, Russ Allbery a écrit :
> Yes, speaking as someone who has, on several occasions, uploaded arch: all
> binary packages with source package problems and not discovered that until
> months later via a FTBFS bug from an archive rebuild, I think we should
> rebuild all arch: all packages on upload as well.

Cough.. cough.. it's actually possible to upload binary-independent
packages only, among with the sources, by cleaning up the changes file
(and renaming _arch into _indep), and re-signing it. This way, you are
still building everything on the local arch, but have the guarantee that
the final binaries will be produced in a clean environment.

Building binaries is not sufficient to prevent breakages, anyway (I did
choke several times due to incomplete build-depend on the control file -
something that you can not detect unless you setup a complete chrooted
build environment, which is a bit cumbersome to do)

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/51850F1...@httrack.com

[Lucas Nussbaum]

Hi,

I worked on archive-wide tests of double-builds a long time ago. Hacking
sbuild to do that was quite easy (patch available at
http://anonscm.debian.org/viewvc/collab-qa/debcluster/configs/sbuild-twice.patch?view=log
for an ancient version of sbuild). Our amazon 'Debian QA' credits could be
used to perform such tests if someone is interested but doesn't have the
required rebuild infrastructure. Note that it should also be tested
whether the resulting packages are identical (FSVO identical).

Now, I'm not so sure that we should spend developer time to (1) find
those issues and file bugs; (2) fix those issues. I personally find it
easier to create a temporary git repository and to use git clean / git
checkout to return to a pristine environment.

Lucas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130504132...@xanadu.blop.info

[Xavier Roche]

Le 04/05/2013 15:37, Xavier Roche a écrit :
> something that you can not detect unless you setup a complete chrooted
> build environment, which is a bit cumbersome to do)

Replying to myself - I should have pointed out that pbuilder was
actually a really straightforward way to do that (sudo pbuilder --update
&& sudo pbuilder --build foo.dsc)

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/518520B0...@httrack.com

[Michael Gilbert]

On Sat, May 4, 2013 at 6:09 AM, Jakub Wilk wrote:
> We've always treated "FTBFS if built twice in a row" bugs as important:
> http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debi...@lists.debian.org;tag=qa-doublebuild

The real question is whether or not there is a consensus within the
project that severity should be serious (via a policy "must"), making
such issues release critical, which would be the only way to truly
"eliminate" the doesn't build twice problem.

Best wishes,
Mike

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CANTw=MMtX6VPiziRuva0XwyMshR...@mail.gmail.com

[Michael Gilbert]

On Sat, May 4, 2013 at 11:48 AM, Michael Gilbert wrote:
> On Sat, May 4, 2013 at 6:09 AM, Jakub Wilk wrote:
>> We've always treated "FTBFS if built twice in a row" bugs as important:
>> http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debi...@lists.debian.org;tag=qa-doublebuild
>
> The real question is whether or not there is a consensus within the
> project that severity should be serious (via a policy "must"), making
> such issues release critical, which would be the only way to truly
> "eliminate" the doesn't build twice problem.

And/or on the technical side, make the buildds always build twice.
Then, you'll get plenty of release-critical FTBFS bugs (from the
people that monitor that output) for your other archs when your
package doesn't build twice.

Best wishes,
Mike


--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CANTw=MN0h6gZ4vsc_vbfydRW5vS...@mail.gmail.com

[Andrey Rahmatullin]

On Sat, May 04, 2013 at 11:53:04AM -0400, Michael Gilbert wrote:
> >> We've always treated "FTBFS if built twice in a row" bugs as important:
> >> http://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debi...@lists.debian.org;tag=qa-doublebuild
> >
> > The real question is whether or not there is a consensus within the
> > project that severity should be serious (via a policy "must"), making
> > such issues release critical, which would be the only way to truly
> > "eliminate" the doesn't build twice problem.
>
> And/or on the technical side, make the buildds always build twice.

Sounds like a waste of resources.

--
WBR, wRAR

[Michael Gilbert]

Again, as Thijs argued somewhat eloquently already earlier in this
thread, computational time is not the scarce resource to worry about;
human time is.

The one thing Debian is comfortable about spending money on is
hardware, so if we expect to see double build times, then there should
be an associated doubling-down on buildd hardware.

Best wishes,
Mike


--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CANTw=MM3oJzqJCM11e5XKqF2rJA...@mail.gmail.com

[Andrey Rahmatullin]

On Sat, May 04, 2013 at 12:05:06PM -0400, Michael Gilbert wrote:
> Again, as Thijs argued somewhat eloquently already earlier in this
> thread, computational time is not the scarce resource to worry about;
> human time is.
>
> The one thing Debian is comfortable about spending money on is
> hardware, so if we expect to see double build times, then there should
> be an associated doubling-down on buildd hardware.

1-2 years ago I wondered why Debian doesn't do automatical weekly full
archive test rebuilds for at least amd64 and the answer was hardware.
Since then I assumed Debian doesn't have any freedom with machine
resources.

--
WBR, wRAR

[Michael Gilbert]

That is something like 2 orders of magnitude [0] more work (for
reference doubling is 1/5th of an order of magnitude), which I would
agree is intractable, although Lucas did find a way using Amazon's
services.

Best wishes,
Mike

[0] The buildds currently handle around 5-10 packages a day now, and
building the entire archive every two weeks would require handling
around 2678 packages/day (37493 packages in wheezy / 14 days), which
is well over 100 times more work (i.e. 2 orders of magnitude).

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/CANTw=MNYmpaRAb-r7Doe9sLWOQGEeN8Xet=PRdKNQdJwpFY=Y...@mail.gmail.com

[Tollef Fog Heen]

]] Michael Gilbert

> The one thing Debian is comfortable about spending money on is
> hardware, so if we expect to see double build times, then there should
> be an associated doubling-down on buildd hardware.

One thing is the cost to buy the hardware. Other costs (monetary or
otherwise) are getting people to admin them, finding hosting for them,
etc.

(As for having weekly rebuilds, we could probably do that, if somebody
volunteered to do the work involved. I haven't seen anybody request it
since I joined DSA slightly less than two years ago, though, so I assume
there's not a huge interest.)

--
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/87bo8qb...@xoog.err.no

[Andrei POPESCU]

On Sb, 04 mai 13, 00:30:00, Ben Hutchings wrote:
>
> I assume you're concerned that there may be undeclared build-
> conflicts. But testing in the maintainer's development system is not
> a particularly good way to find those. Testing in a maximal
> environment (everything with priority <= optional, minus declared
> Build-Conflicts) might be. I think someone has even tried that
> before.

http://lists.debian.org/debian-devel/2008/01/msg00869.html

Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser
Offtopic discussions among Debian users and developers:
http://lists.alioth.debian.org/mailman/listinfo/d-community-offtopic

[Ryan Kavanagh]

On Fri, May 03, 2013 at 09:18:36AM +0800, Chow Loong Jin wrote:
> While we're at it, can we also have source-only uploads? Uploading
> potentially huge binary packages that just go to /dev/null seems like
> a pointless waste of bandwidth to me, and the only for argument I've
> heard (which I don't buy) is "so that we know maintainers have
> test-built their packages."

I agree with the above. I might just be naive about the following, but
Debian Developers essentially have root access to any Debian box: all it
takes is one crafty postinst script and an upgrade. If Debian and its
users trust developers with that kind of responsibility, it should also
be able to trust developers to follow a basic guideline of "Please
test-build your package and check the resulting binary before doing a
source-only upload."

Ryan

[Wouter Verhelst]

On 04-05-13 17:53, Michael Gilbert wrote:
> And/or on the technical side, make the buildds always build twice.

Not Going To Happen[tm] on my buildd hosts.

--
This end should point toward the ground if you want to go to space.

If it starts pointing toward space you are having a bad problem and you
will not go to space today.

-- http://xkcd.com/1133/

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/51854C6...@debian.org

[Helmut Grohne]

On Fri, May 03, 2013 at 04:53:59PM +0800, Thomas Goirand wrote:
> I think there's a consensus, the problem is who's going
> to do the work for automating dropping of binaries and
> rebuild.

Not implying that I am the one doing this work, I would like to learn
more about what needs to be touched to achieve aspects of the following
features: (Again not implying that they are all desired.)
* Throwing away binaries and rebuilding everything.
* Permitting source-only uploads.
* arch:all binNMUs.

Pointers to relevant documentation appreciated. Specific questions:
* Which tools/infrastructure needs changes?
* What are the general changes needed?
* How can those changes be split in independent pieces and applied
without interfering badly?

Helmut

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130504193...@alf.mars

[Jakub Wilk]

* Ryan Kavanagh <r...@debian.org>, 2013-05-04, 13:48:

>If Debian and its users trust developers with that kind of
>responsibility, it should also be able to trust developers to follow a
>basic guideline of "Please test-build your package and check the
>resulting binary before doing a source-only upload."

No, that's not how trust works.

--
Jakub Wilk

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013050419...@jwilk.net

[Wookey]

+++ Julian Taylor [2013-05-04 11:48 +0200]:

> On 04.05.2013 11:10, Wookey wrote:
> >
> > I am huge fan of both building in clean environments _and_ being able
> > to build twice. I don't think there is any solution to this other than
> > testing it in an automated fashion. An sbuild or pbuilder option for
> > --build-twice would make testing a very simple matter.
> >
>
> pbuilder and cowbuilder already support the --twice option since a while.

OK. I didn't know that. Cheers.

Problem is I've moved to sbuild these days, but lucas's patch could
fairly easily be worked up into an sbuild --twice option too.

The harder question is how/when to do that QA. I resisted making the
suggestion of doing it by default on all builds as that seemed a step
too far, although I see someone else did :-). In fact, given the
significant overhead of build-dep installation, build-twice would
actually be quite a small overhead for many smaller packages (and only
needs to be done on one fast arch, not all of them). It would clearly
be annoying for builds that are large/slow anyway, which implies some
kind of exception list, which was kind of the point where I decided
this wasn't going to fly.

I guess if I (or anyone else) gets round to setting up cross-build QA
test builds sometime, then doing --build-twice alongside makes some
sense as it's in a similar class of 'would be nice/ought to work, but
not absolutely vital'. And lucas is quite right that it's probably not
the most important thing to spend time fixing.

Wookey
--
Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM
http://wookware.org/

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/2013050423...@stoneboat.aleph1.co.uk

[Roger Leigh]

On Sun, May 05, 2013 at 12:18:44AM +0100, Wookey wrote:
> +++ Julian Taylor [2013-05-04 11:48 +0200]:
> > On 04.05.2013 11:10, Wookey wrote:
> > >
> > > I am huge fan of both building in clean environments _and_ being able
> > > to build twice. I don't think there is any solution to this other than
> > > testing it in an automated fashion. An sbuild or pbuilder option for
> > > --build-twice would make testing a very simple matter.
> > >
> >
> > pbuilder and cowbuilder already support the --twice option since a while.
>
> OK. I didn't know that. Cheers.
>
> Problem is I've moved to sbuild these days, but lucas's patch could
> fairly easily be worked up into an sbuild --twice option too.

There's definitely an open bug for adding this, and I'll be happy
for it to be added. It shouldn't be too hard to implement, though
we would probably want to make it configurable whether the repeat
build failing should fail the build as a whole. We probably want
to do the repeat after we've copied the built files out of the
chroot. We could probably also compare the file paths between
the source and binary packages in the first and second builds;
comparing the content itself is probably not realistic.


Regards,
Roger

--
.''`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' schroot and sbuild http://alioth.debian.org/projects/buildd-tools
`- GPG Public Key F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130505001...@codelibre.net

[Adam Borowski]

On Sun, May 05, 2013 at 10:00:07AM +0900, Charles Plessy wrote:
> Anyway, given that our infrastructure builds binary packages from a fresh
> unpacked source package, I would prefer if we keep the compromise that
> imperfect "clean" targets are not release-critical problems.

Note that for a big majority of packages, double builds work just fine.
In fact, that's most of what "make" is for: coping with partially
complete trees.

It's only if you do something strange when extra handling is needed.
So breaking partial builds is not something nice, and certainly contrary
to user expectations.

--
ᛊᚨᚾᛁᛏᚣ᛫ᛁᛊ᛫ᚠᛟᚱ᛫ᚦᛖ᛫ᚹᛖᚨᚲ

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130505010...@angband.pl

[Charles Plessy]

Le Sat, May 04, 2013 at 03:25:34PM +0200, Lucas Nussbaum a écrit :
>
> Now, I'm not so sure that we should spend developer time to (1) find
> those issues and file bugs; (2) fix those issues. I personally find it
> easier to create a temporary git repository and to use git clean / git
> checkout to return to a pristine environment.

I totally agree. Time has passed since the last large-scale control for
double-buildability. In the meantime, I used Git increasingly, and now I
exclusively rely on it for cleaning the source packages that I maintain. I
think that it is the best technical solution, as it saves me from writing
fragile makefile instructions or carrying patches that Upstream is not always
intersted in.

If shortcomings of the "clean" target become release-critical, I will simply
start to build-depend on git. What blocked me to do to for the moment is that
I did not find an easy way to ignore the debian directory; I suppose that
wiping out changes to the files during a "debian/rules clean" would break the
principle of least surprise.

Anyway, given that our infrastructure builds binary packages from a fresh
unpacked source package, I would prefer if we keep the compromise that
imperfect "clean" targets are not release-critical problems.

Have a nice day,

--
Charles Plessy
Debian Med packaging team,
http://www.debian.org/devel/debian-med
Tsurumi, Kanagawa, Japan

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130505010...@falafel.plessy.net

[Lucas Nussbaum]

On 05/05/13 at 03:06 +0200, Adam Borowski wrote:
> On Sun, May 05, 2013 at 10:00:07AM +0900, Charles Plessy wrote:
> > Anyway, given that our infrastructure builds binary packages from a fresh
> > unpacked source package, I would prefer if we keep the compromise that
> > imperfect "clean" targets are not release-critical problems.
>
> Note that for a big majority of packages, double builds work just fine.

[citation needed] :)

> In fact, that's most of what "make" is for: coping with partially
> complete trees.

I think that it depends on what kind of double-builds you are
talking about. For binary-only builds, I agree with you. But if you try
to rebuild the source package after a binary build, I'm quite sure that
a large number of packages will fail to build.

Lucas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130505023...@xanadu.blop.info

[Lucas Nussbaum]

On 04/05/13 at 12:27 -0400, Michael Gilbert wrote:
> On Sat, May 4, 2013 at 12:14 PM, Andrey Rahmatullin wrote:
> > On Sat, May 04, 2013 at 12:05:06PM -0400, Michael Gilbert wrote:
> >> Again, as Thijs argued somewhat eloquently already earlier in this
> >> thread, computational time is not the scarce resource to worry about;
> >> human time is.
> >>
> >> The one thing Debian is comfortable about spending money on is
> >> hardware, so if we expect to see double build times, then there should
> >> be an associated doubling-down on buildd hardware.
> > 1-2 years ago I wondered why Debian doesn't do automatical weekly full
> > archive test rebuilds for at least amd64 and the answer was hardware.
> > Since then I assumed Debian doesn't have any freedom with machine
> > resources.
>
> That is something like 2 orders of magnitude [0] more work (for
> reference doubling is 1/5th of an order of magnitude), which I would
> agree is intractable, although Lucas did find a way using Amazon's
> services.

Clearly, the limiting factor here is manpower. It takes time to maintain
the necessary infrastructure, start the rebuilds, process build logs, turn
them into bug reports, and answer follow-up questions.
The credits we have on AWS are largely sufficient to do as many archive
rebuilds one can reasonably think of.
Btw, if someone wants to take care of that during the first year of the
jessie cycle... :)

Lucas

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130505022...@xanadu.blop.info

[Bob Proulx]

Not every architecture needs to build twice. If only the fastest
building architecture built twice it would catch almost all of the
problems with no change in resources elsewhere.

Bob

[Adam Borowski]

On Sun, May 05, 2013 at 04:30:05AM +0200, Lucas Nussbaum wrote:
> On 05/05/13 at 03:06 +0200, Adam Borowski wrote:
> > On Sun, May 05, 2013 at 10:00:07AM +0900, Charles Plessy wrote:
> > > Anyway, given that our infrastructure builds binary packages from a fresh
> > > unpacked source package, I would prefer if we keep the compromise that
> > > imperfect "clean" targets are not release-critical problems.
> >
> > Note that for a big majority of packages, double builds work just fine.
>
> [citation needed] :)
>
> > In fact, that's most of what "make" is for: coping with partially
> > complete trees.
>
> I think that it depends on what kind of double-builds you are
> talking about. For binary-only builds, I agree with you. But if you try
> to rebuild the source package after a binary build, I'm quite sure that
> a large number of packages will fail to build.

Yeah, I meant binary-only builds. That's where partial builds are important
-- only C/C++ code will be ccached.

I agree with Charles that for source builds, the "clean" target is often
duplicating work that could be better done by git. Yeah, 3.0 (git) would
solve this nicely.

--
ᛊᚨᚾᛁᛏᚣ᛫ᛁᛊ᛫ᚠᛟᚱ᛫ᚦᛖ᛫ᚹᛖᚨᚲ

--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org

Archive: http://lists.debian.org/20130505212...@angband.pl