Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Bug#1007002: lintian: transition to "pointed hints" has invalidated many overrides

68 views
Skip to first unread message

Simon McVittie

unread,
Mar 10, 2022, 7:00:04 AM3/10/22
to
Package: lintian
Version: 2.114.0
Severity: important

There seems to be an ongoing transition in Lintian, in which hints that
report a filename or line in an ad-hoc way are being converted to "pointed
hints" that use a new format with the filename in square brackets.

One of the design decisions that keeps Lintian's value high as a
QA tool is that all of its hints (formerly known as tags) can be
overridden if the maintainer has assessed the hint and determined that
it is a false-positive or otherwise inapplicable to this particular
package. Unfortunately, changing a hint to a pointed hint invalidates
most overrides for that hint (unless they use a very broad wildcard match,
which seems like a bad idea since it could hide unrelated instances of the
hint that genuinely indicate a problem). I can see why it is considered
valuable to have a consistent format for all hints, but mass-invalidating
existing overrides seems like a high price to pay for that.

This is particularly frustrating when the overrides that would be required
to silence a hint on lintian.debian.org and the overrides that would
be required to silence a hint with the released version of Lintian are
mutually incompatible. https://lintian.debian.org/tags/mismatched-override
currently reports 15K mismatched overrides among 3K source packages,
which seems like a lot.

It would be useful if Lintian could treat previously-valid overrides as
still being a valid way to override the new form of a tag, particularly
in the common case where "foo usr/share/bar" has been replaced by
"foo [usr/share/bar]".

smcv

Simon McVittie

unread,
Jun 13, 2022, 4:30:03 AM6/13/22
to
Control: unblock 1006348 by -1

On Mon, 13 Jun 2022 at 05:15:19 +0200, Axel Beckert wrote:
> And #1007002 is only (!) about design decision to change nearly all
> tag formats involving file paths and line numbers. It has nothing to
> do with the other two real bugs and I have no idea why they have been
> merged.

I agree with your reasoning for unmerging, and I don't understand why
they were merged either.

#1007002 was marked as blocking #1006348 "lintian: Tag
improbable-bug-number-in-closes condition considers 7-digit bug numbers
improbable", but I think that was a side-effect of the merge and I don't
consider #1007002 to be RC or a blocker for #1006348, so I'm removing
that metadata.

> Since at least I will not revert such huge changes, I'll tag #1007002
> as "wontfix" for now and downgrade it to its original severity.
>
> We can continue working on that bug report if we find someone who
> either will work on reverting all the related work (although I think,
> it's both, too late and also probably way too much work given all the
> other recent changes) or, probably much better, provide either a
> migration script or some code which also accepts the old override
> formats. In that case, I'd remove the "wontfix" again. (Sorry, Simon,
> I generally agree with #1007002, but we currently have way more severe
> issues with Lintian than #1007002. Hence I also didn't remove the
> confirmed tag.)

That makes sense to me. I should have reported #1007002 earlier (or,
ideally, a Lintian co-maintainer should have pointed out the problems with
a major tag format transition before it got that far), which would have
made pausing or reverting the transition lower-cost.

Getting a new Lintian release into testing and backports would at least
mitigate #1007002 by making sure that maintainers can write one tag format
that is equally valid for stable-backports Lintian, testing Lintian,
unstable Lintian and lintian.debian.org.

Thank you for picking up this package! It's an important QA tool and
I'm glad someone is looking after it again.

smcv

Axel Beckert

unread,
Jun 13, 2022, 8:10:04 AM6/13/22
to
Hi Simon,

Simon McVittie wrote:
> Control: unblock 1006348 by -1

Thanks!

> #1007002 was marked as blocking #1006348 "lintian: Tag
> improbable-bug-number-in-closes condition considers 7-digit bug numbers
> improbable", but I think that was a side-effect of the merge

Definitely. All three unmerged bug reports had that block.

> and I don't consider #1007002 to be RC or a blocker for #1006348, so
> I'm removing that metadata.

Actually it was still on my TODO list to find out which of the bug
reports was related to #1006348 and why. You reduced that part of my
TODO list to 2/3 by no more having to check this bug report. Thanks!
:-)

> > Since at least I will not revert such huge changes, I'll tag #1007002
> > as "wontfix" for now and downgrade it to its original severity.
> >
> > We can continue working on that bug report if we find someone who
> > either will work on reverting all the related work (although I think,
> > it's both, too late and also probably way too much work given all the
> > other recent changes) or, probably much better, provide either a
> > migration script or some code which also accepts the old override
> > formats. In that case, I'd remove the "wontfix" again. (Sorry, Simon,
> > I generally agree with #1007002, but we currently have way more severe
> > issues with Lintian than #1007002. Hence I also didn't remove the
> > confirmed tag.)

Oh, JFTR: This refers to Lintian as it was in Git when I took over.
There are parts of that tag format transition already committed in Git
which aren't yet in Debian Unstable. :-/ There were nearly 200 commits
in git since 2.114.0 before I started working again on Lintian. I
won't triage and revert them either unless they are outright buggy in
a technical sense. Sorry. As mentioned it would need much more man
power to "fix" this issue, even with what is so far only in git.

And with Felix and Chris left the Lintian development, there's much
less man power so far. At least so far I'm the only one who did
commits since then, but there were at least three DDs asking for group
membership on Salsa (Thanks!), so I hope they will start working on
Lintian as well. I'm also updating the development how-tos where I
notice that they're outdated.

> That makes sense to me. I should have reported #1007002 earlier

No offence meant.

> (or, ideally, a Lintian co-maintainer should have pointed out the
> problems with a major tag format transition before it got that far),

That's more a thing. Actually I already thought if we should setup a
similar rule for Lintian tag syntaxes as we did for using epoch in
package versions, maybe just not as stringent. I'm though not sure
where such a rule should be placed. The Debian Policy seems the wrong
place. So that rule should be probably documented inside Lintian
itself or so.

> which would have made pausing or reverting the transition
> lower-cost.

Ack.

> Thank you for picking up this package! It's an important QA tool and
> I'm glad someone is looking after it again.

Should have noticed the resignation of Felix and Chris (or looked for
such a thing after noticing that there were no lintian uploads for a
while) earlier, too. Probably would have been less (or at least more
distributed) work to get it back in shape as the remainder of Debian
Unstable didn't stand still while Lintian did.

Especially I found one case where the amount of lines replacing the
#DEBHELPER# token caused test suite failures after a debhelper update
seem to have added some lines. Took a few hours to understand what
went wrong and why. (Granted, because many things I knew about
Lintian's test suite had changed since I last used it, I first had to
relearn how to use the current setup and this took probably some of
that time as well.)

Regards, Axel
--
,''`. | Axel Beckert <a...@debian.org>, https://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
`- | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE

Axel Beckert

unread,
Jun 29, 2022, 10:00:04 AM6/29/22
to
Hi Andreas,

Andreas Tille wrote:
> I realised that lintian (at least) starting with version 2.115.1 (may be
> earlier) wraps file names into [] which breaks existing
> lintian-overrides.

Correct, except that it happened for quite a while (7 months at least)
and was (and maybe still is — see below) a continuous transition. It
is present since at least 2.114.0 from November 2021. According to the
git history, the implementation started shortly before the 2.114.0
upload, but the bug report which requested this is actually from 2014:
https://bugs.debian.org/743226

And yes, 2.115.0 (but not 2.115.1 or 2.115.2 from today) had more such
changes as there were over 200 commits from Felix included which he
did after 2.114.0, but which weren't uploaded since then. Many of them
were (IMHO irresponsibly) marked "Gbp-Dch: Ignore" and hence didn't
show up in debian/changelog when I generated it with "gbp dch". (I
though ignored that in some cases and added them manually to the
debian/changelog entry, partially even retroactively.)

> I consider these [] not helpful […] no visible advantage.

The advantage is to clearly mark what is a file with potentially a
line number in the output of lintian so that further processors like
the lintian website can do deep links to the proper code position on
e.g. sources.debian.org or salsa.debian.org. Felix called it "pointed
hints".

From my point of view, that's quite an advantage.

See also https://bugs.debian.org/1007002 for the proper bug report
about the issue with invalidating overrides by this transition. I've
added it to the Cc of this thread and dropped lintia...@debian.org
instead as mails to that bug report get forwarded to the Lintian Team
anyway.

> since it breaks lots of lintian-overrides

Felix decided that it's worth to implement this requested feature and
I at least partially agree as I do clearly see the advantage and also
like the possibilities which this now offers.

> Could this change in lintian please be reverted?

I clearly won't do that — for multiple reasons:

* Far too much work for the gain (IMHO)
* Losing a requested feature.
* Invalidating 7 months of already updated overrides.
* Better ways to invest your (and my) time. (See my proposition
below.)

Background:

This seems not to have been one big commit but many small commits
whenever a tag was touched by Felix. Reverting it to the old format
would be a huge effort for which I don't have the time as there are
way more pressing issues in lintian. And I'm very sure it can't be
done by just doing a bunch of "git revert". So far I found 15 commits
by Felix mentioning "pointed hint" reaching from November 2021 to
March 2022. With about 200 other, partially quite invasive commits
inbetween.

And in addition to that, it's IMHO _far_ too late, because many
maintainers already have updated their overrides in the past 7 months.

And I'm currently not sure if I would even accept a Merge Request
which tries to do that.

But I doubt that anyone would a) make that effort and b) make all
those maintainers angry who already updated their overrides in the
past 7 months or more.

Roberto C. Sánchez wrote:
> Or perhaps make the new format default

That was Felix' plan, yes. I just have currently no idea how many tags
have been and how many haven't been converted yet.

If there aren't too many tags left, I might finish this transition.

If there are many tags left, I'd only do that if we have a way to cope
with it with much less effort than it so far caused.

And from my point of view, the only way to get out of this mess
without causing too much work for anyone (maintainers of packages with
affected overrides as well as the current Lintian maintainers):

Someone should write a converter from the old to the new format. That
doesn't sound too difficult. Main work would be gathering for each tag
involved how it looked beforehand and how it looked now. This probably
can be gathered from changes in git to Lintian's test suite.

And maybe it should even try to output overrides which are compatible
with the old and new format, at least in cases where the order of
parameters didn't change. See lintian's own overrides for some examples:
https://salsa.debian.org/lintian/lintian/-/blob/master/debian/source/lintian-overrides
(Although this also accounts for a bug which only ever was present in
Lintian's git repository and never got uploaded, but still got an RC
bug report because people started using lintian off the git repo due
to it no having been maintained for months:
https://bugs.debian.org/1003353)
signature.asc

Samuel Thibault

unread,
Jun 29, 2022, 10:20:04 AM6/29/22
to
Hello,

Axel Beckert, le mer. 29 juin 2022 15:49:11 +0200, a ecrit:
> > I consider these [] not helpful […] no visible advantage.
>
> The advantage is to clearly mark what is a file with potentially a
> line number in the output of lintian so that further processors like
> the lintian website can do deep links to the proper code position on
> e.g. sources.debian.org or salsa.debian.org. Felix called it "pointed
> hints".
>
> From my point of view, that's quite an advantage.

Yes, I fully agree. The format migration poses problem (I had to patch
quite a few lintian files), but it solves a lot other current-and-future
problems: we can now just ignore whole sets of warnings for a given file
(typically one that is generated by some tool such as doxygen)

Samuel

Andreas Tille

unread,
Jun 29, 2022, 10:50:03 AM6/29/22
to
Hi Samuel and Axel,

Am Wed, Jun 29, 2022 at 04:10:49PM +0200 schrieb Samuel Thibault:
> Axel Beckert, le mer. 29 juin 2022 15:49:11 +0200, a ecrit:
> > > I consider these [] not helpful […] no visible advantage.

OK, thanks for making the advantages visible to me then. I'll
start editing lintian-overrides then.

Kind regards

Andreas.

--
http://fam-tille.de

Lucas Nussbaum

unread,
Jun 29, 2022, 11:30:04 AM6/29/22
to
On 29/06/22 at 15:49 +0200, Axel Beckert wrote:
> Correct, except that it happened for quite a while (7 months at least)
> and was (and maybe still is — see below) a continuous transition. It
> is present since at least 2.114.0 from November 2021. According to the
> git history, the implementation started shortly before the 2.114.0
> upload, but the bug report which requested this is actually from 2014:
> https://bugs.debian.org/743226
>
> And yes, 2.115.0 (but not 2.115.1 or 2.115.2 from today) had more such
> changes as there were over 200 commits from Felix included which he
> did after 2.114.0, but which weren't uploaded since then.

Hi Axel,

Just a note that since the last version of lintian to migrate to testing
was 2.111 (which was also the last one to be backported to stable), some
of us might not have updated since 2.111 and might be hit by changes
that happened since then.

(I'm not arguing whether it should be kept or reverted, but I'm just
mentioning this as other disruptive changes might be discovered in the
coming days)

Lucas
signature.asc

Axel Beckert

unread,
Jun 29, 2022, 12:10:04 PM6/29/22
to
Hi Lucas,

Lucas Nussbaum wrote:
> Just a note that since the last version of lintian to migrate to testing
> was 2.111 (which was also the last one to be backported to stable), some
> of us might not have updated since 2.111 and might be hit by changes
> that happened since then.

Oh, right! Thanks for that view on Andreas' mail.

I indeed did not think about that case because I do nearly all Debian
development work on Unstable (and occassionally on Stable when a new
package starts as a quick and dirty hack to scratch my own itches at
work :-).

And until my reply to Andreas I also wasn't aware of when Felix
actually started with this. Just knew that it wasn't a one-shot thing,
but quite distributed over at least the past half year (because that's
the timeframe of git commits I most often looked into in the past few
weeks).

Andreas: Sorry if I was a bit too opposing because of assuming that
every DD uses Unstable by default. (Actually I think the Dev Ref says
you should, though. ;-) Or if DDs pinned Lintian to 2.111.0 from
Testing…

Andreas Tille wrote:
> I'll start editing lintian-overrides then.

Maybe wait a bit with that. Given Lucas' comment, I feel a bit more
urged to provide such a migration script.

I will look into this for the next upload. No promises as of now,
though.

Axel Beckert

unread,
Jun 29, 2022, 1:50:03 PM6/29/22
to
Hi again,

Axel Beckert wrote:
> Andreas Tille wrote:
> > I'll start editing lintian-overrides then.
>
> Maybe wait a bit with that. Given Lucas' comment, I feel a bit more
> urged to provide such a migration script.
>
> I will look into this for the next upload. No promises as of now,
> though.

A first prototype is now available in git in the branch
"migrate-overrides":

https://salsa.debian.org/lintian/lintian/-/blob/migrate-overrides/bin/lintian-migrate-overrides-to-pointed-hints

So far the script only knows about the tags spelling-error-in-binary
and package-contains-documentation-outside-usr-share-doc, but it is
explicitly written to be expandable. Of course it will also get
support for more tags in the (near) future.

Additionally it currently only prints the transformed result to
STDOUT. The plan is to also support inline editing, either with an -i
option or maybe even by default if it detects that the package is
maintained in git.

Please give me feedback if this approach (especially after inline
editing is supported) is feasible — preferably from those who are
annoyed. :-)

It's not yet in the master branch as neither Perl::Critic nor myself
are happy about the usage of (the expression form of) "eval" in there.
(Maybe one of the other JAPH has an idea on this. :-)

Patches and other improvements suggestions as well as pattern sets for
further tags are of course welcome as well.

(Note: I will probably force-push that feature branch over and over
again until I'm satisfied. If someone else wants to work on the same
branch, too, we can also work without forced pushes and squash-merge
the result at the end. Please contact me, if you're interested.)

Andreas Tille

unread,
Jun 29, 2022, 3:10:04 PM6/29/22
to
Hi Axel,

Am Wed, Jun 29, 2022 at 06:01:33PM +0200 schrieb Axel Beckert:
> Andreas: Sorry if I was a bit too opposing because of assuming that
> every DD uses Unstable by default.

No need to be sorry, really not. I simply missed the point of that
change.

> (Actually I think the Dev Ref says
> you should, though. ;-) Or if DDs pinned Lintian to 2.111.0 from
> Testing…

I'm using testing but pinned lintian on unstable - so at least in my
case it was not what Lucas pointed out. It was just that I beared the
change for some time ... and than my amount of patience spilled over.
;-P

But its correct that some users might use lintian from testing anyway.

Soren Stoutner

unread,
Jan 18, 2023, 3:43:02 PM1/18/23
to

It would probably be helpful to update the documentation at https://lintian.debian.org/manual/index.html#format-of-override-files to describe the new pointed hints format.


--

Soren Stoutner

so...@stoutner.com

signature.asc

Axel Beckert

unread,
Jan 18, 2023, 5:00:03 PM1/18/23
to
Control: clone -1 -2
Control: retitle -2 lintian: Update Lintian User's Manual wrt. to Pointed Hints format
Control: tags -2 - wontfix + confirmed

Hi Soren,

Soren Stoutner wrote:
> It would probably be helpful to update the documentation at
> https://lintian.debian.org/manual/index.html#format-of-override-files[1] to describe the new pointed hints format.

Indeed. We though have the issue that lintian.debian.org no more
updates. It's future is unclear, unfortunately. I do not have access
there. :-/

And it's also not yet my focus currently. After the freeeze maybe.

But I'll at least try to update doc/lintian.rst so that at least the
content is ready once we can update the website again.

This is also related to https://bugs.debian.org/1004234 (docs: give
advice how to debug overrides).

Soren Stoutner

unread,
Jan 18, 2023, 6:10:04 PM1/18/23
to
Thank you.

On Wednesday, January 18, 2023 2:47:01 PM MST Axel Beckert wrote:
> But I'll at least try to update doc/lintian.rst so that at least the
> content is ready once we can update the website again.

--
Soren Stoutner
so...@stoutner.com
signature.asc
0 new messages