libpsl v0.16.0 released

[Tim Rühsen]

Hi,

there is a new release v0.16.0.

The new configure option --with-psl-distfile is reflected by psl_latest() and
psl_dist_filename() and should be used by packagers / distributions to point to
a regularly updated PSL DAFSA binary (see updated branch 'debian' (file debian/
rules) where we point to the DAFSA contained in package 'publicsuffix').

I will soon provide patches for wget and curl to benefit from the new API.

16.12.2016 Release V0.16.0
* Add functions psl_latest() and psl_dist_filename()
* Do not taint out variable on error in psl_str_to_utf8lower()
* Replace psl2c by psl-make-dafsa
* Add missing includes for OpenBSD
* Fix typos
* Update copyright year

Project Website
https://github.com/rockdaboot/libpsl

Online Documentation
https://rockdaboot.github.io/libpsl/

Getting the Source Code
git clone https://github.com/rockdaboot/libpsl

Source Code Test Coverage
https://coveralls.io/github/rockdaboot/libpsl?branch=master

The git 'master' branch will reflect the latest release.
The 'develop' branch will be used for future development.
The 'debian' branch will be used for unofficial Debian packaging.

[Daniel Kahn Gillmor]

Hi Tim--

On Sun 2016-12-11 19:48:03 +0100, Tim Rühsen wrote:
> there is a new release v0.16.0.
>
> The new configure option --with-psl-distfile is reflected by psl_latest() and
> psl_dist_filename() and should be used by packagers / distributions to point to
> a regularly updated PSL DAFSA binary (see updated branch 'debian' (file debian/
> rules) where we point to the DAFSA contained in package 'publicsuffix').

thanks for this! can i ask why you've moved from libpsl5 to libpsl6?

in configure.ac, i see this:

# 1. Start with version information of ‘0:0:0’ for each libtool library.
# 2. Update the version information only immediately before a public release of your software. More frequent updates are unnecessary, and only guarantee that the current interface number gets larger faster.
# 3. If the library source code has changed at all since the last update, then increment revision (‘c:r:a’ becomes ‘c:r+1:a’).
# 4. If any interfaces have been added, removed, or changed since the last update, increment current, and set revision to 0.
# 5. If any interfaces have been added since the last public release, then increment age.
# 6. If any existing interfaces have been removed or changed since the last public release, then set age to 0.
-AC_SUBST([LIBPSL_SO_VERSION], [5:2:0])
+AC_SUBST([LIBPSL_SO_VERSION], [6:0:0])


was any interface removed or changed since the last public release? If
not, then step 5 should have had you increment the "age" so this should
be 6:0:1 instead of 6:0:0.

--dkg

[Daniel Kahn Gillmor]

On Mon 2016-12-12 14:21:50 +0100, Daniel Kahn Gillmor wrote:
> thanks for this! can i ask why you've moved from libpsl5 to libpsl6?

just to clarify: the reason i'm asking is because we really want to keep
the SONAME the same if the only thing that happens is *adding*
interfaces. Keeping the SONAME the same means that code that doesn't
know about the new interfaces but just uses pre-existing ones can
continue to work with the new library, without recompilation.

it also means we don't need to wait for the NEW queue in debian, and we
don't need to trigger a recompilation of any upstream projects.

--dkg

[Tim Ruehsen]

Hi Daniel,

On Monday, December 12, 2016 2:21:50 PM CET Daniel Kahn Gillmor wrote:
> Hi Tim--
>
> On Sun 2016-12-11 19:48:03 +0100, Tim Rühsen wrote:
> > there is a new release v0.16.0.
> >
> > The new configure option --with-psl-distfile is reflected by psl_latest()
> > and psl_dist_filename() and should be used by packagers / distributions
> > to point to a regularly updated PSL DAFSA binary (see updated branch
> > 'debian' (file debian/ rules) where we point to the DAFSA contained in
> > package 'publicsuffix').
> thanks for this! can i ask why you've moved from libpsl5 to libpsl6?

One of the package building tools told me to do so... ;-)
See below.

> in configure.ac, i see this:
>
> # 1. Start with version information of ‘0:0:0’ for each libtool library.
> # 2. Update the version information only immediately before a public
> release of your software. More frequent updates are unnecessary, and only
> guarantee that the current interface number gets larger faster. # 3. If the
> library source code has changed at all since the last update, then
> increment revision (‘c:r:a’ becomes ‘c:r+1:a’). # 4. If any interfaces have
> been added, removed, or changed since the last update, increment current,
> and set revision to 0. # 5. If any interfaces have been added since the
> last public release, then increment age. # 6. If any existing interfaces
> have been removed or changed since the last public release, then set age to
> 0. -AC_SUBST([LIBPSL_SO_VERSION], [5:2:0])
> +AC_SUBST([LIBPSL_SO_VERSION], [6:0:0])

Those rules have no *if* in it and you go through them from 1 to 6 in that
order.
#3: Yes (r+1, but that is superseeded by the following)
#4: Yes added 2 functions. Yes, changed the behavior (the out tainting). So
c=6 and r=0.
#5: Yes, a=1.
#6: Yes, a=0

This results in 6.0.0.
If those rules have to be interpreted in other ways, the documentation has to
say so, IMO. I you think, I interpreted the docs in a wrong fashion, please
explain.

> was any interface removed or changed since the last public release? If
> not, then step 5 should have had you increment the "age" so this should
> be 6:0:1 instead of 6:0:0.

Not the syntax of the API, but the behavior changed slightly on error return.
It changed in a way that code that handles the old behavior still works, code
that doesn't was potentially unstable before and is now stable.

Regards, Tim

[Tim Ruehsen]

Just want to mention that I was unsure about this point, and to be on the save
side I decided for 6.0.0 instead of 6.0.1.

But anyways, having current at 6 means, the Debian build tool (not sure which
one exactly) warns about SO at 6 and package name at 5. Thus, I changed names
from libpsl5 -> libpsl6.

If you think we should leave it at libpsl5, I'll amend the debian branch to do
so.

Regards, Tim

[Tim Ruehsen]

One more question about Debian packaging.

The library and the tools are just compiled with '-g -Wdate-time -
D_FORTIFY_SOURCE=2' which is `dpkg-buildflags --get CPPFLAGS`.

I fiddled with debian/rules to get `dpkg-buildflags --get CFLAGS` as well (e.g.
-O2) but with no success. The docs say, I shouldn't care - it's done
automatically !? Maybe you could give me some advice.

Regards, Tim

[Daniel Kahn Gillmor]

On Mon 2016-12-12 15:03:04 +0100, Tim Ruehsen wrote:
> Those rules have no *if* in it and you go through them from 1 to 6 in that
> order.
> #3: Yes (r+1, but that is superseeded by the following)
> #4: Yes added 2 functions. Yes, changed the behavior (the out tainting). So
> c=6 and r=0.
> #5: Yes, a=1.
> #6: Yes, a=0

yes, i think you're approaching the rules the right way, but my question
was: why is #6 "yes" ?

> Not the syntax of the API, but the behavior changed slightly on error return.
> It changed in a way that code that handles the old behavior still works, code
> that doesn't was potentially unstable before and is now stable.

I see, so this is a subtle point, and might just be up to your
discretion as a maintainer.

The main goal of tracking API changes is to ensure that stable code that
depends on your code doesn't start failing as the result of an upgrade.

the downside of bumping the SONAME is that packages that are built
against a previous version can't simply work with the new version. They
have to be rebuilt against the new package. In practice, this is an
opportunity to do things like *remove* symbols -- then when packages try
to rebuild against the new package, if they don't depend on the removed
symbols, they rebuild fine; if they do depend on the removed symbols
then they get a clear compile-time warning that they need to update to
use the new version correctly. But if no symbols are removed, and there
is only subtle difference in behavior between versions, the most likely
outcome is that old code gets rebuilt against the new code, and since it
interacts with the same interfaces, it will likely build OK, while maybe
not acknowledging any subtle semantic or behavioral changes in the
underlying library :/

So it's not clear that an SONAME bump is actually helpful here, unless
there are additional compile-time hints that will help people avoid
instabilities that they otherwise did.

Given that the change you're describing isn't going to introduce new
instability (it sounds like it's actually stabilizing), and that
rebuilds aren't likely to do anything to catch any problems, i'd suggest
that the soname is better left at 5 (so c:r:a should be 6:0:1)

i know this stuff is frustrating, and thanks for bearing with the
process. i think it'll be better for debian as well to avoid what would
otherwise be called a library "transition" by the bump to libpsl6.
triggering a transition now might mean that it gets kept out of the
stretch freeze, too, i haven't tried one quite this late in the freeze
process. So i'm inclined to just patch configure.ac to 6:0:1 rather
than convert to libpsl6, but it's your call. what do you want to do?

--dkg

[Daniel Kahn Gillmor]

On Mon 2016-12-12 17:42:43 +0100, Daniel Kahn Gillmor wrote:
> On Mon 2016-12-12 15:03:04 +0100, Tim Ruehsen wrote:
>> Not the syntax of the API, but the behavior changed slightly on error return.
>> It changed in a way that code that handles the old behavior still works, code
>> that doesn't was potentially unstable before and is now stable.

Another way of looking at this is: if there was a simple bugfix release,
that would certainly technically change the behavior -- some things were
buggy, but now they are not buggy :) But no one would suggest that a
simple bugfix release should require an SONAME bump.

--dkg

[Tim Rühsen]

Thanks for your explanations.
It sounds reasonable to stay with soname psl5 (and patch library version to
6:0:1) - this is what I have done now in branch 'debian' (needs git fetch -f).

Now 'lintian -i' shows
E: libpsl5: symbols-file-contains-current-version-with-debian-revision on
symbol psl_builtin@Base and 20 others
N:
N: Debian revisions should be stripped from versions in symbols files. Not
N: doing so leads to dependencies unsatisfiable by backports (1.0-1~bpo <<
N: 1.0-1 while 1.0-1~bpo >= 1.0). If the Debian revision can't be stripped
N: because the symbol really appeared between two specific Debian
N: revisions, you should postfix the version with a single "~" (example:
N: 1.0-3~ if the symbol appeared in 1.0-3).
N:
N: This problem normally means that the symbols were added automatically by
N: dpkg-gensymbols. dpkg-gensymbols uses the full version number for the
N: dependency associated to any new symbol that it detects. The maintainer
N: must update the debian/<package>.symbols file by adding the new symbols
N: with the corresponding upstream version.
N:
N: Severity: important, Certainty: certain
N:
N: Check: shared-libs, Type: binary, udeb

I am unsure how to proceed... adding "~" or stripping version numbers.
The above description just means nothing to me... if version numbers are
(sometimes) a problem, why carry them around at all ? Interestingly, the
message suggest to strip the 'debian' version numbers - but I can't see any in
debian/libpsl5.symbols !?

Regards, Tim

[Daniel Kahn Gillmor]

Hey Tim--

On Mon 2016-12-12 21:23:25 +0100, Tim Rühsen wrote:
> Thanks for your explanations.
> It sounds reasonable to stay with soname psl5 (and patch library version to
> 6:0:1) - this is what I have done now in branch 'debian' (needs git fetch -f).

cool, thanks for the work!

> Now 'lintian -i' shows
> E: libpsl5: symbols-file-contains-current-version-with-debian-revision on
> symbol psl_builtin@Base and 20 others
> N:
> N: Debian revisions should be stripped from versions in symbols files. Not
> N: doing so leads to dependencies unsatisfiable by backports (1.0-1~bpo <<
> N: 1.0-1 while 1.0-1~bpo >= 1.0). If the Debian revision can't be stripped
> N: because the symbol really appeared between two specific Debian
> N: revisions, you should postfix the version with a single "~" (example:
> N: 1.0-3~ if the symbol appeared in 1.0-3).
> N:
> N: This problem normally means that the symbols were added automatically by
> N: dpkg-gensymbols. dpkg-gensymbols uses the full version number for the
> N: dependency associated to any new symbol that it detects. The maintainer
> N: must update the debian/<package>.symbols file by adding the new symbols
> N: with the corresponding upstream version.
> N:
> N: Severity: important, Certainty: certain
> N:
> N: Check: shared-libs, Type: binary, udeb

i suspect that the problem is that you've left libpsl.so.6 in the
debian/libpsl5.symbols file instead of reverting it to libpsl.so.5.

Also, because debian wants to see the patches applied to the upstream
source in debian/patches/, the fix to the SONAME probably needs to be
formatted slightly differently, rather than being a direct and simple
patch in the git history (i use "gbp pq" from the git-buildpackage
toolchain to manage such a patch queue).

I've just pushed 7534f8de572c555d980d12d5db20c24bd14ed44a to the debian
branch at https://github.com/dkg/libpsl as a proposal for resolving it.
can you take a look at that and tell me what you think?

--dkg

[Tim Ruehsen]

Hi Daniel,

On Tuesday, December 13, 2016 2:15:07 AM CET Daniel Kahn Gillmor wrote:
> > Now 'lintian -i' shows
> > E: libpsl5: symbols-file-contains-current-version-with-debian-revision on
> > symbol psl_builtin@Base and 20 others
> > N:
> > N: Debian revisions should be stripped from versions in symbols files.
> > Not N: doing so leads to dependencies unsatisfiable by backports
> > (1.0-1~bpo << N: 1.0-1 while 1.0-1~bpo >= 1.0). If the Debian revision
> > can't be stripped N: because the symbol really appeared between two
> > specific Debian N: revisions, you should postfix the version with a
> > single "~" (example: N: 1.0-3~ if the symbol appeared in 1.0-3).
> > N:
> > N: This problem normally means that the symbols were added
> > automatically by N: dpkg-gensymbols. dpkg-gensymbols uses the full
> > version number for the N: dependency associated to any new symbol that
> > it detects. The maintainer N: must update the debian/<package>.symbols
> > file by adding the new symbols N: with the corresponding upstream
> > version.
> > N:
> > N: Severity: important, Certainty: certain
> > N:
> > N: Check: shared-libs, Type: binary, udeb
>
> i suspect that the problem is that you've left libpsl.so.6 in the
> debian/libpsl5.symbols file instead of reverting it to libpsl.so.5.

I just learned something :-)
I was 100% convinced that SO_VERSION 6:0:1 builds a libpsl.so.6 (as 6:0:0
does), thus I kept that in the symbols file on purpose. I am surprised that it
builds to a libpsl.5, which I just tested it does.
It still doesn't explain the misleading messages from lintian, though. But
hey, I take that as 'given'.

> Also, because debian wants to see the patches applied to the upstream
> source in debian/patches/, the fix to the SONAME probably needs to be
> formatted slightly differently, rather than being a direct and simple
> patch in the git history (i use "gbp pq" from the git-buildpackage
> toolchain to manage such a patch queue).

Good to know. I thought about how to patch and then decided for the 'direct
and simple' way. IMO, gbp could be improved in this point.

> I've just pushed 7534f8de572c555d980d12d5db20c24bd14ed44a to the debian
> branch at https://github.com/dkg/libpsl as a proposal for resolving it.
> can you take a look at that and tell me what you think?

Thanks, I synced my branch with yours.

There are just a few questions left...

### What does that ./configure warning mean ?
configure: WARNING: unrecognized options: --disable-maintainer-mode, --enable-
ld-version-script, --with-packager, --with-packager-bug-reports, --with-
packager-version

And how to fix that ?

### How to add -O2 to compilation (text from a previos email)

The library and the tools are just compiled with '-g -Wdate-time -
D_FORTIFY_SOURCE=2' which is `dpkg-buildflags --get CPPFLAGS`.

I fiddled with debian/rules to get `dpkg-buildflags --get CFLAGS` as well (e.g.
-O2) but with no success. The docs say, I shouldn't care - it's done
automatically !? Maybe you could give me some advice.

### Would love to see man pages in libpsl-dev package
To build them, we need --enable-man for dh_auto_configure and a libpsl-
dev.manpages

### How to test building with local commits
I need a quick package build to try my local changes...
The only command that I found that builds packages without error is 'dpkg-
buildpackage -B'. But it looks like the patches are not applied - it uses
6:0:0 and thus creates libpsl.so.6.
Using gbp results in random errors...
I spent hours reading (outdated ?) docs, with no success.
Please help.

Regards, Tim

[Daniel Stenberg]

On Tue, 13 Dec 2016, Tim Ruehsen wrote:

> I just learned something :-) I was 100% convinced that SO_VERSION 6:0:1
> builds a libpsl.so.6 (as 6:0:0 does), thus I kept that in the symbols file
> on purpose. I am surprised that it builds to a libpsl.5, which I just tested
> it does.

Just a little note on this: it will build a libpsl.5 on *most* systems but it
will actually create a libpsl.6 on some (ancient) systems. Which I suppose is
why libtool has the magic made up like that...

--

/ daniel.haxx.se

[Daniel Kahn Gillmor]

On Tue 2016-12-13 10:59:27 +0100, Tim Ruehsen wrote:
> I just learned something :-)
> I was 100% convinced that SO_VERSION 6:0:1 builds a libpsl.so.6 (as 6:0:0
> does), thus I kept that in the symbols file on purpose. I am surprised that it
> builds to a libpsl.5, which I just tested it does.
> It still doesn't explain the misleading messages from lintian, though. But
> hey, I take that as 'given'.

i suspect you missed the very first message from dpkg-gensymbols and
friends, before lintian runs, which was probably less misleading and
would have explained it :) i don't have a copy handy, but i think it
says something about libpsl.so.5 not being present in the .symbols file.

> Good to know. I thought about how to patch and then decided for the 'direct
> and simple' way. IMO, gbp could be improved in this point.

I've thought about this a lot, and the gbp upstream is pretty open to
suggestions, but i'm actually not convinced that there's a better way to
do it. there are certainly *other* ways to do it, but they each have
their own failure modes in different ways (e.g. the git-dpm workflow).

> Thanks, I synced my branch with yours.

cool, and i see you want to ship a libpsl.3 manpage, which is great. :)

> ### What does that ./configure warning mean ?
> configure: WARNING: unrecognized options: --disable-maintainer-mode, --enable-
> ld-version-script, --with-packager, --with-packager-bug-reports, --with-
> packager-version
>
> And how to fix that ?

So these args:

--enable-ld-version-script
--with-packager
--with-packager-bug-reports
--with-packager-version

are all from your git commit 518ae87f, and we've been carrying them
since then -- if you don't want them, we should drop them from
debian/rules :)

--disable-maintainer-mode is a common ./configure option because some
packages build differently in maintainer mode. It's added by
dh_auto_configure, and should be harmless if you have nothing that is
maintainer-mode specific in your configure.ac.

> ### How to add -O2 to compilation (text from a previos email)
> The library and the tools are just compiled with '-g -Wdate-time -
> D_FORTIFY_SOURCE=2' which is `dpkg-buildflags --get CPPFLAGS`.
>
> I fiddled with debian/rules to get `dpkg-buildflags --get CFLAGS` as well (e.g.
> -O2) but with no success. The docs say, I shouldn't care - it's done
> automatically !? Maybe you could give me some advice.

The cflags should be added automatically by the standard debhelper
invocations, without needing to do anything else.

also, if you like, you could update to debhelper 10 by updating
debian/compat and changing the build-dependency on debhelper to have
a better version: "(>= 10~)" (the ~ means that backported versions of
debhelper will be ok)

> ### Would love to see man pages in libpsl-dev package
> To build them, we need --enable-man for dh_auto_configure and a libpsl-
> dev.manpages

i see you've done this already, which is great!

> ### How to test building with local commits
> I need a quick package build to try my local changes...
> The only command that I found that builds packages without error is 'dpkg-
> buildpackage -B'. But it looks like the patches are not applied - it uses
> 6:0:0 and thus creates libpsl.so.6.
> Using gbp results in random errors...
> I spent hours reading (outdated ?) docs, with no success.
> Please help.

I'm confused, because it works for me. can you show me a full build log
from a given git commit? pastebin or private email is fine if you don't
want to flood the list :)

--dkg

[Tim Ruehsen]

On Tuesday, December 13, 2016 2:33:54 PM CET Daniel Kahn Gillmor wrote:
> On Tue 2016-12-13 10:59:27 +0100, Tim Ruehsen wrote:
> > ### How to add -O2 to compilation (text from a previos email)
> > The library and the tools are just compiled with '-g -Wdate-time -
> > D_FORTIFY_SOURCE=2' which is `dpkg-buildflags --get CPPFLAGS`.
> >
> > I fiddled with debian/rules to get `dpkg-buildflags --get CFLAGS` as well
> > (e.g. -O2) but with no success. The docs say, I shouldn't care - it's
> > done automatically !? Maybe you could give me some advice.
>
> The cflags should be added automatically by the standard debhelper
> invocations, without needing to do anything else.

No optimization automatically added, though
$ dpkg-buildflags --get CFLAGS
-g -O2 -fdebug-prefix-map=/usr/oms/src/libpsl=. -fstack-protector-strong -
Wformat -Werror=format-security

But when building with gdp, I see no -O2 anywhere.

> also, if you like, you could update to debhelper 10 by updating
> debian/compat and changing the build-dependency on debhelper to have
> a better version: "(>= 10~)" (the ~ means that backported versions of
> debhelper will be ok)

Did that, but couldn't test yet. No time left over now...

But if that is ok to for you, please upload the packages.

Regards, Tim

[Tim Ruehsen]

Just added a script debian/build which recovers the directory structure so
that gbp works smoothly. It basically does

rm -rf list
git reset --hard
git clean -d -x -f
gbp buildpackage

It removes all files that are not part of the branch, so use with care !

Regards, Tim