[PATCH] meta-isar: hello: Use <orig-version> in CHANGELOG_V

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Make use of the new placeholder to append "+isar" to the original
version. We cannot consolidate the recipes carrying the upstream
versions, though, as that information needs to be propagated into ${S}.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Depends on https://groups.google.com/d/msgid/isar-users/ebf1ab25-74ea-b3ba-4c48-2ecac873c6f7%40siemens.com

meta-isar/recipes-app/hello/hello.inc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta-isar/recipes-app/hello/hello.inc b/meta-isar/recipes-app/hello/hello.inc
index c5f322bb..d81085d8 100644
--- a/meta-isar/recipes-app/hello/hello.inc
+++ b/meta-isar/recipes-app/hello/hello.inc
@@ -11,7 +11,7 @@ inherit dpkg
SRC_URI = "apt://${PN}"

MAINTAINER = "isar-users <isar-...@googlegroups.com>"
-CHANGELOG_V = "${PV}-99+isar"
+CHANGELOG_V = "<orig-version>+isar"

do_prepare_build() {
deb_add_changelog
--
2.16.4

[Henning Schild]

I think with this we might be able to put everything into one .bb. At
the moment we have the inc plus two bbs that have PV in their names.

Taking PV from a bb filename is wrong when the inc uses <orig-version>.

Henning

[Jan Kiszka]

On 28.04.20 18:00, Henning Schild wrote:
> I think with this we might be able to put everything into one .bb. At
> the moment we have the inc plus two bbs that have PV in their names.
>
> Taking PV from a bb filename is wrong when the inc uses <orig-version>.
>

"We cannot consolidate the recipes carrying the upstream
versions, though, as that information needs to be propagated into
${S}."

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Henning Schild]

Am Tue, 28 Apr 2020 18:01:53 +0200
schrieb Jan Kiszka <jan.k...@siemens.com>:

> On 28.04.20 18:00, Henning Schild wrote:
> > I think with this we might be able to put everything into one .bb.
> > At the moment we have the inc plus two bbs that have PV in their
> > names.
> >
> > Taking PV from a bb filename is wrong when the inc uses
> > <orig-version>.
>
> "We cannot consolidate the recipes carrying the upstream
> versions, though, as that information needs to be propagated into
> ${S}."

I read that but did not fully understand it. If we can not do that or
fully do that, does that mean the tracking of "<orig-version>" can not
be fully implemented? Maybe we can fill the gaps or document them. To
me it sounds like a "half-feature" to introduce orig-version as
proposed.

Henning

[Jan Kiszka]

On 29.04.20 07:54, Henning Schild wrote:
> Am Tue, 28 Apr 2020 18:01:53 +0200
> schrieb Jan Kiszka <jan.k...@siemens.com>:
>
>> On 28.04.20 18:00, Henning Schild wrote:
>>> I think with this we might be able to put everything into one .bb.
>>> At the moment we have the inc plus two bbs that have PV in their
>>> names.
>>>
>>> Taking PV from a bb filename is wrong when the inc uses
>>> <orig-version>.
>>
>> "We cannot consolidate the recipes carrying the upstream
>> versions, though, as that information needs to be propagated into
>> ${S}."
>
> I read that but did not fully understand it. If we can not do that or
> fully do that, does that mean the tracking of "<orig-version>" can not
> be fully implemented? Maybe we can fill the gaps or document them. To
> me it sounds like a "half-feature" to introduce orig-version as
> proposed.

We have two problems to solve here:

- appending to the exact debian version our own suffix when patching it
- tracking the upstream version under which the source package is
extracted to so that ${S} is set correctly

The former is addressed by my patch to debianize, for the latter I'm
lacking a good idea. However, that is generally not a bit issue because
Debian does less frequently the update of the major version, often only
when switching its own major release. Also, patches you carry are more
likely to require adjustments anyway in that case.

The problem for setting S is not the version extraction like we do for
the changelog. It is that this has to happen rather early - but after
recipe parsing where it normally happens. It would require to rewrite
do_apt_fetch so that it identifies the path and sets S, somehow.

Jan

[Henning Schild]

Am Wed, 29 Apr 2020 08:10:07 +0200

But S can be anything, it is a temporary directory ... afaik.

We have S="${WORKDIR}/git", why not have
S="${WORKDIR}/${PN}-orig-version" ... really that string and no
substitution later on. Or even PN-42-nobody-cares

I might be wrong, got a little lost in S and D in isar and still have
those gentoo meanings in my head. But i guess it should be the same as
in gentoo.

Henning

[Henning Schild]

Am Wed, 29 Apr 2020 08:27:24 +0200
schrieb "[ext] Henning Schild" <henning...@siemens.com>:

gentoo uses PV=9999 for "tracking", the real number will come out of
the code you fetch .. i.e. with git. Some packages require more 9s, the
idea is to have the tracking recipe always be the most recent. And all
those 9s tell devs in the filename that it is the tracking one. In fact
it might even be the "same" recipe there.

https://github.com/gentoo/gentoo/blob/master/net-vpn/openconnect/openconnect-9999.ebuild

is basically the same as

https://github.com/gentoo/gentoo/blob/master/net-vpn/openconnect/openconnect-8.08.ebuild

So we want to use <orig-version> for when we fetch with apt:// but we
might want a PV-pattern that also covers git fetching or tarball
fetching.

I.e. in jailhouse the recipe could git fetch the tag when PV looks like
a release. And if PV looks like tracking it could take latest master or
next. Same recipe, just a different filename.

Whatever works there should be taken for the apt://-fetcher as well,
not different PV pattern depending on the fetcher.

Henning

[Henning Schild]

Hi,

here some notes on a PV pattern for "tracking"

- choose a pattern that bitbake can order in case of multiple recipes
for one package release < master < next
2.0 < 9999 < 99999, using strings instead of numbers might not allow
bitbake to order
- choose a pattern that people see as "tracking"
- that PV would only be used in bitbake, the DEBIAN_V would come from
the code and might be partially constructed with isar patchlevels
- look at how oe chooses its PV for "tracking", if it does not support
that, look at gentoo
- use the pattern across all recipes, no matter how they fetch

Henning

Am Wed, 29 Apr 2020 08:37:20 +0200

[Baurzhan Ismagulov]

On Wed, Apr 29, 2020 at 08:10:07AM +0200, Jan Kiszka wrote:
> We have two problems to solve here:
>
> - appending to the exact debian version our own suffix when patching it
> - tracking the upstream version under which the source package is
> extracted to so that ${S} is set correctly
>
> The former is addressed by my patch to debianize, for the latter I'm lacking
> a good idea. However, that is generally not a bit issue because Debian does
> less frequently the update of the major version, often only when switching
> its own major release. Also, patches you carry are more likely to require
> adjustments anyway in that case.
>
> The problem for setting S is not the version extraction like we do for the
> changelog. It is that this has to happen rather early - but after recipe
> parsing where it normally happens. It would require to rewrite do_apt_fetch
> so that it identifies the path and sets S, somehow.

This is a very typical Debian vs. bitbake problem which falls into the 20% that
bitbake doesn't solve nicely. Another example is static DEPENDS of bitbake vs.
Debian's dynamic Depends that become known only after fetching. Besides hacking
the relevant places like do_apt_fetch, these currently tend to require
non-intuitive, hard-to-debug stuff like S="${@...}" or anonymous functions. I
think this particular issue is a good minimal example how we could tackle such
stuff with bitbake. That said, it requires careful consideration beyond the
useful step that the patch addresses. So, I've applied this to next, thanks.

With kind regards,
Baurzhan.

[Jan Kiszka]

On 07.05.20 22:09, Baurzhan Ismagulov wrote:
> On Wed, Apr 29, 2020 at 08:10:07AM +0200, Jan Kiszka wrote:
>> We have two problems to solve here:
>>
>> - appending to the exact debian version our own suffix when patching it
>> - tracking the upstream version under which the source package is
>> extracted to so that ${S} is set correctly
>>
>> The former is addressed by my patch to debianize, for the latter I'm lacking
>> a good idea. However, that is generally not a bit issue because Debian does
>> less frequently the update of the major version, often only when switching
>> its own major release. Also, patches you carry are more likely to require
>> adjustments anyway in that case.
>>
>> The problem for setting S is not the version extraction like we do for the
>> changelog. It is that this has to happen rather early - but after recipe
>> parsing where it normally happens. It would require to rewrite do_apt_fetch
>> so that it identifies the path and sets S, somehow.
>
> This is a very typical Debian vs. bitbake problem which falls into the 20% that
> bitbake doesn't solve nicely. Another example is static DEPENDS of bitbake vs.
> Debian's dynamic Depends that become known only after fetching. Besides hacking
> the relevant places like do_apt_fetch, these currently tend to require
> non-intuitive, hard-to-debug stuff like S="${@...}" or anonymous functions. I

Even that Python inlining won't help because it runs too early, during
parsing, as well and we need that after some tasks have been completed,
namely the apt-src fetching and unpacking.

Jan

> think this particular issue is a good minimal example how we could tackle such
> stuff with bitbake. That said, it requires careful consideration beyond the
> useful step that the patch addresses. So, I've applied this to next, thanks.
>
> With kind regards,
> Baurzhan.
>

[Baurzhan Ismagulov]

On Fri, May 08, 2020 at 08:05:33AM +0200, Jan Kiszka wrote:
> > This is a very typical Debian vs. bitbake problem which falls into the 20% that
> > bitbake doesn't solve nicely. Another example is static DEPENDS of bitbake vs.
> > Debian's dynamic Depends that become known only after fetching. Besides hacking
> > the relevant places like do_apt_fetch, these currently tend to require
> > non-intuitive, hard-to-debug stuff like S="${@...}" or anonymous functions. I
>
> Even that Python inlining won't help because it runs too early, during
> parsing, as well and we need that after some tasks have been completed,
> namely the apt-src fetching and unpacking.

IIRC, we had a PoC that did early pre-fetching, but that goes then completely
outside of the bitbake pipeline, which is a hack that would eventually bite,
given enough packages to build. So our next step was to look at managing the
pipeline dynamically, which would be a major change.

With kind regards,
Baurzhan.

[Henning Schild]

Jan,

this is what i talked about earlier. Not sure about the details anymore
but never got a reply.

Henning

Am Wed, 29 Apr 2020 08:51:49 +0200

[Jan Kiszka]

On 05.06.20 19:46, Henning Schild wrote:
> Jan,
>
> this is what i talked about earlier. Not sure about the details anymore
> but never got a reply.
>

Yes, I think I forgot to write a reply here:

I think we are still (luckily) talking about two things here. One, the
templating of the changelog with some "original" or whatever version is
done and remains useful. The other, processing of arbitrarily versioned
debian source packages is still a separate todo.

What is missing to build a tracking deb package rebuilder is extraction
of the debian source package to a stable path so that S can refer to
that without knowing the version in advance (which would be impossible).
We basically need to detect what do_apt_fetch unpacked and rename that
to comply with S. Can we tell "apt-get source" which directory to create
(rather than <package-name>-<upstream-version>? Didn't find anything.

Jan

[Henning Schild]

Am Fri, 5 Jun 2020 19:55:50 +0200
schrieb Jan Kiszka <jan.k...@siemens.com>:

> On 05.06.20 19:46, Henning Schild wrote:
> > Jan,
> >
> > this is what i talked about earlier. Not sure about the details
> > anymore but never got a reply.
> >
>
> Yes, I think I forgot to write a reply here:
>
> I think we are still (luckily) talking about two things here. One, the
> templating of the changelog with some "original" or whatever version
> is done and remains useful. The other, processing of arbitrarily
> versioned debian source packages is still a separate todo.

I think it is very closely related. The version to use is always in
what you fetch, no matter if you track "apt://" or "git://". So you
only know it after unpack.

> What is missing to build a tracking deb package rebuilder is
> extraction of the debian source package to a stable path so that S
> can refer to that without knowing the version in advance (which would
> be impossible). We basically need to detect what do_apt_fetch
> unpacked and rename that to comply with S. Can we tell "apt-get
> source" which directory to create (rather than
> <package-name>-<upstream-version>? Didn't find anything.

Yes if you only use apt://PN you will have to discover what you got. In
fact i am afraid you might never get tracking without a
"do_cleanall_apt" inbetween consecutive runs. Because you will not
download again if you have something that looks like it matches.
Note that do_apt_fetch creates a directory for every URI to run in. So
you can use apt://PN and apt://PN=PV and will get subdirs "PN" and
"PN=PV" in. The first one is basically tracking. But in reality the
others can be partial tracking if you skip revisions, apt-get source
does only partially match PV.

Because of that you technically always have to extract the most recent
changelog entry from something that did contain a changelog ...

We will probably mix up hello-2.1_debian2 with hello-2.1_debian5~6 and
just call both hello-2.1-isar99 if we apt://hello=2.1

Henning

[Baurzhan Ismagulov]

On Fri, Jun 05, 2020 at 07:55:50PM +0200, Jan Kiszka wrote:
> Can we tell "apt-get source" which directory to create
> (rather than <package-name>-<upstream-version>? Didn't find anything.

I think it would unpack whatever has been packed. IIRC,
<package-name>-<upstream-version> is required by the Debian Policy, and build
tools would complain if it isn't the case. Possibly that is just a warning and
might be forceable.

If the goal is to encode the version into the package name, the usual Debian
way is doing that without the dash (libqtwebkit4, libqt5webkit5, ges1.0-tools).

With kind regards,
Baurzhan.

[Jan Kiszka]

The goal is to provide ${S} without knowing <upstream-version>. Or can
we adjust ${S} in do_apt_fetch so that all succeeding task will see that?

Jan