[PATCH 0/3] allow do_patch to use the "git" tool

[Henning Schild]

From: Henning Schild <henning...@siemens.com>

I am still not too happy with the patching when rebuilding debian
packages. This is a first step, getting "quilt" right remains hard
and will hopefully follow later.

p1 fixes a problem, and p2 uses the feature
p3 is optional because it breaks an API

Henning Schild (3):
dpkg: adjust task order to allow using "git" for patching
meta-isar: apply a "git" patch in cowsay
dpkg-gbp: default to "git" patching

RECIPE-API-CHANGELOG.md | 5 ++++
meta-isar/recipes-app/cowsay/cowsay_git.bb | 1 +
meta-isar/recipes-app/cowsay/files/isar.patch | 24 +++++++++++++++++++
meta/classes/dpkg-base.bbclass | 4 ++--
meta/classes/dpkg-gbp.bbclass | 2 ++
5 files changed, 34 insertions(+), 2 deletions(-)
create mode 100644 meta-isar/recipes-app/cowsay/files/isar.patch

--
2.26.2

[Henning Schild]

From: Henning Schild <henning...@siemens.com>

Doing that will make sure we cover "git" patching in CI, it also shows
how to use it.

Signed-off-by: Henning Schild <henning...@siemens.com>
---
meta-isar/recipes-app/cowsay/cowsay_git.bb | 2 ++
meta-isar/recipes-app/cowsay/files/isar.patch | 24 +++++++++++++++++++
2 files changed, 26 insertions(+)
create mode 100644 meta-isar/recipes-app/cowsay/files/isar.patch

diff --git a/meta-isar/recipes-app/cowsay/cowsay_git.bb b/meta-isar/recipes-app/cowsay/cowsay_git.bb
index 7aca1b4560..2e885a8aa3 100644
--- a/meta-isar/recipes-app/cowsay/cowsay_git.bb
+++ b/meta-isar/recipes-app/cowsay/cowsay_git.bb
@@ -5,5 +5,7 @@

inherit dpkg-gbp

+PATCHTOOL = "git"
SRC_URI = "git://salsa.debian.org/debian/cowsay.git;protocol=https"
+SRC_URI += "file://isar.patch"
SRCREV = "756f0c41fbf582093c0c1dff9ff77734716cb26f"
diff --git a/meta-isar/recipes-app/cowsay/files/isar.patch b/meta-isar/recipes-app/cowsay/files/isar.patch
new file mode 100644
index 0000000000..4333d4d635
--- /dev/null
+++ b/meta-isar/recipes-app/cowsay/files/isar.patch
@@ -0,0 +1,24 @@
+diff -Nru git.old/debian/patches/isar.patch git/debian/patches/isar.patch
+--- git.old/debian/patches/isar.patch 1970-01-01 01:00:00.000000000 +0100
++++ git/debian/patches/isar.patch 2021-01-21 14:21:26.439040449 +0100
+@@ -0,0 +1,12 @@
++Index: git/cows/elephant.cow
++===================================================================
++--- git.orig/cows/elephant.cow
+++++ git/cows/elephant.cow
++@@ -1,6 +1,7 @@
++ ##
++ ## An elephant out and about
++ ##
+++##
++ $the_cow = <<EOC;
++ $thoughts /\\ ___ /\\
++ $thoughts // \\/ \\/ \\\\
+diff -Nru git.old/debian/patches/series git/debian/patches/series
+--- git.old/debian/patches/series 2021-01-21 14:20:59.404038258 +0100
++++ git/debian/patches/series 2021-01-21 14:21:26.440040449 +0100
+@@ -18,3 +18,4 @@
+ utf8_width
+ moofasa-tabs
+ 03-ansi_code_width_color_widechar.patch
++isar.patch
--
2.26.2

[Henning Schild]

From: Henning Schild <henning...@siemens.com>

That class is about packages coming from git, so that is the natural
choice for the patching tool.

Signed-off-by: Henning Schild <henning...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 5 +++++
meta-isar/recipes-app/cowsay/cowsay_git.bb | 1 -
meta/classes/dpkg-gbp.bbclass | 2 ++
3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index cc45f3f66e..c4ec5894a8 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -282,3 +282,8 @@ like /boot/efi to avoid such issues.
### Deprecate BUILD_DEPENDS in u-boot-custom.inc

Use DEBIAN_BUILD_DEPENDS instead, to align with deb_debianize.
+
+### Default to PATCHTOOL ?= "git" for dpkg-gbp
+
+Migrate your patches so they can be applied with "git am", or
+"unset PATCHTOOL" to get old behaviour.
diff --git a/meta-isar/recipes-app/cowsay/cowsay_git.bb b/meta-isar/recipes-app/cowsay/cowsay_git.bb
index 2e885a8aa3..c3a1f4942a 100644
--- a/meta-isar/recipes-app/cowsay/cowsay_git.bb
+++ b/meta-isar/recipes-app/cowsay/cowsay_git.bb
@@ -5,7 +5,6 @@

inherit dpkg-gbp

-PATCHTOOL = "git"

SRC_URI = "git://salsa.debian.org/debian/cowsay.git;protocol=https"

SRC_URI += "file://isar.patch"
SRCREV = "756f0c41fbf582093c0c1dff9ff77734716cb26f"

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ba5c3ebb89..d956e8c3bc 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -7,6 +7,8 @@ inherit dpkg

S = "${WORKDIR}/git"

+PATCHTOOL ?= "git"
+
GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

--
2.26.2

[Henning Schild]

From: Henning Schild <henning...@siemens.com>

The adjust_git task breaks the git tree for use outside the buildchroot.
And our patch implementation supports using "git am" when setting
PATCHTOOL = "git". In order to actually apply git patches we have to
adjust after patching.

Signed-off-by: Henning Schild <henning...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 01c6eb658e..5c7bddc9ba 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -47,10 +47,10 @@ python do_adjust_git() {
bb.fatal(str(e))
}

-addtask adjust_git after do_unpack before do_patch
+addtask adjust_git before do_dpkg_build

inherit patch
-addtask patch after do_adjust_git before do_dpkg_build
+addtask patch before do_adjust_git

SRC_APT ?= ""

--
2.26.2

[Jan Kiszka]

On 22.01.21 09:42, [ext] Henning Schild wrote:
> From: Henning Schild <henning...@siemens.com>
>
> The adjust_git task breaks the git tree for use outside the buildchroot.
> And our patch implementation supports using "git am" when setting
> PATCHTOOL = "git". In order to actually apply git patches we have to
> adjust after patching.

That will likely not work as is when rerunning the job for a partial
rebuild, e.g. after adding a patch and not changing the git URL. We now
need a "restore_git" prior to patch, no?

Jan

>
> Signed-off-by: Henning Schild <henning...@siemens.com>
> ---
> meta/classes/dpkg-base.bbclass | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 01c6eb658e..5c7bddc9ba 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -47,10 +47,10 @@ python do_adjust_git() {
> bb.fatal(str(e))
> }
>
> -addtask adjust_git after do_unpack before do_patch
> +addtask adjust_git before do_dpkg_build
>
> inherit patch
> -addtask patch after do_adjust_git before do_dpkg_build
> +addtask patch before do_adjust_git
>
> SRC_APT ?= ""
>
>

--

Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[Jan Kiszka]

On 22.01.21 09:42, [ext] Henning Schild wrote:

Sounds reasonable to me.

Jan

[Henning Schild]

Am Fri, 22 Jan 2021 11:53:58 +0100
schrieb Jan Kiszka <jan.k...@siemens.com>:

> On 22.01.21 09:42, [ext] Henning Schild wrote:
> > From: Henning Schild <henning...@siemens.com>
> >
> > The adjust_git task breaks the git tree for use outside the
> > buildchroot. And our patch implementation supports using "git am"
> > when setting PATCHTOOL = "git". In order to actually apply git
> > patches we have to adjust after patching.
>
> That will likely not work as is when rerunning the job for a partial
> rebuild, e.g. after adding a patch and not changing the git URL. We
> now need a "restore_git" prior to patch, no?

I was also afraid of that. So i tried modifying a patch, which triggerd
a new fetch/unpack cycle and was fine. I guess adding a patch will do
the same.
Changing the patch function would become problematic, but that is a
task for isar developers.

So i am almost certain that will not cause fun on partial rebuilds.

Henning

[Anton Mikanovich]

22.01.2021 16:40, Henning Schild wrote:
> Am Fri, 22 Jan 2021 11:53:58 +0100
> schrieb Jan Kiszka <jan.k...@siemens.com>:
>
>> On 22.01.21 09:42, [ext] Henning Schild wrote:
>>> From: Henning Schild <henning...@siemens.com>
>>>
>>> The adjust_git task breaks the git tree for use outside the
>>> buildchroot. And our patch implementation supports using "git am"
>>> when setting PATCHTOOL = "git". In order to actually apply git
>>> patches we have to adjust after patching.
>> That will likely not work as is when rerunning the job for a partial
>> rebuild, e.g. after adding a patch and not changing the git URL. We
>> now need a "restore_git" prior to patch, no?
> I was also afraid of that. So i tried modifying a patch, which triggerd
> a new fetch/unpack cycle and was fine. I guess adding a patch will do
> the same.
> Changing the patch function would become problematic, but that is a
> task for isar developers.
>
> So i am almost certain that will not cause fun on partial rebuilds.
>
> Henning
>

If you've already tested partial rebuild and there is no issues can we
proceed with apply?

--
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[Henning Schild]

Am Mon, 8 Feb 2021 19:06:56 +0300
schrieb Anton Mikanovich <ami...@ilbers.de>:

> 22.01.2021 16:40, Henning Schild wrote:
> > Am Fri, 22 Jan 2021 11:53:58 +0100
> > schrieb Jan Kiszka <jan.k...@siemens.com>:
> >
> >> On 22.01.21 09:42, [ext] Henning Schild wrote:
> >>> From: Henning Schild <henning...@siemens.com>
> >>>
> >>> The adjust_git task breaks the git tree for use outside the
> >>> buildchroot. And our patch implementation supports using "git am"
> >>> when setting PATCHTOOL = "git". In order to actually apply git
> >>> patches we have to adjust after patching.
> >> That will likely not work as is when rerunning the job for a
> >> partial rebuild, e.g. after adding a patch and not changing the
> >> git URL. We now need a "restore_git" prior to patch, no?
> > I was also afraid of that. So i tried modifying a patch, which
> > triggerd a new fetch/unpack cycle and was fine. I guess adding a
> > patch will do the same.
> > Changing the patch function would become problematic, but that is a
> > task for isar developers.
> >
> > So i am almost certain that will not cause fun on partial rebuilds.
> >
> > Henning
> >
> If you've already tested partial rebuild and there is no issues can
> we proceed with apply?

I think so. In fact just backported this to a layer where at again
performed as expected.

Henning

[Anton Mikanovich]

22.01.2021 11:42, Henning Schild wrote:
> From: Henning Schild <henning...@siemens.com>
>
> I am still not too happy with the patching when rebuilding debian
> packages. This is a first step, getting "quilt" right remains hard
> and will hopefully follow later.
>
> p1 fixes a problem, and p2 uses the feature
> p3 is optional because it breaks an API

Applied to next, thanks.

[Jan Kiszka]

...but breaks rebuilding as long as we do not clone into the buildchroot
and rather patch alternates:

ERROR: mc:qemuarm64-buster:cowsay-git-r0 do_patch: Command Error: 'sh -c 'git --work-tree=/build/tmp/work/debian-buster-arm64/cowsay/git-r0/git reset --hard HEAD'' exited with 0 Output:
error: object directory /downloads/git/salsa.debian.org.debian.cowsay.git/objects does not exist; check .git/objects/info/alternates
error: unable to read sha1 file of cows/elephant-in-snake.cow (1003cdf63bd9ae64e6674b47f8ed4799b5f809a6)
error: unable to read sha1 file of cows/elephant.cow (959cd3f6dd2b628b2471572a06e162b31dbcac96)
error: unable to read sha1 file of cows/head-in.cow (2e51774886392c2d295e5659b03dbfa04f14aeb4)
error: unable to read sha1 file of cows/luke-koala.cow (78acd6988fa397cc1dfcd98f04373af94288430e)
error: unable to read sha1 file of cows/moofasa.cow (15119d6782fec20424c648fcf0174f291dd22026)
error: unable to read sha1 file of cowsay (900ca46014cf0454075a6390830b399b2a4b2a34)
error: unable to read sha1 file of cowsay.1 (7133866af2f7ea7cccdfa79378a61b27c1573e6b)
fatal: Could not reset index file to revision 'HEAD'.
ERROR: Logfile of failure stored in: /build/tmp/work/debian-buster-arm64/cowsay/git-r0/temp/log.do_patch.71
ERROR: Task (mc:qemuarm64-buster:/repo/meta-isar/recipes-app/cowsay/cowsay_git.bb:do_patch) failed with exit code '1'

Did this change have a practical motivation or was just out of esthetic
reasons? If it were only the latter, we should revert until our bitbake
can resolve the underlying problem.

[Henning Schild]

Am Wed, 11 Aug 2021 20:14:56 +0200
schrieb Jan Kiszka <jan.k...@siemens.com>:

It does have a practical motivation. Patching with quilt in isar has
nasty side-effects when debian/ is using quilt as well.
Using git as patchtool when git is also doing fetch/unpack seems like
the best solution to that. The patching with quilt gets especially
nasty if you can not add a quilt patch-patch to just deal with the code
... when you find yourself also wanting to patch debian/.

I would say that "adjust_git" is to blame here and we need to live with
that until we version-bump bitbake. In recent bitbake we will have
https://github.com/openembedded/bitbake/commit/6ae6f1865d5e666ebc670f70b7401a7b41648102
and will be able to ditch "adjust_git" by simply setting a global
BB_GIT_NOSHARED for isar.
That will also result in much improved usability ... since git repos
will end up usable insisde/outside chroots.

Henning

> Jan
>

[Henning Schild]

Am Wed, 11 Aug 2021 20:36:49 +0200
schrieb Henning Schild <henning...@siemens.com>:

Doing BB_GIT_NOSHARED=1 in general and dropping "adjust_git" is not
something i have tested. But i am almost sure it will work.
It will have an impact on performance and disk-usage.

If a layer wanted to early-adopt ... it would be cherry-picking that
bitbake commit into isar, setting that variable global and dropping
"adjust_git".

Henning

> Henning
>
> > Jan
> >
>

[Jan Kiszka]

That reasoning was missing from the commit log, and also the cover
letter declared this patch optional.

But you can patch debian packages via quilt, though not by do_patch
directly, see various examples in our layers. So the reasoning needs to
be a bit more precises in pointing out that also this approach did not
work (and ideally why).

[Jan Kiszka]

That's not a layer topic, that's an Isar topic if that feature is so
important. If the patch is upstream in master, pick it on top of our
release and do the refactorings. That's better then fighting adjust_git
by randomly breaking it.

[Henning Schild]

Am Wed, 11 Aug 2021 21:15:49 +0200

I did send a patch that will "unshare" in isar without having to wait
for a bitbake version bump or doing cherry-pick forking. Let us hope
that patch is accepted and we will simply carry adjust_git until that
bitbake bump.

Henning

> Jan
>

[Henning Schild]

Am Wed, 11 Aug 2021 21:12:35 +0200

There is long standing RFC mail from me that goes into detail on
patching. Long story short ... using quilt when debian uses quilt as
well is "problematic". And patching debian/ can not be done with
debian-quilt. This is where you end up with "git" as the best patchtool
in isar when doing dpkg-gbp.

And with salsa dpkg-gbp becomes increasingly interesting as
"new-normal".

Henning

> Jan
>