bitbake doesn't use cache

[Jörg Sommer]

Hello,

I'm seeing that bitbake parses the recipes every time, even if nothing
changed:

```
% ./kas-container shell
builder@807ddfdc0326:/build$ bitbake -e strace2
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |#########################################################################################################################| Time: 0:03:30
Parsing of 3222 .bb files complete (0 cached, 3222 parsed). 4786 targets, 556 skipped, 3 masked, 0 errors.
ERROR: Nothing PROVIDES 'strace2'. Close matches:
ltrace
strace
uftrace

Summary: There was 1 ERROR message, returning a non-zero exit code.
builder@807ddfdc0326:/build$ bitbake -e strace2
Loading cache: 100% |###########################################################################################################################| Time: 0:00:01
Loaded 4784 entries from dependency cache.
Parsing recipes: 100% |#########################################################################################################################| Time: 0:00:02
Parsing of 3222 .bb files complete (3220 cached, 2 parsed). 4786 targets, 556 skipped, 3 masked, 0 errors.
ERROR: Nothing PROVIDES 'strace2'. Close matches:
ltrace
strace
uftrace

Summary: There was 1 ERROR message, returning a non-zero exit code.
builder@807ddfdc0326:/build$
exit
2024-08-06 06:39:59 - ERROR - Shell returned non-zero exit status
2024-08-06 06:39:59 - ERROR - Command "/bin/bash" failed with error 1

% ./kas-container shell
builder@e56adc537dd4:/build$ bitbake -e strace2
Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |#########################################################################################################################| Time: 0:04:59
Parsing of 3222 .bb files complete (0 cached, 3222 parsed). 4786 targets, 556 skipped, 3 masked, 0 errors.
ERROR: Nothing PROVIDES 'strace2'. Close matches:
ltrace
strace
uftrace

Summary: There was 1 ERROR message, returning a non-zero exit code.
```

So there's something with the container setup that make bitbake does not use
the cache. Can anyone reproduce this?

Might it be that kas recreates the local.conf with a different inode and
bitbake looks for the inode? Would it be possible that kas does not override
the file if it would have the same content?


Regards Jörg

--
Navimatix GmbH T: 03641 - 327 99 0
Tatzendpromenade 2 F: 03641 - 526 306
07745 Jena www.navimatix.de

Geschäftsführer: Steffen Späthe, Jan Rommeley
Registergericht: Amtsgericht Jena, HRB 501480

[Jan Kiszka]

No, this generally works.

> Might it be that kas recreates the local.conf with a different inode and
> bitbake looks for the inode? Would it be possible that kas does not override
> the file if it would have the same content?
>

Can you set up a minimal reproducer for this? That may already rule out
causes or even point to the reason.

Jan

--
Siemens AG, Technology
Linux Expert Center

[Jörg Sommer]

Jan Kiszka schrieb am Di 06. Aug, 12:37 (GMT):

Here it is:

```
% cat conf.yaml
header:
version: 11

repos:
poky:
url: "https://git.yoctoproject.org/git/poky"
branch: kirkstone
commit: bba60774f74f8c05b8fdd609e8cdc40afd59c6a8
path: "layers-3rdparty/poky"

layers:
meta:
meta-poky:

% cat patch.yaml
header:
version: 11

repos:
poky:
patches:
libtirpc:
repo: poky
path: ../../libtirpc.patch

% cat libtirpc.patch
diff --git a/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb b/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb
index 6980135a92..aa526e9e1c 100644
--- a/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb
+++ b/meta/recipes-extended/libtirpc/libtirpc_1.3.2.bb
@@ -10,8 +10,8 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=f835cce8852481e4b2bbbdd23b5e47f3 \
PROVIDES = "virtual/librpc"

SRC_URI = "${SOURCEFORGE_MIRROR}/${BPN}/${BP}.tar.bz2 \
file://CVE-2021-46828.patch \
- "
+"
UPSTREAM_CHECK_URI = "https://sourceforge.net/projects/libtirpc/files/libtirpc/"
UPSTREAM_CHECK_REGEX = "(?P<pver>\d+(\.\d+)+)/"
SRC_URI[sha256sum] = "e24eb88b8ce7db3b7ca6eb80115dd1284abc5ec32a8deccfed2224fc2532b9fd"
```

Without the patch the cache gets used (every time »Loaded 1643 entries«):

```
% ./kas-container shell -c 'bitbake -e strace2; bitbake -e strace2' conf.yaml; ./kas-container shell -c 'bitbake -e strace2' conf.yaml
2024-08-13 11:32:41 - INFO - kas 4.4 started
2024-08-13 11:32:41 - INFO - Repository poky already contains bba60774f74f8c05b8fdd609e8cdc40afd59c6a8 as commit
2024-08-13 11:32:41 - INFO - Repository poky checked out to bba60774f74f8c05b8fdd609e8cdc40afd59c6a8
2024-08-13 11:32:41 - INFO - To start the default build, run: bitbake -c build core-image-minimal
WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |#####################################################################################################################################| Time: 0:00:00
Loaded 1643 entries from dependency cache.
Parsing recipes: 100% |###################################################################################################################################| Time: 0:00:00
Parsing of 882 .bb files complete (881 cached, 1 parsed). 1643 targets, 46 skipped, 0 masked, 0 errors.

ERROR: Nothing PROVIDES 'strace2'. Close matches:

strace

Summary: There was 1 WARNING message.

Summary: There was 1 ERROR message, returning a non-zero exit code.

WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |#####################################################################################################################################| Time: 0:00:00
Loaded 1643 entries from dependency cache.

ERROR: Nothing PROVIDES 'strace2'. Close matches:

strace

Summary: There was 1 WARNING message.

Summary: There was 1 ERROR message, returning a non-zero exit code.

2024-08-13 11:32:50 - ERROR - Shell returned non-zero exit status
2024-08-13 11:32:50 - ERROR - Command "/bin/bash -c 'bitbake -e strace2; bitbake -e strace2'" failed with error 1
2024-08-13 11:32:51 - INFO - kas 4.4 started
2024-08-13 11:32:51 - INFO - Repository poky already contains bba60774f74f8c05b8fdd609e8cdc40afd59c6a8 as commit
2024-08-13 11:32:52 - INFO - Repository poky checked out to bba60774f74f8c05b8fdd609e8cdc40afd59c6a8
2024-08-13 11:32:52 - INFO - To start the default build, run: bitbake -c build core-image-minimal
WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |#####################################################################################################################################| Time: 0:00:00
Loaded 1643 entries from dependency cache.

ERROR: Nothing PROVIDES 'strace2'. Close matches:

strace

Summary: There was 1 WARNING message.

Summary: There was 1 ERROR message, returning a non-zero exit code.

2024-08-13 11:32:56 - ERROR - Shell returned non-zero exit status
2024-08-13 11:32:56 - ERROR - Command "/bin/bash -c 'bitbake -e strace2'" failed with error 1
```

With the patch the cache is not used after startup (»Loaded 0 entries«):

```
% ./kas-container shell -c 'bitbake -e strace2; bitbake -e strace2' conf.yaml:patch.yaml; ./kas-container shell -c 'bitbake -e strace2' conf.yaml:patch.yaml
2024-08-13 11:34:19 - INFO - kas 4.4 started
2024-08-13 11:34:19 - INFO - Repository poky already contains bba60774f74f8c05b8fdd609e8cdc40afd59c6a8 as commit
2024-08-13 11:34:20 - INFO - Repository poky checked out to bba60774f74f8c05b8fdd609e8cdc40afd59c6a8
2024-08-13 11:34:20 - INFO - Patch applied. (patch path: /work/layers-3rdparty/poky/../../libtirpc.patch, repo: poky, patch entry: libtirpc)
2024-08-13 11:34:20 - INFO - To start the default build, run: bitbake -c build core-image-minimal
WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.

Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.

Parsing recipes: 100% |###################################################################################################################################| Time: 0:00:32
Parsing of 882 .bb files complete (0 cached, 882 parsed). 1643 targets, 46 skipped, 0 masked, 0 errors.

ERROR: Nothing PROVIDES 'strace2'. Close matches:

strace

Summary: There was 1 WARNING message.

Summary: There was 1 ERROR message, returning a non-zero exit code.

WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Loading cache: 100% |#####################################################################################################################################| Time: 0:00:00
Loaded 1643 entries from dependency cache.

ERROR: Nothing PROVIDES 'strace2'. Close matches:

strace

Summary: There was 1 WARNING message.

Summary: There was 1 ERROR message, returning a non-zero exit code.

2024-08-13 11:34:59 - ERROR - Shell returned non-zero exit status
2024-08-13 11:34:59 - ERROR - Command "/bin/bash -c 'bitbake -e strace2; bitbake -e strace2'" failed with error 1
2024-08-13 11:35:00 - INFO - kas 4.4 started
2024-08-13 11:35:01 - INFO - Repository poky already contains bba60774f74f8c05b8fdd609e8cdc40afd59c6a8 as commit
2024-08-13 11:35:01 - INFO - Repository poky checked out to bba60774f74f8c05b8fdd609e8cdc40afd59c6a8
2024-08-13 11:35:01 - INFO - Patch applied. (patch path: /work/layers-3rdparty/poky/../../libtirpc.patch, repo: poky, patch entry: libtirpc)
2024-08-13 11:35:01 - INFO - To start the default build, run: bitbake -c build core-image-minimal
WARNING: Host distribution "debian-12" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.

Loading cache: 100% | | ETA: --:--:--
Loaded 0 entries from dependency cache.

Parsing recipes: 100% |###################################################################################################################################| Time: 0:00:32
Parsing of 882 .bb files complete (0 cached, 882 parsed). 1643 targets, 46 skipped, 0 masked, 0 errors.

ERROR: Nothing PROVIDES 'strace2'. Close matches:

strace

Summary: There was 1 WARNING message.

Summary: There was 1 ERROR message, returning a non-zero exit code.

2024-08-13 11:35:36 - ERROR - Shell returned non-zero exit status
2024-08-13 11:35:36 - ERROR - Command "/bin/bash -c 'bitbake -e strace2'" failed with error 1
```

[Tobias Schaffner]

Hi,

I am able to reproduce this. Looks like bitbake takes the commit hash of
the final layer into account for the recipe cache.

Kas checks out the repository and applies the custom patches on every
build. The current implementation how the patch is applied leads to a
different commit hash of the final layer on every build.

I am able to fix this by creating a reproducible commit hash. As we
already have a fixed author name/email and a fixed commit message, only
the git commit date and author date have to be set to a fixed value.

Currently I am using "Th Jan 01 1970 00:00:00 +0000" for both, which
seems to be the earliest timestamp git accepts. But I am not sure if
choosing a fixed timestamp way in the past or future could have any
unexpected side effects. Maybe it would make more sense to take the
values used in the original patch.

best wishes!

[Jan Kiszka]

Ah, interesting finding! Things start to make sense.

> I am able to fix this by creating a reproducible commit hash. As we
> already have a fixed author name/email and a fixed commit message, only
> the git commit date and author date have to be set to a fixed value.
>
> Currently I am using "Th Jan 01 1970 00:00:00 +0000" for both, which
> seems to be the earliest timestamp git accepts. But I am not sure if
> choosing a fixed timestamp way in the past or future could have any
> unexpected side effects. Maybe it would make more sense to take the
> values used in the original patch.
>

Better use the date of the patch file. it should not change randomly.
And if the patch has an author date recorded, that would be best, of course.

[Jörg Sommer]

Jan Kiszka schrieb am Mo 19. Aug, 16:54 (GMT):

Git-rebase recreates the files with a rebase.

> And if the patch has an author date recorded, that would be best, of course.

Does kas use `git am` or `git apply`?

[Jan Kiszka]

The latter. Might be related to the desire to also support pure diffs,
not only full patches.