Hi cedric, jan
I will explain the issue in the details below:
To reproduce the issue ( Ref:
https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-of-local-apt-repo-caching-upstream-debian-packages ):
Trigger the warmup build to download deb and deb-src packages ( Enable the BASE_REPO_FEATURES = "cache-deb-src" in conf/local.conf)
e.x: bitbake mc:qemuamd64-bookworm:isar-image-base
Once all the deb and deb-src downloaded for the particular BSP under /download folder
Trigger the base-apt target to cache all the deb and deb-src (Enable ISAR_USE_CACHED_BASE_REPO = "1" in conf/local.conf)
e.x: bitbake base-apt
You will see the failures:
Logs:
NOTE: Resolving any missing task queue dependencies
Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% match, 0% complete)############################################# | ETA: 0:00:00
Initialising tasks: 100% |####################################################################################################################| Time: 0:00:00
NOTE: No setscene tasks
NOTE: Executing Tasks
ERROR: base-apt-1.0-r0 do_cache: One or more sources are missing in repo. bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb
bookworm bmap-tools 3.6-2 pool/main/b/bmap-tools/bmap-tools_3.6-2_all.deb
bookworm debconf 1.5.82 pool/main/d/debconf/debconf_1.5.82_all.deb
bookworm debian-archive-keyring 2023.3+deb12u1 pool/main/d/debian-archive-keyring/debian-archive-keyring_2023.3+deb12u1_all.deb
bookworm audit 1:3.0.9-1 pool/main/a/audit/libaudit-common_3.0.9-1_all.deb
bookworm libfile-find-rule-perl 0.34-3 pool/main/libf/libfile-find-rule-perl/libfile-find-rule-perl_0.34-3_all.deb
bookworm libnumber-compare-perl 0.03-3 pool/main/libn/libnumber-compare-perl/libnumber-compare-perl_0.03-3_all.deb
bookworm pam 1.5.2-6+deb12u1 pool/main/p/pam/libpam-runtime_1.5.2-6+deb12u1_all.deb
bookworm libsemanage 3.4-1 pool/main/libs/libsemanage/libsemanage-common_3.4-1_all.deb
bookworm libtext-glob-perl 0.11-3 pool/main/libt/libtext-glob-perl/libtext-glob-perl_0.11-3_all.deb
bookworm perl 5.36.0-7+deb12u1 pool/main/p/perl/perl-modules-5.36_5.36.0-7+deb12u1_all.deb
bookworm six 1.16.0-4 pool/main/s/six/python3-six_1.16.0-4_all.deb
bookworm tzdata 2024b-0+deb12u1 pool/main/t/tzdata/tzdata_2024b-0+deb12u1_all.deb
bookworm usrmerge 37~deb12u1 pool/main/u/usrmerge/usrmerge_37~deb12u1_all.deb
bookworm adduser 3.134 pool/main/a/adduser/adduser_3.134_all.deb
Just i captured few packages, but many of the deb packages don't have their source files
In the base-apt recipes we are calling the repo_sanity_test function finally to make sure all the deb's having
the corresponding deb-src packages should be present in the base-apt.
Yes, as you said, we are parsing deb files (from /var/cache/apt/archives) that were downloaded to determine which source packages should be downloaded.
but in the function "is_not_part_of_current_build "${package}" we are skipping the download-src if parsing deb package (status installed) is not available in the below files
```
"${IMAGE_ROOTFS}"/var/log/dpkg.log \
"${SCHROOT_HOST_DIR}"/var/log/dpkg.log \
"${SCHROOT_TARGET_DIR}"/var/log/dpkg.log \
"${SCHROOT_HOST_DIR}"/tmp/dpkg_common.log \
"${SCHROOT_TARGET_DIR}"/tmp/dpkg_common.log
```
As I mentioned above, the adduser package was installed in the stage of bootstrap, but that package's status was not captured in the bootstrap's /var/log/dpkg.log file.
because mmdebstrap deletes those files by default, hence we didn't capture those bootstrap package status.
As we know bootstrap rootfs is used for sbuildchroot creation, but here whatever packages are installed during sbuild-chroot creation,
${SCHROOT_TARGET_DIR}"/var/log/dpkg.log this file only contains the sbuildchroot related package status, but our expectation is it should have
bootstrap + sbuildchroot package status available in /var/log/dpkg.log file in sbuildchroot.
My first patch solves the above issue
with the introduction of "skip=cleanup/reproducible", the bootstrap rootfs generated with /var/log/dpkg.log files, which contain all the bootstrap package status
along with that if you use that boostrap for sbuildchroot creation , the sbuildchroot rootfs contains the existing bootstrap plus
additionally installed sbuildchroot related packages status is available in sbuldchroot /var/log/dpkg.log file
My second patch resolve the below issue:
During image creation (IMAGE_FSTYPES = "wic") few packages were installed in the session (e.g: gdisk, bmap-tools) but those packages list available
only in the sbuild environment, not outside, carry those packages status to tmp/dpkg_common.log in the rootfs to get the src packages for
imager_install related deb packages, presently the cache-deb-src function calls in the ROOTFS_POSTPROCESS_COMMAND but we need to wait till imager generation
That's why we pushed that deb-src-caching after image creation.
My third path resolve the image rootfs related deb-src missing issue:
By default we are removing the /var/log/dpkg.log file in the rootfs_postprocess_clean_log_files function, but this is needed when
We enable the BASE_REPO_FEATURES = "cache-deb-src", hence making this function as a configurable one rather than enabled by default.
to download IMAGE_ROOTFS related deb-src packages
I hope I explain the issue, presently am checking the reproducible part (for jan queries), will update that part very soon.
Many thanks,
Srinu
Sure
>
> Jan
>