Bug#1037075: diffoscope: Get's killed trying to diff 2 large images (> 5GB)

Evangelos Ribeiro Tzaras

unread,

Jun 3, 2023, 8:21:27 AM6/3/23

to

Package: diffoscope
Version: 242
Severity: normal

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Dear Maintainer,

I was trying to see if Mobian images build for the Librem5 would be reproducible.
They were not.reproducible, so I wanted to run diffoscope on it, but sadly it get's killed:

> $ ls -l l5-phosh-{1,2}/mobian-librem5-phosh-20230603.img
- -rw-r--r-- 1 fortysixandtwo fortysixandtwo 6000000000 Jun 3 06:22 l5-phosh-1/mobian-librem5-phosh-20230603.img
- -rw-r--r-- 1 fortysixandtwo fortysixandtwo 6000000000 Jun 3 06:45 l5-phosh-2/mobian-librem5-phosh-20230603.img

> $ diffoscope --debug l5-phosh-{1,2}/mobian-librem5-phosh-20230603.img
2023-06-03 13:58:43 D: diffoscope.main: Starting diffoscope 242
2023-06-03 13:58:43 D: diffoscope.presenters.formats: Will generate the following presenter formats: text
2023-06-03 13:58:43 D: diffoscope.environ: Normalising locale, timezone, etc. PATH is /usr/lib/ccache:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/x86_64-linux-gnu
2023-06-03 11:58:43 D: diffoscope.main: Starting comparison
2023-06-03 11:58:43 D: diffoscope.comparators: Loaded 90 comparator classes
2023-06-03 11:58:43 D: diffoscope.comparators.utils.specialize: l5-phosh-1/mobian-librem5-phosh-20230603.img not identified by any comparator. Magic says: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 11718749 sectors, extended partition table (last)
2023-06-03 11:58:43 D: diffoscope.comparators.utils.specialize: l5-phosh-2/mobian-librem5-phosh-20230603.img not identified by any comparator. Magic says: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 11718749 sectors, extended partition table (last)
2023-06-03 11:58:43 D: diffoscope.comparators.utils.compare: Comparing l5-phosh-1/mobian-librem5-phosh-20230603.img (FilesystemFile) and l5-phosh-2/mobian-librem5-phosh-20230603.img (FilesystemFile)
2023-06-03 11:58:43 D: diffoscope.comparators.utils.specialize: l5-phosh-1/mobian-librem5-phosh-20230603.img not identified by any comparator. Magic says: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 11718749 sectors, extended partition table (last)
2023-06-03 11:58:43 D: diffoscope.comparators.utils.specialize: l5-phosh-2/mobian-librem5-phosh-20230603.img not identified by any comparator. Magic says: DOS/MBR boot sector; partition 1 : ID=0xee, start-CHS (0x0,0,2), end-CHS (0x3ff,255,63), startsector 1, 11718749 sectors, extended partition table (last)
2023-06-03 11:58:43 D: diffoscope.comparators.utils.file: has_same_content(l5-phosh-1/mobian-librem5-phosh-20230603.img, l5-phosh-2/mobian-librem5-phosh-20230603.img)
2023-06-03 11:58:43 D: diffoscope.comparators.utils.command: Executing xxd {}
[1] 21386 killed diffoscope --debug l5-phosh-{1,2}/mobian-librem5-phosh-20230603.img

I can upload the images, in case that helps (but will take some amount of time)

- -- System Information:
Debian Release: 12.0
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: arm64, i386

Kernel: Linux 6.1.0-9-amd64 (SMP w/32 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages diffoscope depends on:
ii diffoscope-minimal 242

Versions of packages diffoscope recommends:
ii aapt 1:10.0.0+r36-10
ii abootimg 0.6-1+b2
ii acl 2.3.1-3
ii androguard 3.4.0~a1-6
ii apksigcopier 1.1.1-1
ii apksigner 31.0.2-1
pn apktool <none>
pn binutils-multiarch <none>
ii bzip2 1.0.8-5+b1
ii caca-utils 0.99.beta20-3
ii colord 1.4.6-2.2
ii coreboot-utils 4.15~dfsg-3
ii db-util 5.3.2
ii default-jdk [java-sdk] 2:1.17-74
ii default-jdk-headless 2:1.17-74
ii device-tree-compiler 1.6.1-4+b1
ii dexdump 11.0.0+r48-5
ii docx2txt 1.4-5
ii e2fsprogs 1.47.0-2
ii enjarify 1:1.0.3-5
ii ffmpeg 7:5.1.3-1
ii fontforge-extras 1:20230101~dfsg-1
ii fonttools 4.38.0-1
ii fp-utils 3.2.2+dfsg-20
ii fp-utils-3.2.2 [fp-utils] 3.2.2+dfsg-20
ii genisoimage 9:1.1.11-3.4
ii gettext 0.21-12
ii ghc 9.0.2-4
ii ghostscript 10.0.0~dfsg-11
ii giflib-tools 5.2.1-2.5
ii gnumeric 1.12.55-1
ii gnupg 2.2.40-1.1
ii gnupg-utils 2.2.40-1.1
ii hdf5-tools 1.10.8+repack1-1
ii html2text 1.3.2a-28
ii imagemagick 8:6.9.11.60+dfsg-1.6
ii imagemagick-6.q16 [imagemagick] 8:6.9.11.60+dfsg-1.6
ii jsbeautifier 1.14.4-1
ii libarchive-tools 3.6.2-1
ii libxmlb-dev 0.3.10-2
pn llvm <none>
ii lz4 [liblz4-tool] 1.9.4-1
ii lzip 1.23-5
ii mono-utils 6.8.0.105+dfsg-3.3
ii ocaml-nox 4.13.1-4
ii odt2txt 0.5-7
ii oggvideotools 0.9.1-6
ii openjdk-11-jdk [java-sdk] 11.0.18+10-1
ii openjdk-17-jdk [java-sdk] 17.0.6+10-1
ii openssh-client 1:9.2p1-2
ii openssl 3.0.9-1
ii pgpdump 0.34-1
ii poppler-utils 22.12.0-2+b1
ii procyon-decompiler 0.6.0-1
ii python3-argcomplete 2.0.0-1
ii python3-binwalk 2.3.4+dfsg1-1
ii python3-debian 0.1.49
ii python3-defusedxml 0.7.1-2
ii python3-guestfs 1:1.48.6-2
ii python3-jsondiff 2.0.0-2
ii python3-pdfminer 20221105+dfsg-1
ii python3-progressbar 2.5-3
ii python3-pypdf 3.4.1-1
ii python3-pyxattr 0.8.1-1
ii python3-rpm 4.18.0+dfsg-1+b1
ii python3-tlsh 3.4.4+20151206-1.4+b4
ii r-base-core 4.3.0-1
ii radare2 5.5.0+dfsg-1+b1
ii rpm2cpio 4.18.0+dfsg-1+b1
ii sng 1.1.0-4
ii sqlite3 3.40.1-2
ii squashfs-tools 1:4.5.1-1
ii tcpdump 4.99.3-1
ii u-boot-tools 2023.01+dfsg-2
ii unzip 6.0-28
ii wabt 1.0.32-1
ii xmlbeans 4.0.0-2
ii xxd 2:9.0.1378-2
ii xz-utils 5.4.1-0.2
ii zip 3.0-13
ii zstd 1.5.4+dfsg2-5

Versions of packages diffoscope suggests:
ii libjs-jquery 3.6.1+dfsg+~3.5.14-1

- -- no debconf information

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEuThlVLfdJmvLjimpkPDJsYprShkFAmR7LSwACgkQkPDJsYpr
Shmffw/+Jg0aNlB6hosMqgumnEN6E0els3eXBkcN+/B5bXPe7rxHAhWkOwB+4exW
H5+XdSIlAug3xpZ69wx0pnwQqx2XlYf9amOrc4WENL2rE3GDft8VnF3XHUyYgsVR
Hf8g301yvm7syAj+7+g+SkBfFh+JmKLRHBVaLdFwdGyg8BJB0s2lzOWAbH9wktoa
2QLwCbwUq1RyGwUIlex8Q9Pe8BMx4FO/Sa7qI1w8ARLgnuqndSvbQmUqL/uxtSf/
rPj8K/eScNOV1SeW6VZCR8qvDszMvNUJ/tpIgH3iMxBsTiKl+gnHxelSJx8wgB0p
OO/23xN0DRedlRCjPBiHiOnRVDXDSXd2+VgKK49gD/Bmaw0KCZLg2iWN0jmWtzU3
f1sTnEsplIHlvC9oSa7REhdjnyp/ewkpVt9NSOhhKcBq84CH1PqlbAEj4tiCNrNd
nryiLicJglU8vaJNHPQz2Z2ty3I8X+lgFkTOBwp7S5ht002jLg7NpJiWZKzH+F3V
gesrNDjTQ9b0jcysKv9GkA1XlYCS4GSrfI/pknG1BlBSUrore4Eq5VSjo+FC7hkA
SvasKAdawF8cU7KhqIotAo9Ztbk15IlN9eakONTRQbLOj4C/BzaGesVrJqd0qmZS
JzAAvxHy0jcwnKUhFnhMSY1kXhuvR15n5eWl00toMbJ/RhrznfQ=
=F8xP
-----END PGP SIGNATURE-----

Evangelos Ribeiro Tzaras

unread,

Jun 3, 2023, 9:01:44 AM6/3/23

to

I forgot to add how to build the images:

On Sat, 03 Jun 2023 14:08:12 +0200 Evangelos Ribeiro Tzaras
<devrtz...@fortysixandtwo.eu> wrote:
> Package: diffoscope
> Version: 242
> Severity: normal
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA512
>
> Dear Maintainer,
>
> I was trying to see if Mobian images build for the Librem5 would be
reproducible.

To build the images grab the mobian-recipes [0] repo from salsa.
The recipes are used by debos to build images.

My invocation looked like:

./build.sh -t librem5

[0] https://salsa.debian.org/Mobian-team/mobian-recipes/

> They were not.reproducible, so I wanted to run diffoscope on it, but sadly it
get's killed:

[...]

--
Cheers,

Evangelos
PGP: B938 6554 B7DD 266B CB8E 29A9 90F0 C9B1 8A6B 4A19

signature.asc

Holger Levsen

unread,

Jun 4, 2023, 8:40:05 AM6/4/23

to

hi,

thanks Evangelos, for filing this bug and providing the images exposing it to
https://fortysixandtwo.eu/upload/mobian-librem5-phosh-20230603-{1,2}.img now.

--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

I miss the old days were billionaires’ vanity projects were to build 1000 public
libraries or giant music venues.

signature.asc

Holger Levsen

unread,

Jun 4, 2023, 9:51:54 AM6/4/23

to

On Sat, Jun 03, 2023 at 02:08:12PM +0200, Evangelos Ribeiro Tzaras wrote:
> [1] 21386 killed diffoscope --debug l5-phosh-{1,2}/mobian-librem5-phosh-20230603.img

fwiw, I can reproduce this bug on bullseye and unstable, with and without
--no-default-limits.

--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

»Sieh, dass du Mensch bleibst. Mensch sein ist von allem die Hauptsache.
Und das heißt fest und klar und heiter sein, ja heiter, trotz alledem.«
(Rosa Luxemburg)

signature.asc

Chris Lamb

unread,

Jun 5, 2023, 2:10:05 PM6/5/23

to

forwarded 1037075 https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/342
thanks

I've forwarded this "upstream" here:

https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/342

Regards,

--
,''`.
: :' : Chris Lamb
`. `'` la...@debian.org / chris-lamb.co.uk
`-