Bug#1064453: fdupes: Mistakenly assumes duplicates to have changed on armhf due to conflicting off_t definition
Ivan Krylov
Version: 1:2.2.1-1
Severity: normal
Dear Maintainer,
A script I am running on an armhf machine produces database snapshots.
Frequently, the snapshot is identical to the previous one, so I would
like to remove those using fdupes -qdN /path/to/dumps/.
After upgrading from Stretch to Bookworm (hitting all intermediate
releases in process), I started receiving the following output:
[+] /path/to/dumps/dump.2024-01-17_04:15:01.sql.xz
[!] /path/to/dumps/dump.2024-01-24_04:15:01.sql.xz -- unable to delete file: File contents changed during processing!
[!] /path/to/dumps/dump.2024-01-31_04:15:01.sql.xz -- unable to delete file: File contents changed during processing!
[!] /path/to/dumps/dump.2024-02-07_04:15:01.sql.xz -- unable to delete file: File contents changed during processing!
[!] /path/to/dumps/dump.2024-02-21_04:15:01.sql.xz -- unable to delete file: File contents changed during processing!
[!] /path/to/dumps/dump.2024-02-22_12:47:16.sql.xz -- unable to delete file: File contents changed during processing!
Using GDB, I have traced this to the following situation: from fdupes.c, the
contents of dupelist look exactly right:
(gdb) frame 1
#1 0x004037e8 in deletefiles (files=0x42c7f8, prompt=0, tty=<optimized out>, logfile=<optimized out>) at fdupes.c:1010
1010 if (removeifnotchanged(dupelist[x], &errorstring) == 0) {
(gdb) p *dupelist[x]
$47 = {d_name = 0x42c4d8 "/path/to/dumps/dump.2024-01-24_04:15:01.sql.xz", size = 45664,
crcpartial = 0x423c68 "BD}\005\a\203\265H\004\204\206n\301T\343.15:0\031",
crcsignature = 0x423c98 "\232W\f\230B\2045\226\252\355\232Q\276\256\272ב\347U\342A\001", device = 2050, inode = 41419299, mtime = 1706058902,
ctime = 1706058902, hasdupes = 0, duplicates = 0x42c390, next = 0x42c410}
The moment the execution reaches the function removeifnotchanged(), the
contents of dupelist start looking corrupted:
(gdb) frame 0
#0 removeifnotchanged (file=file@entry=0x42c490, errorstring=errorstring@entry=0xbefff2d4) at removeifnotchanged.c:7
7 {
(gdb) p *file
$48 = {d_name = 0x42c4d8 "/path/to/dumps/dump.2024-01-24_04:15:01.sql.xz", size = 0, crcpartial = 0xb260 <error: Cannot access memory at address 0xb260>,
crcsignature = 0x0, device = 18643971999939688, inode = 2050, mtime = 0, ctime = 41419299, hasdupes = 0, duplicates = 0x65b06496, next = 0x65b06496}
The inode field now holds the value of the device field, and all the
pointers except the first one are plain wrong. This happens due to the
differences in the off_t type:
(gdb) frame 1
#1 0x004037e8 in deletefiles (files=0x42c7f8, prompt=0, tty=<optimized out>, logfile=<optimized out>) at fdupes.c:1010
1010 if (removeifnotchanged(dupelist[x], &errorstring) == 0) {
(gdb) ptype /o dupelist[x]
type = struct _file {
/* 0 | 4 */ char *d_name;
/* XXX 4-byte hole */
/* 8 | 8 */ off_t size;
/* 16 | 4 */ md5_byte_t *crcpartial;
/* 20 | 4 */ md5_byte_t *crcsignature;
/* 24 | 8 */ dev_t device;
/* 32 | 8 */ ino_t inode;
/* 40 | 4 */ time_t mtime;
/* 44 | 4 */ time_t ctime;
/* 48 | 4 */ int hasdupes;
/* 52 | 4 */ struct _file *duplicates;
/* 56 | 4 */ struct _file *next;
/* XXX 4-byte padding */
/* total size (bytes): 64 */
} *
(gdb) frame 0
#0 removeifnotchanged (file=file@entry=0x42c490, errorstring=errorstring@entry=0xbefff2d4) at removeifnotchanged.c:7
7 {
(gdb) ptype /o file
type = const struct _file {
/* 0 | 4 */ char *d_name;
/* 4 | 4 */ off_t size;
/* 8 | 4 */ md5_byte_t *crcpartial;
/* 12 | 4 */ md5_byte_t *crcsignature;
/* 16 | 8 */ dev_t device;
/* 24 | 4 */ ino_t inode;
/* 28 | 4 */ time_t mtime;
/* 32 | 4 */ time_t ctime;
/* 36 | 4 */ int hasdupes;
/* 40 | 4 */ struct _file *duplicates;
/* 44 | 4 */ struct _file *next;
/* total size (bytes): 48 */
} *
I think that fdupes.h must #include "config.h" in order to provide the
proper definition of off_t to every user of struct _file:
$ diff -u <(cpp removeifnotchanged.c) <((echo '#include "config.h"'; cat removeifnotchanged.c) | cpp) | grep off_t
-typedef __off_t off_t;
+typedef __off64_t off_t;
- __off_t st_size;
-extern int fseeko (FILE *__stream, __off_t __off, int __whence);
-extern __off_t ftello (FILE *__stream) ;
The same problem may be present on other 32-bit architectures.
-- System Information:
Debian Release: 12.5
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'oldoldstable'), (500, 'stable')
Architecture: armhf (armv7l)
Kernel: Linux 4.14.180+ (SMP w/8 CPU threads; PREEMPT)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: sysvinit (via /sbin/init)
Versions of packages fdupes depends on:
ii libc6 2.36-9+deb12u4
ii libncursesw6 6.4-4
ii libpcre2-32-0 10.42-1
ii libtinfo6 6.4-4
fdupes recommends no packages.
fdupes suggests no packages.
-- no debconf information
Ivan Krylov
thanks
The upstream fixed this by accident in either [1] (adding confing.h to
fdupes.h) or [2] (adding config.h to removeifnotchanged.c). The
attached patch fixes the problem on my armhf machine.
--
Best regards,
Ivan
[1]:
https://github.com/adrianlopezroche/fdupes/commit/ab5ef95e2b2633d0ca1a5ae0b8ac41abde160100#diff-6bb3626e773afb49e4d7267d0caa25a5d5b3f65d9705477654a5df6b95bcbe11
[2]:
https://github.com/adrianlopezroche/fdupes/commit/8b223d6a2037b6c28d4c00b60dbc9d359309c07f#diff-4d4b4474dd3708d45d2c9ac887f496a3b16342221db4d139a8110c452e771b2a