bextract segfault on 22.1.6, 22.1.7

[Joshua Myles]

I'm trying to use bextract to pull a few files out of a Bareos volume that is no longer in the catalog. bls on this volume works fine, but bextract fails a bit into the second restored file.

[root@bareos-sd-pub-test ~]# bextract -i file.list -V pauling-AI-Consolidated-4324  /var/lib/bareos/storage /mnt/pauling_restore
bextract: stored/butil.cc:304-0 Using device: "/var/lib/bareos/storage" for reading.
13-May 13:27 bextract JobId 0: Ready to read from volume "pauling-AI-Consolidated-4324" on device "FileStorage" (/var/lib/bareos/storage).
bextract JobId 0: -rw-rw-r--   1 1016     1016             107 2023-07-20 10:36:55  /mnt/pauling_restore/data/redacted/CONTROL
bextract JobId 0: -rw-rw-r--   1 1016     1016        11789547 2023-07-20 10:36:55  /mnt/pauling_restore/data/redacted/beta
Segmentation fault (core dumped)
[root@bareos-sd-pub-test ~]#

dmesg says this:

[Tue May 13 13:31:03 2025] bextract[20209]: segfault at 0 ip 00007f29758528e6 sp 00007ffd49524e00 error 6 in libbareosfind.so.22.1.7[7f2975842000+15000]
[Tue May 13 13:31:03 2025] Code: 5b 5d 41 5c 41 5d 41 5e 41 5f c3 66 0f 1f 84 00 00 00 00 00 8b 53 0c 48 39 c2 75 99 f6 43 08 02 75 a1 f3 0f 1e fa 48 8b 43 18 <83> 00 01 b8 03 00 00 00 eb ae 48 8b 1b 48 8d 3d 36 24 00 00 e8 b1

Any thoughts here? Same on Bareos 22.1.6 and 22.1.7 subscription, RHEL 8 and 9. Not sure how to debug bextract specifically.

Josh

[Bruno Friedmann (bruno-at-bareos)]

Hi Joshua,

I'm a bit surprized by your result. I've just redone a test on EL9 with bareos 22.1.7 and it works fine

su bareos -s /usr/bin/bash -c "/usr/sbin/bextract -i /tmp/file.list -V Full-0001 /var/lib/bareos/storage /var/tmp/bareos_restore"

bextract: stored/butil.cc:304-0 Using device: "/var/lib/bareos/storage" for reading.

14-May 09:28 bextract JobId 0: Ready to read from volume "Full-0001" on device "FileStorage" (/var/lib/bareos/storage).
bextract JobId 0: -rw-r--r--   1 bareos   bareos         83452 2025-05-14 09:09:17  /var/tmp/bareos_restore/var/lib/bareos/bareos.sql
14-May 09:28 bextract JobId 0: End of Volume at file 0 on device "FileStorage" (/var/lib/bareos/storage), Volume "Full-0001"
14-May 09:28 bextract JobId 0: End of all volumes.
14-May 09:28 bextract JobId 0: Releasing device "FileStorage" (/var/lib/bareos/storage).
1 files restored.

So what might happen, be sure all the files you want to restore are exactly on this volume (-V can be a list of volumes), maybe you have a file that overlaps on a second volume ?

[Joshua Myles]

Here's another run inside gdb.

(gdb) run
Starting program: /usr/sbin/bextract -i file.list -V pauling-AI-Consolidated-4324 /var/lib/bareos/storage /mnt/pauling_restore
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

bextract: stored/butil.cc:304-0 Using device: "/var/lib/bareos/storage" for reading.

14-May 07:34 bextract JobId 0: Ready to read from volume "pauling-AI-Consolidated-4324" on device "FileStorage" (/var/lib/bareos/storage).

bextract JobId 0: -rw-rw-r--   1 1016     1016             107 2023-07-20 10:36:55  /mnt/pauling_restore/data/redacted/CONTROL
bextract JobId 0: -rw-rw-r--   1 1016     1016        11789547 2023-07-20 10:36:55  /mnt/pauling_restore/data/redacted/beta

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff79468e6 in ParseXattrStreams (jcr=0x665300, xattr_data=xattr_data@entry=0x640580 <xattr_data>,
    stream=stream@entry=1998, content=0x69cb30 "", content_length=65) at ../../../../core/src/findlib/xattr.cc:3463
3463    ../../../../core/src/findlib/xattr.cc: No such file or directory.
(gdb)

The file names were copied directly from bls output. The files in file.list appeared only on volume pauling-AI-Consolidated-4324.

Josh

[Joshua Myles]

Repeating this and trying to just extract all files in the volume was even less successful.

[root@bareos-sd-pub-test ~]# bextract -p -V pauling-AI-Consolidated-4324  /var/lib/bareos/storage /mnt/pauling_restore

bextract: stored/butil.cc:304-0 Using device: "/var/lib/bareos/storage" for reading.

14-May 07:55 bextract JobId 0: Ready to read from volume "pauling-AI-Consolidated-4324" on device "FileStorage" (/var/lib/bareos/storage).
bextract JobId 0: -rw-r--r--   1 1016     1016          290029 2023-09-20 22:21:36  /mnt/pauling_restore/data/redacted/hbond_R171A_sub_acceptor.dat
bextract JobId 0: drwxrwxr-x   4 1016     1016            4096 2023-09-20 22:23:53  *None*
bextract JobId 0: -rw-rw-r--   1 1016     1016         3941000 2023-10-22 17:54:32  /mnt/pauling_restore/data/redacted/EFE_small_fc.log

Segmentation fault (core dumped)
[root@bareos-sd-pub-test ~]#

It seems like the volume may be corrupt somehow, even though it lists fully with bls and was otherwise fully functional as far as we knew.

Josh

[Bruno Friedmann (bruno-at-bareos)]

The crash is more about xattrs do you know if your restore location is able to handle the same way the source was doing xattrs ?

What happen if you restore those 2 files locally first.

[Joshua Myles]

Ah...restoring locally appears to work. I was trying to restore to an NFS share.

Strangely, I'm doing a separate 24 TB restore (from the catalog) for the same client to an NFS share mounted on another client and it's working fine. Might bareos-fd be doing something differently than bextract when it comes to writing attributes?

Also, could there be better error handling for this by bextract?

Josh

On Wed, May 14, 2025 at 8:08 AM Bruno Friedmann (bruno-at-bareos) <bruno.f...@bareos.com> wrote:

Caution: The sender of this email could not be verified, which may indicate an impersonation attempt. Verify the email's authenticity with the sender using your organization's trusted contact list before replying or taking further action.

Secured by Check Point

--
You received this message because you are subscribed to a topic in the Google Groups "bareos-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/bareos-users/wgiQc12yZsI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to bareos-users...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/bareos-users/c7d2a0bd-fde8-411e-a469-502dae8d795cn%40googlegroups.com.

--

Joshua Myles

he/him

Senior Storage Administrator

Michigan Tech IT

We can help.

mtu.edu/it

906-369-1870

[Bruno Friedmann (bruno-at-bareos)]

Bextract being a kinda last resort tool, I don't think it will get a lot of effort to support all special case the normal FD support, especially version under maintenance only like 22.

We have some work in progress about xattr, like replacing error message by warning in future versions.

Usually a needed volume shouldn't get out of the catalog, the catalog can support several billions of record in file table with according configuration and hardware.

So there's normally no needs for purging those volumes too early.

Regards.

[Joshua Myles]

Thanks for the note on xattr work.

I do understand about ideally not needing volumes outside the catalog and this is definitely not a common case for us. In this situation, the client system had a damaged filesystem for longer than we realized, and we saw "ERR=No such file or directory" in job logs (common here for systems with computational jobs using temp files) when what was really happening was I/O errors (eventually we did see "ERR=Input/output error"). We didn't realize this fully until some of the volumes were pruned.

Either way, the expectation is for a Bareos volume to be usable if it still exists on disk. In this case it was and I just didn't understand how to use bextract correctly. Whew. :-D

Thanks for your help!

Josh