Metadata server refuses to start with "Failed to store disposal directory"

[Christian]

I have inherited a BeeGFS 2015.03-r27 filesystem without metadata redundancy - only one metadata server. After a filesystem error and an xfs_repair, the metadata service now refuses to start, the only error message being "Failed to store disposal directory. Unable to proceed."

I cannot find any information about this error message in the whole internet (the only Google result is the source code itself), so my stupid question is: What does this error message mean, and is there any chance for me to get the metadata filesystem back to work?

Any help would be greatly appreciated, since I fear that I will have to restore everything from backups...

[Philipp Falk]

Hello Christian,

this is Philipp from ThinkParQ Support and I would like to help you with
this.

Do you know how bad the data loss was or do you still have the output of
xfs_repair? If so, could you post it here?

Is the underlying filesystem mounted read-write at the moment? It looks
like the service is trying to create a directory it needs
(<storeMetaDirectory>/dentries/35/5B/disposal) and fails while doing that.
Is that directory present on the underlying filesystem?

Could you check if your storeMetaDirectory roughly looks like this:

# ls
buddymir dentries format.conf inodes mirroredSessions nodeNumID originalNodeID sessions

and whether you have all directories that should be present:

# find inodes/ -maxdepth 3 | wc -l
16564
# find dentries/ -maxdepth 3 | wc -l
16564

Best regards
- Philipp

* 'Christian' via beegfs-user <fhgfs...@googlegroups.com> [210512 13:14]:

> --
> You received this message because you are subscribed to the Google Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/5e14364f-faf8-4f64-8105-abebcd01ea9fn%40googlegroups.com.


--
Philipp Falk | System Engineer | m: philip...@thinkparq.com
ThinkParQ GmbH | Trippstadter Strasse 113 | 67663 Kaiserslautern | Germany
CEO: Frank Herold | COB: Dr. Franz-Josef Pfreundt | Registered: Amtsgericht Kaiserslautern HRB 31565 I VAT-ID-No.:DE 292001792

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential or proprietary information.

[Christian]

Hello,

thank you very much for your offer! I was busy setting up a new fileserver from the backups, but a few users lost several days of work, so any help is appreciated.

The root cause of the problem appears to be a hardware failure in the RAID controller ("L2/L3 Cache error") that happened right after a power failure.

I made two copies of what was left of the metadata filesystem and shut down the server. I did not save the output from xfs_repair, but it reported a lot of orphaned inodes. Although further checks did not show any more errors, the original filesystem is still not okay: an "ls -lR" showed a few files with question marks in all coumns (exept the filename). Plus, the original filesystem shows only 128 GB used, whereas the copy shows 241 GB used.

The ls for the meta dir shows dentries,  format.conf,  lock.pid,    nodeNumID, disposal, inodes, lost+found, and originalNodeID. lost+found contains 12853 files and 760 directories...

find inodes/ -maxdepth 3 | wc -l counts 1023146 entries,  and find dentries/ -maxdepth 3 | wc -l counts 835187.

It would be great if there is a way to map the chunks of at least some of the newest files to their files.

Thank you again and best regards,

   Christian

[Philipp Falk]

Hello Christian,

sounds like xfs_repair has moved a lot of the metadata files to lost+found.
There are two important parts of information for those files, which are
their names and their xattrs. Could you check for a couple of the files
below lost+found if they still have their correct filenames, which would be
either human-readable names like on the BeeGFS filesystem, or names that
match the pattern '[0-9A-Z]{1,8}-[0-9A-Z]{8}-[0-9A-Z]{1-8}', e.g.
D8D-5F1696AD-1.

Unfortunately, xfs_repair does not always preserve xattrs, which is where
BeeGFS stores all its metadata. Could you also check whether those files
still have the xattr 'user.fhgfs' like this for a couple of files in
lost+found:

$ getfattr -e base64 -n user.fhgfs -d <filename>

which should look similar to this:

$ getfattr -e base64 -n user.fhgfs -d /beegfs/meta/inodes/35/34/D8D-5F1696AD-1
# file: beegfs/meta/inodes/35/34/D8D-5F1696AD-1
user.fhgfs=0sBAMKAAAAAAAAAAAA7UEAAL+WFl8AAAAAv5YWXwAAAAC/lhZfAAAAAL+WFl8AAAAAAAAAAAAAAABkAAAADgAAAEQ4RC01RjE2OTZBRC0xAAAEAAAAcm9vdAAAAAABAAAAAQAAABoAAAABAAAAAAAIAAEACAAAAAgAAAAAAAAA

A hexdump of that metadata should look similar to this:

$ getfattr -e hex -n user.fhgfs --only-values -d /beegfs/meta/inodes/35/34/D8D-5F1696AD-1 | hexdump -C
00000000 04 03 0a 00 00 00 00 00 00 00 00 00 ed 41 00 00 |.............A..|
00000010 bf 96 16 5f 00 00 00 00 bf 96 16 5f 00 00 00 00 |..._......._....|
*
00000030 00 00 00 00 00 00 00 00 64 00 00 00 0e 00 00 00 |........d.......|
00000040 44 38 44 2d 35 46 31 36 39 36 41 44 2d 31 00 00 |D8D-5F1696AD-1..|
00000050 04 00 00 00 72 6f 6f 74 00 00 00 00 01 00 00 00 |....root........|
00000060 01 00 00 00 1a 00 00 00 01 00 00 00 00 00 08 00 |................|
00000070 01 00 08 00 00 00 08 00 00 00 00 00 00 00 |..............|
0000007e

The first byte, which is 04 in this case, indicates that this is a
directory inode. You can also see the entryID again in the dump and also
the entryID of its parent directory, which is the special ID root in this
case.

The metadata for a file inode (which is inlined into the dentry) looks
similar to this:

$ getfattr -e hex -n user.fhgfs --only-values -d /beegfs/meta/dentries/35/34/D8D-5F1696AD-1/test | hexdump -C
00000000 01 06 13 00 02 00 00 00 80 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 a4 81 00 00 4a 6f c0 60 00 00 00 00 |........Jo.`....|
00000020 4a 6f c0 60 00 00 00 00 4a 6f c0 60 00 00 00 00 |Jo.`....Jo.`....|
00000030 4a 6f c0 60 00 00 00 00 00 00 00 00 00 00 00 00 |Jo.`............|
00000040 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 0c 00 00 00 30 2d 36 30 43 30 36 46 32 39 2d 31 |....0-60C06F29-1|
00000060 00 00 00 00 20 00 00 00 01 00 00 00 00 00 08 00 |.... ...........|
00000070 01 00 08 00 00 00 0e 00 00 00 03 00 00 00 0c 00 |................|
00000080 0f 00 0b 00 00 00 00 00 00 00 00 00 |............|
0000008c

In my case, all of these are still in their correct locations, but with the
meta information from those files, there is a chance that they can be put
in their original place again, or that you can at least match a filename to
a set of chunks on your system.

Best regards
- Philipp

* 'Christian' via beegfs-user <fhgfs...@googlegroups.com> [210601 15:55]:

> To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/c987ddb8-78a9-489f-a541-47b432b20e4an%40googlegroups.com.

[Christian]

Hello,

thank you very much for your reply!

The strange thing with my metadata structure is that none of the files shows a user.fhgfs. I even did a "getfattr -dR" over the whole metadata directory (which took until today). It did not show a single attribute.

It is not possible that BeeGFS 2015.03 (which the old fileserver was using) was using something different than extended attributes, is it? Provided that it used xattrs, too, I think that there is no more chance of restoring the filenames.

Kind regards,

   Christian

[Philipp Falk]

Hi Christian,

there is an option to store the metadata in file data blocks instead of
inside the inode (storeUseExtendedAttribs). I am not sure if that setting
was present in 2015.03, which was a bit before my time at ThinkParQ, but if
the metadata was stored in data blocks in your system, those metadata files
should have a nonzero size. Could you check whether those recovered files
actually aren't empty files?

If they are not empty, you should be able to dump the metadata like this:

$ hexdump -C <filename>

Best regards
- Philipp

* 'Christian' via beegfs-user <fhgfs...@googlegroups.com> [210621 16:56]:

> To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/00107287-6064-44fc-9789-9f2ecee39ba3n%40googlegroups.com.

[Christian]

Hi,

that's probably good news! The old setup indeed hat storeUseExtendedAttribs set to false in beegfs-meta.conf. And, luckily, the vast majority of files are not empty - "find inodes -type f  -size 0" finds only 228 empfy files, whereas the negated expression yields more than a million non-empty files. At first sight, I cannot make much sense of the file contents yet, though - e.g F79-603FBC35-1 contains only two filename-like strings, one being F79-603FBC35-1 itself, the other being F79-603FBC35-1.

Kind regards,

   Christian

[Philipp Falk]

Hi Christian,

yes, in this case this is good news, because extended attributes sometimes
do not survive filesystem repairs. The file contents are binary and not
easy to parse by eye, but the one you opened is probably a directory inode.
Those reference two entryIDs, their own (same as the filename) and their
parent's. The hexdump of the binary data probably starts with a 04,
correct?

To get back the name of that directory, you need to find the dentry for
that directory, which is a different file that contains the entryID (the
filename of the file you mentioned about) and starts with a 02. The name of
that dentry file is the original name of the directory. You can do the same
for the parent's entryID to get more information about the original tree.

There will also be files for which the contained data starts with a 01 and
only contains a single entryID. Those are file inodes inlined into their
dentries. The name of those files is the original file name, but it will not
be possible to find the parent directory if the directory structure wasn't
preserved in the filesystem repair.

If you want to get a more detailed explanation about the meta structure, I
invite you to join the Support Stammtisch next Tuesday at 17:00 CEST, where
we can talk about this in person and can go through some of the possible
recovery steps together.

Best regards
- Philipp

* 'Christian' via beegfs-user <fhgfs...@googlegroups.com> [210622 17:27]:

> To view this discussion on the web visit https://groups.google.com/d/msgid/fhgfs-user/0ad41f85-6775-4227-b0dd-874d48a759d7n%40googlegroups.com.