Orphaned chunks (BeeGFS 7.4.6)

[Steffen Grunewald]

Good morning,
this may be interesting for several people, not only the ones with a
support contract...

After a few power (supply) failures (the exact reason of which remains to
be investigated), I had to run an fsck on the affected BeeGFS filesystem.
This is 7.4.6, with 4 storage targets.

I ended up with almost 2000 "orphaned chunks" which subsequently were not
handled by --automatic repair, "ra:Nothing". This seems to have been
default behaviour for quite a while (I found a note in 7.2.2 Release
Notes mentioning it)?

The documentation doesn't seem to contain any recipe how to efficiently
handle this (manually, I suppose), so I'd have to look up each and every
chunk matching the chunk ID and target before doing "something" about
them.
There seem to be only three patterns, i.e. ${x}-685EB512-4 with x taken
from a series of hexadecimal numbers (apparently not starting with 1!),
so this might make things a bit easier.

I know that there's a mapping (--getentryinfo; --find) of files to chunks
but this wouldn't work here.

Any suggestions how to clean this mess up in an efficient way?

Thanks,
S

--
Steffen Grunewald, Cluster Administrator
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany
~~~
Fon: +49-331-567 7274
Mail: steffen.grunewald(at)aei.mpg.de
~~~

[Jan Behrend]

Hi Steffen,

I had the same or similar problem some time ago, and was offered a patched
version of beegfs-fsck by Thinkparq which took care of this.

Cheers, Jan

--
MAX-PLANCK-INSTITUT fuer Radioastronomie
Jan Behrend - Backend Development Group
----------------------------------------
Auf dem Huegel 69, D-53121 Bonn                                  
Tel: +49 (228) 525 248
http://www.mpifr-bonn.mpg.de

[Joe McCormick]

We plan to re-enable the ability for beegfs-fsck to cleanup orphaned chunks in the next 7.4.x and 8.x releases. This was disabled to be extra cautious about avoiding accidental data loss scenarios. For example if a metadata node was started with a newly initialized (empty) target due to the underlying file system being improperly mounted, orphaned chunks could be identified by fsck, but the corrective action is to properly mount the meta target (not delete the chunks). Those scenarios should never happen as long as safeguards like storeFsUUID and storeAllowFirstRunInit=false, but those may not be set on every system. The compromise we landed on was to add an extra confirmation guard better explaining the implications of deleting orphaned chunks, and that you may have other options.

~Joe