BeeGFS Read-Only.

[Laurence Mayer]

We are using Beegfs version 7

Is there a way to move Beegfs to read-only via command line?

beegfs-ctl --listtargets --nodetype=storage --state

There is --state which gives the state of the Targets, but there doesn't seem to be a way that I can find to move the storage to read-only and then when ready back to read-write.

Any suggestions?

Thanks

Laurence 

[Guan Xin]

Hi,

Although this is possible, ro-ing a storage target seems more like a solution.

What was the original requirement/problem?

Regards,

Guan

[Laurence Mayer]

Hi Gaun,

We're encountering an issue where BeeGFS clients hang when the file system runs out of space. This behavior has been consistent across all versions we've used. 

Specifically, when we reach zero available space, commands like df -h and ls /mnt/beegfs -al become unresponsive, which complicates our ability to free up space since all clients are effectively hung.

I have two questions:

1. Is this hanging behavior expected when BeeGFS reaches full disk capacity?
2. If this is expected, would you recommend any best practices for handling such scenarios? I’m considering implementing a script to set the device to read-only when space approaches zero to prevent this issue, but I'm open to other suggestions that might be more effective.

Thank you for your guidance.

Thanks

Laurence 

[Guan Xin]

Hi Laurence,

I've not seen such behavior before.

BeeGFS does hang when the underlying filesystem of storage targets hangs,

e.g. when an old version of xfs is almost full, or too many failed disks in a zfs.

Related but a little off topic -- I've seen beegfs-storage server (and so all its targets)

hangs when any of its targets hangs.

That's something to be fixed by BeeGFS developers.

Expected behavior: Only the suspended/blocked target hangs.

Regards,

Guan

[Laurence Mayer]

Hi Guan,

The hang/freeze happens on all our beegfs clusters (we have many)

That's the thing, the filesystem is not hung i.e. that's actually how we free up space so that we can "release" the hang and continue working.

(I've created a dummy file on the filesystem of 10GB and then just delete that by ssh'ing into the Storage and deleting the file.

Once there is space, everything automatically returns to normal and I can continue to deleting via the beegfs mount.

So my thought was when we getting close to zero space I could have a script that puts the Storage in RO to avoid reaching 100%.

Thanks

Laurence 

[Toby Darling]

Hi Laurence

I'm doing something like this with a script run every 10 minutes from cron

It checks all the pools in the cluster, and if any are 99% full, it does
a "zfs set userquota@$user=1" for every user on all pools. When the
fullest pool gets back to 97% full, it sets quotas back to none.

Cheers
Toby

On 06/11/2024 07:29, Laurence Mayer wrote:
> CAUTION: This email originated from outside of the LMB:
> *.-fhgfs-user+bncBDGP3HPD4MEBB5FVVS4QMGQE26P753Y@googlegroups.com-.*
> Do not click links or open attachments unless you recognize the sender
> and know the content is safe.
> If you think this is a phishing email, please forward it to
> phis...@mrc-lmb.cam.ac.uk
>
>
> --

> 1. Is this hanging behavior expected when BeeGFS reaches full
> disk capacity?
> 2. If this is expected, would you recommend any best practices

> --
> You received this message because you are subscribed to the Google
> Groups "beegfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com <mailto:fhgfs-
> user+uns...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/fhgfs-
> user/6d32dad3-1702-4acd-8e1d-ca154d61bd2fn%40googlegroups.com <https://
> groups.google.com/d/msgid/fhgfs-user/6d32dad3-1702-4acd-8e1d-
> ca154d61bd2fn%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Toby Darling, Scientific Computing (2N249)
MRC Laboratory of Molecular Biology
https://www.mrc-lmb.cam.ac.uk/scicomp/

[Laurence Mayer]

Hi Toby,

Thank you for your response.

To clarify, we’re using XFS as the underlying filesystem. The issue we’re experiencing is with BeeGFS freezing rather than the underlying filesystem itself. I’m looking for a way to configure BeeGFS in read-only mode as a workaround.

Of course, identifying the root cause of why BeeGFS freezes when the filesystem reaches full capacity would be the ideal solution.

Best regards,
Laurence

[Laurence Mayer]

Hi Gaun,

We're encountering an issue where BeeGFS clients hang when the file system runs out of space. This behavior has been consistent across all versions we've used. 

Specifically, when we reach zero available space, commands like df -h and ls /mnt/beegfs -al become unresponsive, which complicates our ability to free up space since all clients are effectively hung.

I have two questions:

1. Is this hanging behavior expected when BeeGFS reaches full disk capacity?

1. If this is expected, would you recommend any best practices for handling such scenarios? I’m considering implementing a script to set the device to read-only when space approaches zero to prevent this issue, but I'm open to other suggestions that might be more effective.

Thank you for your guidance.

Thanks

Laurence 

--
You received this message because you are subscribed to a topic in the Google Groups "beegfs-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fhgfs-user/JKTcQzbx1CI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fhgfs-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/fhgfs-user/cf437cb2-21e1-4997-9312-610617fee65dn%40googlegroups.com.

[Guan Xin]

Hi

> Is this hanging behavior expected when BeeGFS reaches full disk capacity?

It should not. & Never seen during my tests.

That said, we use quota in production env to minimize the chance of BeeGFS getting full.

Regards,

Guan