File system consistency issue

[Le Xu]

Hi:

I'm encountering issue with creating files in my filesystem and fsck from the client does not work (however I don't have access to the NFS hast to check what's going on). Every time when I try to make write operation the message says "cannot create directory.... Read-only file system". The link to the experiment is: https://www.emulab.net/portal/status.php?uuid=7033e6ae-f2eb-11eb-84f8-e4434b2381fc

Anyone has suggestions on how to fix this?

Thanks!

Le

[Le Xu]

(Not sure if this is helpful) The link to the dataset URN is urn:publicid:IDN+emulab.net:Stella+ltdataset+statefun

Thanks,

Le

[Mike Hibler]

Looks like a multi-disk failure on the storage array. This is not good,
and I will not get a chance to look at it before tomorrow. Sorry.

> --
> You received this message because you are subscribed to the Google Groups
> "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to emulab-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/
> emulab-users/e47aad63-2366-435b-aec3-fbc2fafc7bfen%40googlegroups.com.

[Mike Hibler]

We lost three disks in three days on our RAID6. I was able to revive two
of the disks and get the array limping again. It appears the only dataset
that got corrupted was your statefun-2 dataset. But it is beyond repair.
Since you just created it, hopefully you did not lose anything important.

I have replaced the remaining failed disk, and the RAID is rebuilding.
After that, I will replace the other two one at a time.

We apparently were not getting alerts from the array, but I am not sure we
could have replaced the drives in time anyway. It could be the drives are
just getting old (the one I replaced today was manufactured in 2012) so we
will see if we can stay ahead of the failures.

On Sun, Aug 01, 2021 at 08:15:42PM -0600, Mike Hibler wrote:
> Looks like a multi-disk failure on the storage array. This is not good,
> and I will not get a chance to look at it before tomorrow. Sorry.
>
> On Sun, Aug 01, 2021 at 04:04:49PM -0700, Le Xu wrote:
> > (Not sure if this is helpful) The link to the dataset URN
> > is urn:publicid:IDN+emulab.net:Stella+ltdataset+statefun
> >
> >
> > Thanks,
> >
> > Le
> >
> > On Sunday, August 1, 2021 at 3:49:02 PM UTC-5 Le Xu wrote:
> >
> > Hi:
> >
> > I'm encountering issue with creating files in my filesystem and fsck from
> > the client does not work (however I don't have access to the NFS hast to
> > check what's going on). Every time when I try to make write operation the
> > message says "cannot create directory.... Read-only file system". The link

> > to the experiment is:??https://www.emulab.net/portal/status.php?uuid=

> > 7033e6ae-f2eb-11eb-84f8-e4434b2381fc
> >
> >
> > Anyone has suggestions on how to fix this?
> >
> > Thanks!
> >
> > Le
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "emulab-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email
> > to emulab-users...@googlegroups.com.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/
> > emulab-users/e47aad63-2366-435b-aec3-fbc2fafc7bfen%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "emulab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emulab-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/emulab-users/20210802021542.GP54609%40flux.utah.edu.