nfs dataset read-only
73 views
Skip to first unread message
Minhao Jin
Jan 21, 2026, 3:29:06 PMJan 21
to cloudlab-users
Hi Admin,
I'm running the experiment te-msr under the project https://www.cloudlab.us/show-project.php?pid=NetSyn
My nfs dataset urn:publicid:IDN+wisc.cloudlab.us:netsyn-pg0+stdataset+te suddenly becomes read-only. Can you help to take a look at it. Thanks!
My nfs dataset urn:publicid:IDN+wisc.cloudlab.us:netsyn-pg0+stdataset+te suddenly becomes read-only. Can you help to take a look at it. Thanks!
Mike Hibler
Jan 21, 2026, 5:00:39 PMJan 21
to cloudla...@googlegroups.com
Sorry, we had an emergency shutdown of the storage server and it stayed down
long enough to disconnect all the iSCSI sessions. Linux saw this as a file
system error and remounted the filesystem RO to prevent further damage.
You will need to reboot you "nfs" node to reestablish the connection.
> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 546484c6-c27c-45fb-bfc2-bd288f5e5f62n%40googlegroups.com.
long enough to disconnect all the iSCSI sessions. Linux saw this as a file
system error and remounted the filesystem RO to prevent further damage.
You will need to reboot you "nfs" node to reestablish the connection.
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 546484c6-c27c-45fb-bfc2-bd288f5e5f62n%40googlegroups.com.
Minhao Jin
Jan 21, 2026, 7:35:53 PMJan 21
to cloudlab-users
Hi admin,
I rebooted the nfs instance and it doesn't seem to work. So I tried to terminate the experiment and relaunch a new one. Then it seems that the "nfs" node is stuck at the "booting" state
https://www.cloudlab.us/status.php?uuid=d4a5ceef-f728-11f0-bc80-e4434b2381fc
I rebooted the nfs instance and it doesn't seem to work. So I tried to terminate the experiment and relaunch a new one. Then it seems that the "nfs" node is stuck at the "booting" state
https://www.cloudlab.us/status.php?uuid=d4a5ceef-f728-11f0-bc80-e4434b2381fc
Minhao Jin
Jan 22, 2026, 10:46:02 AMJan 22
to cloudlab-users
Hi Admin,
Sorry for the frequent message but it's a bit urgent for me. Can I ask whether the storage in Wisc is back or is there an eta for that? The nfs node booting still failed. Thanks again!
Mike Hibler
Jan 22, 2026, 11:09:46 AMJan 22
to cloudla...@googlegroups.com
I am working on this now.
On Thu, Jan 22, 2026 at 07:46:02AM -0800, Minhao Jin wrote:
> Hi Admin,
>
> 2af595ae-cc81-49fd-bf33-dc6093940372n%40googlegroups.com.
On Thu, Jan 22, 2026 at 07:46:02AM -0800, Minhao Jin wrote:
> Hi Admin,
>
Mike Hibler
Jan 22, 2026, 11:52:35 AMJan 22
to cloudla...@googlegroups.com
Go ahead and try again.
I have cleaned up all the bad state on the storage server.
On Thu, Jan 22, 2026 at 09:09:42AM -0700, Mike Hibler wrote:
> I am working on this now.
>
> On Thu, Jan 22, 2026 at 07:46:02AM -0800, Minhao Jin wrote:
> > Hi Admin,
> >
> > Sorry for the frequent message but it's a bit urgent for me. Can I ask whether
> > the storage in Wisc is back or is there an eta for that? The nfs node booting
> > still failed. Thanks again!
> >
> > https://www.cloudlab.us/status.php?uuid=40d98338-f79c-11f0-bc80-e4434b2381fc
> >
> > On Wednesday, January 21, 2026 at 7:35:53???PM UTC-5 Minhao Jin wrote:
> >
> > Hi admin,
> >
> > I rebooted the nfs instance and it doesn't seem to work. So I tried to
> > terminate the experiment and relaunch a new one. Then it seems that the
> > "nfs" node is stuck at the "booting" state
> > https://www.cloudlab.us/status.php?uuid=
> > d4a5ceef-f728-11f0-bc80-e4434b2381fc??
> > www.cloudlab.us/
> > > show-project.php?pid=NetSyn
> > >
> > > My nfs
> > dataset??urn:publicid:IDN+wisc.cloudlab.us:netsyn-pg0+stdataset+te
I have cleaned up all the bad state on the storage server.
On Thu, Jan 22, 2026 at 09:09:42AM -0700, Mike Hibler wrote:
> I am working on this now.
>
> On Thu, Jan 22, 2026 at 07:46:02AM -0800, Minhao Jin wrote:
> > Hi Admin,
> >
> > Sorry for the frequent message but it's a bit urgent for me. Can I ask whether
> > the storage in Wisc is back or is there an eta for that? The nfs node booting
> > still failed. Thanks again!
> >
> > https://www.cloudlab.us/status.php?uuid=40d98338-f79c-11f0-bc80-e4434b2381fc
> >
> > On Wednesday, January 21, 2026 at 7:35:53???PM UTC-5 Minhao Jin wrote:
> >
> > Hi admin,
> >
> > I rebooted the nfs instance and it doesn't seem to work. So I tried to
> > terminate the experiment and relaunch a new one. Then it seems that the
> > "nfs" node is stuck at the "booting" state
> > https://www.cloudlab.us/status.php?uuid=
> >
> > On Wednesday, January 21, 2026 at 5:00:39???PM UTC-5 Mike Hibler wrote:
> >
> > Sorry, we had an emergency shutdown of the storage server and it stayed
> > down
> > long enough to disconnect all the iSCSI sessions. Linux saw this as a
> > file
> > system error and remounted the filesystem RO to prevent further damage.
> > You will need to reboot you "nfs" node to reestablish the connection.
> >
> > On Wed, Jan 21, 2026 at 12:29:06PM -0800, Minhao Jin wrote:
> > > Hi Admin,
> > >
> > > I'm running the experiment te-msr under the project??https://
> > On Wednesday, January 21, 2026 at 5:00:39???PM UTC-5 Mike Hibler wrote:
> >
> > Sorry, we had an emergency shutdown of the storage server and it stayed
> > down
> > long enough to disconnect all the iSCSI sessions. Linux saw this as a
> > file
> > system error and remounted the filesystem RO to prevent further damage.
> > You will need to reboot you "nfs" node to reestablish the connection.
> >
> > On Wed, Jan 21, 2026 at 12:29:06PM -0800, Minhao Jin wrote:
> > > Hi Admin,
> > >
> > www.cloudlab.us/
> > > show-project.php?pid=NetSyn
> > >
> > > My nfs
> > dataset??urn:publicid:IDN+wisc.cloudlab.us:netsyn-pg0+stdataset+te
> > > suddenly becomes read-only. Can you help to take a look at it.
> > Thanks!
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups
> > > "cloudlab-users" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > send an email
> > > to cloudlab-user...@googlegroups.com.
> > > To view this discussion visit https://groups.google.com/d/msgid/
> > cloudlab-users/
> > > 546484c6-c27c-45fb-bfc2-bd288f5e5f62n%40googlegroups.com.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "cloudlab-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email
> > to cloudlab-user...@googlegroups.com.
> > To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> > 2af595ae-cc81-49fd-bf33-dc6093940372n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/20260122160942.GG42617%40flux.utah.edu.
> > Thanks!
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups
> > > "cloudlab-users" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > send an email
> > > to cloudlab-user...@googlegroups.com.
> > > To view this discussion visit https://groups.google.com/d/msgid/
> > cloudlab-users/
> > > 546484c6-c27c-45fb-bfc2-bd288f5e5f62n%40googlegroups.com.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "cloudlab-users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email
> > to cloudlab-user...@googlegroups.com.
> > To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> > 2af595ae-cc81-49fd-bf33-dc6093940372n%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages