Cloudlab Utah cluster down tomorrow (1/18)

[Mike Hibler]

As mentioned in the News item (https://www.cloudlab.us/portal-news.php?idx=51), I will be continuing the series of upgrades to Cloudlab related servers tomorrow, doing the Cloudlab Utah cluster.

I am planning on an 8am MST start tomorrow and hope to be done by noon MST. You will not be able to start or terminate experiments on the Utah cluster while the upgrade is in progress. Existing experiments may continue to run unaffected, depending on their use of DNS (which will be slowed) or /proj (which will cause a hang). Nodes will also not reboot properly. So it is best to assume that you will not get anything done on Cloudlab Utah tomorrow.

[Yucheng Yin]

Hi Mike,

Sorry for the possible duplication.

We have a short-term dataset on Utah cluster (urn:publicid:IDN+utah.cloudlab.us:cloudmigration-pg0+stdataset+DG-UTAH-BASELINES) which has entered the grace period and we are not able to extend the dataset due to the ongoing system upgrade.

The dataset holds ALL the data we are using for the upcoming sigcomm submission.

Would you mind helping us extend that dataset after the deadline (e.g., Feb 5th ish) if possible?

Thank you!

Yucheng

[Kirk Webb]

Yungchen,

Mike can probably provide better answers, but I just want to reassure
you that your dataset won't be deleted automatically, and in fact we
would reach out before deleting anything even manually to be sure
people are not losing data. Having said that, you might consider
making an offsite copy of your dataset since we don't perform any sort
of backups on datasets. There is RAID 6 redundancy for the volume, but
of course other unforeseen data corruption or accidental file removal
can happen.

-Kirk

> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/52bf49ad-2392-4788-93fa-de4596eec50dn%40googlegroups.com.

[Mike Hibler]

Cloudlab Utah is done. Took longer than Apt because we were also reconfiguring some experimental trunk links.

[Yucheng Yin]

Thank you Kirk and Mike for your help!

[Mike Hibler]

I have extended that dataset til Feb 6th.

Note that short-term datasets were intended to have a lifespan of days to
weeks and not months. That is what long-term datasets are for. Short-term
datasets will actually auto-delete after a short grace period. I had to turn
that on when we were short of space. Long-term datasets are still kept around
after the grace period expires. The current parameters at the Utah cluster are:

stdataset:
Maximum size: 1.00 TiB (1099511627776 bytes).
Expiration: after a lease-specific time period (maximum of 7.0 days from creation).
Disposition: destroyed after expiration plus 1.0 days grace period.
Extensions: allows up to 2 1.0 day extensions during grace period.

ltdataset:
Maximum size: determined by project quota.
Expiration: after 150.0 days idle.
Disposition: locked-down after expiration plus 180.0 days grace period.
Extensions: none.

These are obnoxiously stingy I will admit. :-) We are hoping to bring more
storage online.

> You received this message because you are subscribed to a topic in the Google Groups "cloudlab-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/cloudlab-users/OPiIkoK2KUY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to cloudlab-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/CAHhYwBihHA4-XH8W%3DQ%2BVR5EPneqW40DVunf8VMexG3U8Xj-b0Q%40mail.gmail.com.

[Yucheng Yin]

Thanks a lot Mike for your generous help!!

We are also actively scp-ing our data back. Sorry our experiment lasted a bit longer than expected and we will pay more attention to the dataset usage in the future.

Hope to see more storage space in the future! :)

Best,
Yucheng

To view this discussion on the web visit https://groups.google.com/d/msgid/cloudlab-users/20220118234311.GF89093%40flux.utah.edu.