Re: [euler-users] Performance Impacts of Storage Upgrades

[Colin Vanden Heuvel]

Hello Euler Users,

Euler is just wrapping up phase 2 of this summer's storage upgrade, meaning that the vast majority of filesystem data is now stored on newer, faster systems. There are still a few older systems operating as replicas at this time, but those will be phased out during the final phases of the upgrade.

Phase 3 will start soon, but not as soon as we had hoped. Due to some supply chain problems, we won't have the full set of network hardware we need in time to do the upgrades for phases 3 and 4 sequentially. Instead, we will perform them at the same time, which will require the filesystem to go completely offline for a brief period.

It is still unclear exactly when the requisite hardware will arrive. As such, we have a broad window for when this outage could happen. It is our hope that the hardware will arrive in enough time that upgrades can be conducted between the end of the summer term (Aug. 13) and the start of the fall semester (Sep. 6). I will send notice via this mailing list once we have determined a date for the outage.

Until then, please enjoy the modest increase to Euler's filesystem performance, and please let us know at euler-...@engr.wisc.edu if you run into any issues while using Euler.

Regards,

Colin Vanden Heuvel

---

From: Colin Vanden Heuvel <colin.van...@wisc.edu>
Sent: Wednesday, July 5, 2023 9:38
To: 'Colin Vanden Heuvel' via euler-users <euler...@g-groups.wisc.edu>; Colin Vanden Heuvel <colin.van...@wisc.edu>
Subject: Re: [euler-users] Performance Impacts of Storage Upgrades

 

Hi,

Phase 2 is not moving as quickly as I had hoped, so I'm going to increase the amount of backend traffic that Euler is allowed to generate at once. This change will more than double the current overhead. If your work is I/O-bound and you have a deadline coming up in the next week or so, please let me know ASAP and I will attempt to accommodate you.

-- Colin

---

From: 'Colin Vanden Heuvel' via euler-users <euler...@g-groups.wisc.edu>
Sent: Thursday, June 29, 2023 17:19
To: 'Colin Vanden Heuvel' via euler-users <euler...@g-groups.wisc.edu>
Subject: Re: [euler-users] Performance Impacts of Storage Upgrades

 

Hello again,

Euler's storage upgrade has been proceeding slowly but steadily, and we're ready to proceed to phase 2 of the migration. This means that, once again the storage backend will be quite busy for a while. It will be the busiest for the next few days over what I hope for many of you is a long weekend, and will begin to taper off starting around the July 4th holiday.

Since the end of Phase 1, I've implemented better monitoring and alerting for problems, so hopefully we'll be able to avoid or at least quickly resolve any issues which might pop up. Feel free to reach out if you discover a problem or have any further questions.

Regards,

Colin Vanden Heuvel

---

From: 'Colin Vanden Heuvel' via euler-users <euler...@g-groups.wisc.edu>
Sent: Monday, June 5, 2023 11:00
To: 'Colin Vanden Heuvel' via euler-users <euler...@g-groups.wisc.edu>
Subject: [euler-users] Performance Impacts of Storage Upgrades

 

Hello Euler users,

Euler will be receiving a number of storage upgrades in the coming month or so, raising the usable space to 1PB and incorporating a second datacenter for improved resiliency during power failures. This process will also introduce the prerequisite technologies for an encrypted-at-rest storage tier; interested PIs may feel free to open a support ticket to request more information.

During this process, the storage backend will be quite busy as it balances data across the new drives. It is likely performance-sensitive data transfers will see some impact to both latency and throughput during this time, but I will do my best to notify here when each stage of the transfer begins and ends.

There are four stages planned, and the first will begin before EOB tomorrow.

Thanks for your patience, and please look forward to the improved quality of service at the end of these upgrades!

Regards,

Colin

--
You received this message because you are subscribed to the Google Groups "euler-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to euler-users...@g-groups.wisc.edu.
To view this discussion on the web visit https://groups.google.com/a/g-groups.wisc.edu/d/msgid/euler-users/CY4PR0601MB3604E9A7E2BED3925C9D78A3F64DA%40CY4PR0601MB3604.namprd06.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "euler-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to euler-users...@g-groups.wisc.edu.
To view this discussion on the web visit https://groups.google.com/a/g-groups.wisc.edu/d/msgid/euler-users/DM5PR0601MB36064FC4F601B96C862122CCF625A%40DM5PR0601MB3606.namprd06.prod.outlook.com.