d6515 node with NPS2/NPS4

[Mitch Gerhardt]

Hi CloudLab ops!

I'm running a NUMA latency characterization experiment on d6515 nodes and need NPS4 mode enabled, but I'm hitting a hardware obstacle. When running on amd024, dmidecode -t 17 shows that only slots A1–A8 (DMI Set 1) are populated, while slots A9–A16 (Set 2) are empty. Because NPS4 requires DIMMs distributed across all four memory channel groups, the BIOS falls back to NPS1 regardless of the NPS setting.

I'm curious if NPS2 or NPS4 is supported on d6515 nodes? I wasn't able to find BIOS parameter options in the CloudLab experiment portal for this hardware type, so I'm not sure whether NPS configuration is something ops can set per-node, or whether it's unsupported/locked on the d6515 image entirely.

If it is supported, could you either:
- Reassign one of my reserved experiment nodes tomorrow to a d6515 node with all 16 DIMM slots populated and NPS4 (or NPS2) enabled, or
- Enable NPS2 as a fallback if the DIMM layout there can support 2 domains?

My CloudLab username is mitchg10 and the experiment runs under the Utah cluster. Happy to provide more details if helpful.

Thanks for your time.

Mitch Gerhardt

[u045...@gcloud.utah.edu]

Hi Mitch, 

Just want to make sure we're on the same page here.  The d6515 nodes are single-socket machines that support 8-channels per socket, and all memory channels are populated at 1x 16GB DIMM per channel.  The d6515 nodes have a uniform spec, so all nodes have the same hardware.  The result is that on each node, slots A1-A8 (slot 1 in each channel) are populated while A9-A16 (slot 2 in each channel) are empty, and there are no d6515 nodes with all 16 slots filled.  The EPYC 7452 CPUs in these servers support NPS4, where you would get two CCDs (2 CCXs/8 physical cores total) and two channels (32GB total) per NUMA domain.  Do you actually need all of the slots filled for your work, or just all of the channels populated?

Regarding BIOS settings, that information is indeed not available on the portal.  We also lock down the BIOS to prevent users from making arbitrary changes, but we can make these changes for you.  I have set NPS4 on d6515 nodes for a few users in the past, so I can do the same for you for your next experiment (it doesn't look like you have anything actively running right now).

Best,

 - Aleks

[Mitch Gerhardt]

Hi Aleks,

Thanks for the clarification, that makes sense. No, I don't need both slots per channel filled; all channels being populated is exactly what I need. Good to know the d6515 spec is uniform.

I'd love to take you up on setting NPS4 for my next experiment. I have a reservation starting tomorrow at 6AM for 4 nodes. Could you set NPS4 on whichever gets allocated? I can send you the experiment ID once it's up if that's easier.

Thanks again for the quick turnaround.

Best,
Mitch

[Aleksander Maricq]

Sure, send us the link to the experiment page once the experiment is set up, that makes it the easiest for us.  I can't guarantee that NPS4 will be enabled right when your experiment starts (I'm usually not awake at 6 AM!) but I'll get to it that morning.  How long is your reservation for out of curiosity?

[Mitch Gerhardt]

Sounds good! No problem! I have the experiment running until tomorrow at 10PM. 

The Subject's Distinguished Name is as follows
countryName           :PRINTABLE:'US'
stateOrProvinceName   :PRINTABLE:'Utah'
localityName          :PRINTABLE:'Salt Lake City'
organizationName      :PRINTABLE:'Utah Network Testbed'
organizationalUnitName:PRINTABLE:'aptlab.CS620426SP.mitchg-303009'
commonName            :PRINTABLE:'626040f6-022a-4540-acb0-5f14d590b5c0'

https://www.cloudlab.us/status.php?uuid=553bbb45-3f3d-4dbd-a24d-95f0de3fb4a2

[Aleksander Maricq]

The changes have been staged on all four of your nodes, reboot them at your discretion to apply the changes.  Sorry for the delay!

Will you be extending this experiment, or do you only need it until 10 PM?

[Mitch Gerhardt]

Thanks! No worries about the delay, been running other tests. I'll be extending it -- could only reserve 16 hours.

[Mitch Gerhardt]

Hi Alek! Sorry to bother again about this, but could you do the same for this active experiment with one d6515 Utah node? 

https://www.cloudlab.us/status.php?uuid=86d4c7a9-4b1a-4456-aa35-e74513f81578

Best,

Mitch

[Aleksander Maricq]

Hi Mitch,

I've applied the change and rebooted your node (it didn't look like anything was running).  When it comes back up, it should have NPS set to 4.  Keep in mind that this experiment can't be extended due to other conflicting reservations, so its expiry time is set in stone.

Best,

 - Aleks

[Mitch Gerhardt]

Thank you, Aleks! Sorry for my last message being to "Alek," by the way. 

Mitch

[Mitch Gerhardt]

Morning Aleks! Sorry, but one last time, could you make the change for this experiment? There's nothing running so it should be okay to reboot. 

https://www.cloudlab.us/status.php?uuid=46dd045f-3cd9-4c81-b78f-6ab0cdf2bc99

Thank you again for all your help. My project is due, so this is the last request.

Best,

Mitch

[Mitch Gerhardt]

Sorry, this is the URL. I think something might be happening with the Utah nodes, though.

https://www.cloudlab.us/status.php?uuid=adcd7c5f-908e-43c6-a1bc-95f458270eb2

Mitch

[Aleksander Maricq]

Hi Mitch,

Apologies, there is indeed something going on at CloudLab Utah.  We're currently working on tracking down issues with high packet-loss to the cluster, and are working with our upstream network provider to get this resolved.  We'll let you know once things have been resolved and then hopefully there will be enough time still for you to get your project done.  Sorry for the inconvenience!

Best,

 - Aleks

[Mitch Gerhardt]

No worries! Looks like we're back in action: https://www.cloudlab.us/status.php?uuid=fae19bc7-8208-4629-844e-08825ff1d9c8

Thank you!

Mitch

[Aleksander Maricq]

Indeed we are!  Took quite a while to track down the issue, but it's fixed now.

BIOS change has been made, node is rebooting right now.

[Mitch Gerhardt]

Glad to hear that! I'm currently finding that standard apt-get package installation is non-functional. The nodes have no route to Canonical's Ubuntu archive servers: connections to us.archive.ubuntu.com and archive.ubuntu.com time out on both IPv4 and IPv6. The only reachable apt server is repos.emulab.net, which carries emulab-specific packages but not the standard Ubuntu package set.

Is there a recommended way to get general internet access for these nodes?

Thanks!

Mitch

[David M Johnson]

I'm not sure exactly what the current problem is, but the Ubuntu Apt
mirror infrastructure had a difficult weekend, and maybe some issues
remain (or are recurring with new attacks, possibly -- I haven't been
following in detail). (e.g.
https://news.ycombinator.com/item?id=47972213,
https://status.canonical.com/, etc).

Looks like you are already doing this, but you can set a different
mirror manually. For instance, to use the Utah CS department's Ubuntu
Apt mirror, you could do something like

sudo sed -i
's/http:\/\/archive.ubuntu.com/http:\/\/ubuntu.cs.utah.edu\/ubuntu/g'
/etc/apt/sources.list
sudo sed -i
's/http:\/\/archive.ubuntu.com/http:\/\/ubuntu.cs.utah.edu/ubuntu\/g'
/etc/apt/sources.list

You'd probably want to make sure to revert back to archive.ubuntu.com
before you create an image that might be used at non-Utah Cloudlab clusters.

David

On 5/4/26 16:31, Mitch Gerhardt wrote:
> Glad to hear that! I'm currently finding that standard apt-get package
> installation is non-functional. The nodes have no route to Canonical's
> Ubuntu archive servers: connections to us.archive.ubuntu.com and
> archive.ubuntu.com time out on both IPv4 and IPv6. The only reachable
> apt server is repos.emulab.net, which carries emulab-specific packages
> but not the standard Ubuntu package set.
>
> Is there a recommended way to get general internet access for these nodes?
>
> Thanks!
> Mitch
>
> On Monday, May 4, 2026 at 5:39:58 PM UTC-4 Aleksander Maricq wrote:
>
> Indeed we are!  Took quite a while to track down the issue, but it's
> fixed now.
>
> BIOS change has been made, node is rebooting right now.
>
> On Monday, May 4, 2026 at 3:27:33 PM UTC-6 mit...@vt.edu wrote:
>
> No worries! Looks like we're back in action: https://
> www.cloudlab.us/status.php?

> uuid=fae19bc7-8208-4629-844e-08825ff1d9c8 <https://
> www.cloudlab.us/status.php?
> uuid=fae19bc7-8208-4629-844e-08825ff1d9c8>

>
> Thank you!
> Mitch
>
> On Monday, May 4, 2026 at 2:48:01 PM UTC-4 Aleksander Maricq wrote:
>
> Hi Mitch,
>
> Apologies, there is indeed something going on at CloudLab
> Utah.  We're currently working on tracking down issues with
> high packet-loss to the cluster, and are working with our
> upstream network provider to get this resolved.  We'll let
> you know once things have been resolved and then hopefully
> there will be enough time still for you to get your project
> done.  Sorry for the inconvenience!
>
> Best,
>  - Aleks
>
> On Monday, May 4, 2026 at 6:35:40 AM UTC-6 mit...@vt.edu wrote:
>
> Sorry, this is the URL. I think something might be
> happening with the Utah nodes, though.
>
> https://www.cloudlab.us/status.php?

> uuid=adcd7c5f-908e-43c6-a1bc-95f458270eb2 <https://
> www.cloudlab.us/status.php?uuid=adcd7c5f-908e-43c6-
> a1bc-95f458270eb2>
>
> Mitch
>
> --
> You received this message because you are subscribed to the Google
> Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to cloudlab-user...@googlegroups.com
> <mailto:cloudlab-user...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/
> cloudlab-users/a103196a-0510-480e-b960-f33081845dd3n%40googlegroups.com
> <https://groups.google.com/d/msgid/cloudlab-users/a103196a-0510-480e-
> b960-f33081845dd3n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Mitch Gerhardt]

Got it, thanks!