Disabling Memory Channel Interleaving

[Omar Eqbal]

Hi CloudLab Team,

We are working on a research project that requires access to a server whose BIOS allows disabling memory channel interleaving within a single socket ("Disable Hardware Interleaving" section in https://ramryder-project.github.io/docs/hardware-support/dimm#disable-hardware-interleaving).

We would like to work with one of these nodes - Intel Ice Lake (r650, sm220u, sm110p, d750) or AMD EPYC Zen 4 (r6615, d7615). Does the BIOS of any of these nodes allow disabling memory channel interleaving? If so, would it be possible to disable the interleaving?

Thanks,
Omar

[Aleksander Maricq]

Hi Omar,

I'll check on these hardware types and get back to you.

Best,

 - Aleks

[Aleksander Maricq]

Just to confirm, do you specifically need Ice Lake, or should I read that as "Ice Lake or newer"?  From my recollection, we have a number of hardware types that have Node Interleaving disabled by default, including the c6620 nodes which have Emerald Rapids CPUs.  If you can't use Emerald Rapids, could you let me know for my own curiosity why Ice Lake is necessary?

Out of the other types you listed, I was able to quickly confirm that the d750 nodes have "Node Interleaving" set to disabled by default, and the d7615 nodes have no obvious setting to control that.  I will check on the others.

[Omar Eqbal]

Thanks for the response.

We are looking to disable channel interleaving, not node interleaving. If node interleaving is enabled, then contiguous physical addresses map to all the DIMM channels of all the NUMA nodes in an interleaved manner. Disabling node interleaving partitions the physical address ranges between NUMA nodes, but it still leaves the physical addresses interleaved across the DIMM channels within a single NUMA node. After disabling channel interleaving, the physical address ranges will be partitioned between the DIMM channels within a single NUMA node (refer to Figure 10 in https://www.usenix.org/system/files/osdi26-zhou-yanbo.pdf).

Different BIOS might have the setting under different names; e.g., IMC Interleaving (Auto / 1-way / 2-way; set to 1-way for disabling) or Channel Interleaving (Auto / 1-way / 2-way / 3-way / 4-way; set to 1-way for disabling) or Memory Interleaving / Interleave Mode (set to disabled).

We wanted to specifically check Ice Lake because our current system implementation uses Ice Lake-specific performance counters, which have changed in newer generations. However, if an Ice Lake server with the ability to disable channel interleaving is not available, we would be open to looking at newer generations and porting our system accordingly. If neither Ice Lake nor newer Intel nodes have the option to disable channel interleaving, then we would like to check if any of the AMD nodes mentioned above have this option.

[Aleksander Maricq]

On more recent Dell machines at least, this setting is called "Node Interleaving".  On the d750 hwtype (which is an Ice Lake-based Dell PowerEdge R750), Node Interleaving is set to Disabled by default.  Ditto for the aforementioned c6620 hwtype, which is Emerald Rapids-based.  I looked more into the Zen 4-based d7615 nodes (Dell PowerEdge R7615), and despite there not being an explicit setting for Node Interleaving that I could see, it looks like the behavior is tied to the NUMA node per socket setting.  NPS is set to 1 by default, and it seems like that would also mimic the behavior that you're looking for.  In fact, I would hazard a guess that all of our modern hardware types that have a Memory Interleaving setting just kept it on whatever the factory default was, and I get the sense from reading around that the factory default is typically "Disabled" these days since modern OSes don't really need it to be Enabled anymore.  I haven't looked at the nodes at Clemson or Wisconsin yet, but I get the sense that we could accommodate your experiment pretty easily.

[Omar Eqbal]

Thanks for checking this.

We had tested the "Node Interleaving" setting on our Intel Ice Lake server and had found that disabling Node Interleaving does not disable Channel Interleaving, and there is no setting to disable Channel Interleaving on our server. The two settings are different - Node Interleaving setting is for interleaving across NUMA nodes, and Channel Interleaving is for interleaving within a single NUMA node.

More concretely, we would like to confirm the following:

• In any of the Intel Ice Lake (or newer) servers, is there an interleaving related setting, other than "Node Interleaving"?
• In any of the AMD Zen 4 servers, is there an interleaving related setting, other than "Node Interleaving" or "NPS"?

[Aleksander Maricq]

Ah, thanks for clarifying.  From my recollection, there is not such a setting on any of the Dell servers.  I will check the Supermicro servers, as well as double-check the Dell servers, and get back to you tomorrow.