Attach SSDs to r7525 nodes

[Felix Zhang]

Hi,

I previously sent this request in an email but it may have been missed so I'm re-posting it below:

I'm working on a project which plans to have DPUs (i.e., BlueField-2s) directly interact with SSDs over PCIe, using either SPDK or DOCA SNAP. We were wondering if a few (ideally 4) of the r7525 nodes in the CloudLab Clemson cluster could be equipped with a PCIe-accessible SSD to allow us to benchmark and test our prototype.
 
Many thanks,
Felix

[Robert Ricci]

We might have an easier time putting a BF2 in one of the 'flex' nodes at
Utah - those are chassis specifically selected for spare expansion
slots. Would that work for you?

> --
> You received this message because you are subscribed to the Google Groups "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/c3022cd2-2e2d-454d-9847-35933ea34b82n%40googlegroups.com.

[Scott Groel]

To add:

The R7525 machines we have at Clemson are not configured to accept NVMe devices. The storage backplane does not handle NVMe devices, and with the GPU configuration in this system, we are limited with what we can install from an expandability standpoint. It's not impossible, but we cannot do it with anything we have on hand.

Robert's solution may be the best for what you are looking to do.

Thanks,

Scott Groel

Clemson University

To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/onn5wc25vmydtg6xjkcisktpsg3t4dx5yhigabps6dhbyddpar%40gpt4zpjaat6n.

[Felix Zhang]

That would be fine. We just need a setup where we have at least 2, ideally 4, nodes where the BF-2 can access an attached SSD over PCIe. Would these 'flex' nodes be the 2U c6525-100g servers mentioned in the manual?

Thanks,
Felix

[Felix Zhang]

Hi,

Sorry for the delayed response. I didn't realize my original reply was automatically deleted.

For our experiments, we just need 2 (ideally 4) nodes each with a BF-2 that can access an SSD over PCIe. Are the 'flex nodes' the 2U c6525-100g boxes mentioned in the manual? If so, I think equipping some of those nodes with BF-2s would work.

Regards,
Felix

 

On Monday, November 10, 2025 at 2:56:56 PM UTC-5 sgr...@g.clemson.edu wrote:

[ajma...@gmail.com]

Hi Felix,

The 'flex' nodes are either the d750, d760, d760-hbm, or d7615 nodes at CloudLab Utah.  These are 2U servers with extra available PCIe slots as well as mixed-use NVMe Gen 5 drives (except for the d750s which have Gen 4).  The c6525-100g nodes are not 2U, they are 1U half-width sleds (four to a 2U chassis) and would not fit BF2s due to their various limitations.  Two nodes at CloudLab Utah, flex07 and flex08 (AMD-based d7615 nodes) already have BF2s in them.  I bought a couple of them secondhand and re-flashed them to enable both 100Gb ports, and while that part was successful, I never got around to verifying whether the BF2s worked at all or not.  If you're fine with AMD CPUs you can allocate those two nodes and play around with them, otherwise I would have to move them elsewhere.  If you need more than two nodes, we would have to figure out whether we could reasonably get more BF2s or not.

Let me know if you have any other questions.

Best,

 - Aleks

[Felix Zhang]

HI Aleks,

Thanks for the information. I'll take some time to play around with the two nodes you mentioned. Is there a way for me to directly specify nodes "flex07" and "flex08" in a resource reservation request, or do I request two d7615 nodes and then they get assigned to my reservation? If you could also look into getting two more nodes with BF-2s and PCIe-accessible drives that would be great. I need them to test the scalability of my system.

Warm regards,

Felix

[Mike Hibler]

Unfortunately, there is not a way at the moment to reserve specific nodes.
You can allocate specific nodes by setting

node = request.RawPC()
...
node.component_id = "flex07"

in your profile genilib script. Right now those two nodes are in use but
I am working to free them up so that you can allocate them.

> onn5wc25vmydtg6xjkcisktpsg3t4dx5yhigabps6dhbyddpar%40gpt4zpjaat6n.

>
>  
>
> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/

> 34782632-2a42-4228-b20f-db73b97d4df9n%40googlegroups.com.