Trouble running DAOS with NVMe on sm110p nodes (VMD vs direct NVMe exposure)

[Yuan Liang]

Hello,

I’m a beginner trying to learn the DAOS storage system on CloudLab, using the Rocky Linux 9 image on sm110p nodes. My goal is to run DAOS with NVMe so I can perform benchmark tests (e.g., IO500).

I understand that DAOS normally requires IOMMU to be enabled for NVMe, but on CloudLab if I edit GRUB to enable IOMMU the node fails to reboot (goes into shutdown/recovery). So I’ve been trying the recommended workaround of running DAOS without IOMMU, which seems to work only when the node exposes NVMe devices directly.

For example:

On sm110p-10s10615, NVMe drives show up directly (Samsung PM9A1/PM9A3/980PRO), and I was able to get the DAOS server running successfully.

On other nodes such as sm110p-10s10604 and sm110p-10s10617, the drives appear behind a VMD controller, and DAOS fails during server startup because it cannot validate the NVMe devices.

My questions:

Is there a way to use DAOS with NVMe on sm110p nodes that expose drives through VMD, given that I cannot enable IOMMU?

If not, is the only option to keep trying until I get an sm110p node that exposes NVMe directly?

Any advice or clarification would be very helpful. I’m still learning the system, so apologies if I’m missing something obvious.

Thanks in advance!

[Mike Hibler]

We are going to disable VMD on these nodes by default. It seems to only get
in the way of people. Can I reboot your two nodes: sm110p-10s10604 and
sm110p-10s10617? I will fix them now if so.

> --
> You received this message because you are subscribed to the Google Groups
> "cloudlab-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to cloudlab-user...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/cloudlab-users/
> 7f893a61-9579-4e08-b07a-d7adbb2546f4n%40googlegroups.com.

[Yuan Liang]

Hello Mike,

Thank you for the reply, yes you can go ahead and reboot those two nodes to make the change on VMD.