Experience with DeepOps?

[Jan Kreps]

Hello everyone,

I operate a small cluster based on warewulf stateless provision and CentOS 7. Images for warewulf are based on OpenHPC and composed by Ansible playbooks. 

Now we have bought some Nvidia DGX-1 servers and I am exploring their DeepOps tools. 

Does anybody here have an experience with transition from the warewulf to DeepOps, or with some combination of both approaches? The concepts seems quite different. 

Thanks Jan

[Adam DeConinck]

Hi Jan,

I’m currently one of the lead developers for DeepOps, and I used to work quite a bit with Warewulf. So while I haven’t done any direct work on transitioning from one to the other, I have a decent grasp on the philosophical differences. :)

The focus in DeepOps is very much on correctly configuring a cluster which has been statefully provisioned in some manner. It works well with a collection of DGX hosts with DGX OS installed, or with some set of GPU servers running Ubuntu or CentOS.

This doesn’t (necessarily!) work well with Warewulf’s approach of building images in advance to be provisioned on the nodes, either in RAM or on disk. There are a number of components in DeepOps which assume you are running on the live node.

I have previously done some work where I captured an “image” of a DeepOps-managed node into a form that could be provisioned in the future (e.g. as a VNFS). However, there’s some work to do to ensure the image you capture is generic enough to be configured when provisioned in the future. I haven’t tried this with Warewulf but it seems possible — albeit not a direct “out of the box” workflow.

DeepOps is also pretty modular, so you could pretty easily use it as a “post-provisioning” step to finish setting up a node after it was deployed. Or simply borrow the useful bits of the Ansible playbooks for your own workflow.

Hope this helps!
Adam

[Jan Kreps]

Hi Adam,

thanks for your detailed reply. Probably we will do it gradually, moving nodes from Warewulf to DeepOps cluster one by one. I intend to share our experience when we will finish the transition.

Best Regards,

Jan