Speed improvements when baking data into a custom SLURM image

[Bo Langgaard Lind]

After some fighting, I have successfully built a custom image for my SLURM instances, based on Rocky Linux 8. On this image, I have also added a roughly 12GB python environment that our jobs use.

Previously, this environment was served by the controller, via NFS, which was a clear bottleneck when deploying hundreds of worker instances.

Does anyone have experience with baking data into the image, and does that improve performance? Who serves the images, and what is the bottleneck?

I'll give things a try, but I have a few more components to build before I can launch a test.

[Joseph Schoonover]

Hey Bo,

Great work on getting this far - we've had this solution up for some time in marketplace ( https://console.cloud.google.com/marketplace/product/fluid-cluster-ops/rcc-rocky ); usage of this solution or its images comes with support.

In our experience, writing data used by your applications to disk to be accessible on compute nodes can have its benefits. With that design, compute nodes don't have to deal with contending for attention from an NFS server or other file storage solution. In terms of "who" serves the image; you do. If you bake the image in your project, you pay the costs for storing the image, in addition to the regular upkeep and maintenance (OS image updates, data updates,etc). 

How much data are you talking about though ? This impacts the image size and the minimum disk size for your cluster's VMs.

Dr. Joseph Schoonover

Owner, Research Software Engineer

j...@fluidnumerics.com

--
You received this message because you are subscribed to the Google Groups "google-cloud-slurm-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-cloud-slurm-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-cloud-slurm-discuss/65883e94-ce35-41a1-b623-3939ad171a15n%40googlegroups.com.