How do I mount my GCP bucket into an SLURM instance?

[GIOVANNI QUINONES VALDEZ]

Hi all, 

I created a bucket where I store my data and scripts, mainly because I can mount the bucket into my local computer and edit/view/write files easily from there. 

I want to submit a job array into the SLURM instance. These jobs run the scripts from the bucket and use some of those files as arguments as well. So, I mounted the bucket (with gcsfuse) into the "controller" instance of the SLURM. However, the bucket folder does not appear in the "login" instance (it appears empty).

I was hesitant to directly mount the SLURM VMs because I thought the storage bucket would be more suitable (and cheaper) for almost 1 Tb of data. My questions, more specifically, is:

- Is it at all possible to mount a bucket to the SLURM VMs? (and share the files between the "controller" and "login")

- Is there a different setup that will accomplish the same goal? As in, being able to edit/write files from my local computer and submit them in job arrays? 

I appreciate any pointers! Thanks for your help! I happy to provide any more details if needed. 

- Gio

 

[Derek Murphy]

Hello,

I have just looked into this issue briefly and so far I can only come up with the following statements:

Key differences from a POSIX file system : 

https://cloud.google.com/storage/docs/gcs-fuse#notes

“Directories: By default, only directories that are explicitly defined (that is, they are their own object in Cloud Storage) will appear in the file system. Implicit directories (that is, ones that are only parts of the pathname of other files or directories) will not appear by default. If there are files whose pathname contain an implicit directory, they will not appear in the overall directory tree (since the implicit directory containing them does not appear). A flag is available to change this behavior. For more information, see the semantics documentation. [1]”

Mounting : 

https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/mounting.md#mounting

“Important: You should run gcsfuse as the user who will be using the file system, not as root. Similarly, the directory should be owned by that user. Do not use sudo for either of the steps above or you will wind up with permissions issues.”

Please let me know if this is of any help.

from : Derek

[1] Files and directories :

https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/semantics.md#files-and-dirs

[GIOVANNI QUINONES VALDEZ]

Hi, thanks for getting back!

Yes, I used the implicit--dirs flag when mounting on the "controller" VM of the slurm and I am able to see the entire tree structure of the directories. The issue is that I don't see anything on the "login" VM not even the files on the main directory. 

I did make sure I mount as a user and try to access the files as user too. (as in gid uid in both controller and login is the same).  

Do you have any other suggestion? 

Thanks a lot!

[GIOVANNI QUINONES VALDEZ]

I found a similar question asked on StackOverflow https://stackoverflow.com/questions/61652004/slurm-cluster-in-google-cloud-data-in-mounted-directory-in-controller-login-nod 

I think the person asking is trying to do the same thing as me. Unfortunately the answer is not very detailed and I am not able to ask for further explanation. 

Do you think maybe modifying some parameters in the yaml file (during deployment of SLURM) would help?

Thanks! 

[Derek Murphy]

Hello,

I have been looking for more documentation on mounting buckets to a SLURM instance without much success as of yet. I do believe that your second question, whether there is a system in place that can give the same functionality, can be answered. In these links [1][2], it can be seen that Google suggests the use of Cloud Filestore. You may want to look into this to see if it is a viable option for your use case.

Let us know if this helped.

[1] HPC made easy: Announcing new features for Slurm on GCP :  

https://cloud.google.com/blog/products/compute/hpc-made-easy-announcing-new-features-for-slurm-on-gcp

[2] Filestore : 

https://cloud.google.com/filestore