Hi,
I'm seeing:
*** Slurm is currently being configured in the background. ***
forever during startup.
So I tried sudo journalctl -o cat -u google-startup-scripts
Here's the output:
2021/04/29 11:15:13 GCEMetadataScripts: Starting startup scripts (version 20210223.01).
2021/04/29 11:15:13 GCEMetadataScripts: Found startup-script in metadata.
2021/04/29 11:15:13 GCEMetadataScripts: startup-script: running python cluster setup script
2021/04/29 11:15:14 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Set up network storage
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/usr/local/etc/slurm to
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/etc/munge to /etc/munge
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/home to /home
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/apps to /apps
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /usr/local/etc/slurm to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /usr/local/etc/slurm
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /etc/munge to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /etc/munge
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /home to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /home
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /apps to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /apps
2021/04/29 11:15:19 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:25 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /apps to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /apps
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /usr/local/etc/slurm to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /usr/local/etc/slurm
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /etc/munge to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /etc/munge
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /home to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /home
2021/04/29 11:17:30 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
...skipping...
Starting Google Compute Engine Startup Scripts...
2021/04/29 11:15:13 GCEMetadataScripts: Starting startup scripts (version 20210223.01).
2021/04/29 11:15:13 GCEMetadataScripts: Found startup-script in metadata.
2021/04/29 11:15:13 GCEMetadataScripts: startup-script: running python cluster setup script
2021/04/29 11:15:14 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Set up network storage
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/usr/local/etc/slurm to
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/etc/munge to /etc/munge
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/home to /home
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Setting up mount (nfs) nc-slurm-controller:/apps to /apps
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /usr/local/etc/slurm to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /usr/local/etc/slurm
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /etc/munge to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /etc/munge
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /home to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /home
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: Waiting for /apps to be mounted
2021/04/29 11:15:18 GCEMetadataScripts: startup-script: run: mount /apps
2021/04/29 11:15:19 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:24 GCEMetadataScripts: startup-script: mount.nfs: Connection refused
2021/04/29 11:17:25 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /apps to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /apps
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /usr/local/etc/slurm to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /usr/local/etc/slurm
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /etc/munge to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /etc/munge
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: Waiting for /home to be mounted
2021/04/29 11:17:29 GCEMetadataScripts: startup-script: run: mount /home
2021/04/29 11:17:30 logging client: rpc error: code = PermissionDenied desc = The caller does not have permission
Can someone help me understand exactly what permission is missing here?
Also, perhaps it's best to document the exact list of minimally required permissions (for both the user account and the service account) somewhere so that people won't have to spend time debugging these permission issues :)
Thanks