Hi Sidd,
The controller node has the disk volume that gets served out via NFS to the compute nodes. I recommend you make the controller node "large enough" that it can handle all the I/O from all the compute nodes. It's standard NFS.
The login node is separate so that users can't interfere with the controller node functions (nfs server, slurmctld, etc). You can make the login node fairly small.
You can have 0 compute nodes by default and all compute nodes will be on-demand.
There is a "persistent disk" on each compute node, just imagine a standard physical HPC cluster, the compute nodes all have local OS disk. The k8s concepts don't really apply here.
In terms of pricing, IME, the most practical way to determine it is through trial and error, run a standard test workload and then check the billing results a day later. And then you know your real total costs. It will not be cheap.
Regards,
Alex