Using Nix on a cluster with shared homedirs

[Keshav Kini]

Hi,

I and some of my coworkers would like to use Nix at work, and I was
hoping for some advice on what configuration would make the most
sense.

We have a server cluster with many different machines. The machines
all run the same RHEL-like Linux distribution but sometimes with
different releases or with different libraries / software installed.
However, users' home directories are universally shared across all of
our machines. (That is, no matter what machine an individual user
logs into, their $HOME is the same NFS-mounted directory with the same
contents as they would see on any other of the machines.) There are
also other NFS volumes that are automounted at consistent paths
throughout the cluster.

It seems like the best way to set up Nix, given the universal home
directory setup and the fact that we often work on various different
machines rather than a single personal machine, is to create an NFS
volume that can be mounted at /nix/ and set up a multi-user Nix
installation on that volume.

Does this sound reasonable and/or is this what people usually do in
such a situation? I'm a relative newbie at Nix, so does anyone have
any tips or caveats they'd like to point out about what I'm
considering? For example, will locking work properly on the Nix store
when it's on an NFS volume shared by multiple machines? (I.e. what if
Nix builders on two different machines try to build the same
derivation at the same time?) Are there file permission issues / how
would this setup interact with the root_squash issue on NFS? Etc.

Thanks,
Keshav

[Dmitry Kalinkin]

Hi,

This was discussed at least once before [1]. The main problem is with
the store metadata stored in a Sqlite database. On NFS you might get
an error about setting the journalling mode. The workaround is to set
"use-sqlite-wal = false" in your nix.conf. In my experience, trying to
do changes to the store from multiple machines simultaneously will
quickly break that database. It should, however, work fine if you only
write from one place at a time.

Nix is used in some HPC setups at Compute Canada [3] and GRICAD [2]. I
could not find how exactly they implemented it, but maybe you could
ask them directly (don't forget to let us know what they tell you). I
would imagine they export the nix store via some read-only network
filesystem and allow communication to some central build node running
the standard nix daemon via, perhaps, some socket forwarding across
the network.

Best,
Dmitry

[1] https://nixos.org/nix-dev/2016-February/019463.html
[2] https://dl.acm.org/citation.cfm?id=3152556
[3] https://fosdem.org/2018/schedule/event/computecanada/attachments/slides/2480/export/events/attachments/computecanada/slides/2480/CVMFSNixLmodEBCC.pdf

> --
> You received this message because you are subscribed to the Google Groups "nix-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nix-devel+...@googlegroups.com.
> To post to this group, send email to nix-...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/nix-devel/CALBVBvy6akPYxhuDpe8UYOkuVaoVYcem%2BdO%2Bakb_yeTrpfhTkg%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[zimbatm]

Hi,

with Nix 2.0 it's possible to change the location of the store without having to re-compile everything. Eg:

    $ nix run --store $HOME/nix nixpkgs.hello -c hello
    [2 copied (20.4 MiB), 5.5 MiB DL]
    Hello, world!

    $ ls ~/nix/nix
    store/ var/

With individual users each having their own store, the SQLite corruption is much less likely.

One thing that I don't know yet is how to make the `--store` option sticky so you don't have to pass it to each invocations of nix. If you find out, please let me know :)

To view this discussion on the web visit https://groups.google.com/d/msgid/nix-devel/CAM41TOv5Dv4ZAhSxud1s43OW8jNBu-ucydTAx7yOrFoQxqs6Nw%40mail.gmail.com.

[Tuomas Tynkkynen]

On Wednesday, April 4, 2018 at 3:26:03 PM UTC+3, zimbatm wrote:

One thing that I don't know yet is how to make the `--store` option sticky so you don't have to pass it to each invocations of nix. If you find out, please let me know :)

I haven't tried this, but I believe in Nix 2.0 many more of these 'global' options can be set in in nix.conf. So maybe exporting NIX_CONF_DIR set to some user-specified location and putting 'store = /home/foo/nix' there. 

[Will Dietz]

Per-user config can be set in

~/.config/nix/nix.conf

(err $XDG_CONFIG_HOME/nix/nix.conf)

Where you can set "store = /home/user/nix" or so.

~Will

--

You received this message because you are subscribed to the Google Groups "nix-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nix-devel+...@googlegroups.com.
To post to this group, send email to nix-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nix-devel/a024a51a-14d8-496d-953b-a399633c768c%40googlegroups.com.