RFC: Underlays (solves a complementary problem to bind-dirs)

[Zaz Brown]

bind-dirs allows an AppVM to persist changes outside user directories.
But what if a TVM wants to persist changes inside user directories?

I'm seeking feedback on a system I set up to solve this problem: Is
there a better way to achieve this goal? Or on the contrary, is this
something that might be useful to merge in to Qubes?


Motivation
==========

These days, a lot of software is installed in the home directory and
updated outside of the package manager. We want a way to share those
packages across all AppVMs of a TVM.

E.g. say you want to install Doom Emacs, configure it, then make it
available to all AppVMs of a template. This would require the TVM to:

1. Make /home/user/.config/{doom,emacs} available in AppVMs
2. (ideally) allow AppVMs to store persistent changes to these to allow
customization on a per-VM basis


Solution
========

The solution I came up with was to have a bash script read
a file /etc/underlay/underlay.list and, for each item:

1. mount /etc/underlay/item to /home/user/item, read-only
2. mount /home/user/.overlay/item over that mount, read-write

So the files come from the TVM, but any changes are written (CoW) to the
overlay in /home/user/.overlay where they are persisted.

The procedure for updating would be to perform the update in a dispVM
based on the TVM, then copy the update to the TVM (/etc/underlay/...).


Implementation
==============

In the TVM, create:

/usr/bin/mount-underlays.sh

#!/bin/bash

# Only run in AppVMs
if [ ! -f /var/run/qubes/this-is-appvm ]; then
echo "Error: This should only be run in AppVMs" >&2
exit 8
fi

DEST="/home/user"
UNDERLAY_BASE="/etc/underlay"
OVERLAY_BASE="$DEST/.overlay"
WORKDIR_BASE="$DEST/.cache/overlay-workdir"
SYSTEM_LIST="$UNDERLAY_BASE/underlay.list"
USER_LIST="$OVERLAY_BASE/underlay.list"

# Function to mount a directory with overlay
mount_overlay() {
local rel_path="$1"
local lower_dir="$UNDERLAY_BASE/$rel_path"
local mount_point="$DEST/$rel_path"
local upper_dir="$OVERLAY_BASE/$rel_path"
local work_dir="$WORKDIR_BASE/$rel_path"

# Check if lower directory exists
if [ ! -d "$lower_dir" ]; then
echo "Warning: underlay directory $lower_dir does not exist, skipping" >&2
return 1
fi

# Create necessary directories
mkdir -p "$mount_point" "$upper_dir" "$work_dir"

# Check if already mounted
if mountpoint -q "$mount_point" 2>/dev/null; then
echo "Already mounted: $mount_point" >&2
return 0
fi

# Mount overlay filesystem
if sudo mount -t overlay overlay \
-o lowerdir="$lower_dir",upperdir="$upper_dir",workdir="$work_dir" \
"$mount_point" 2>/dev/null; then
echo "Mounted: $rel_path at $mount_point" >&2
else
echo "Failed to mount overlay for $rel_path" >&2
return 1
fi
}

# Function to process a list file
process_list() {
local list_file="$1"

if [ ! -f "$list_file" ]; then
# list file does not exist
return 0
fi

echo "Processing: $list_file" >&2

while IFS= read -r line; do
# Skip comments and empty lines
# / at the start of a line denotes a comment
[[ "$line" == /* ]] && continue
[[ -z "$line" ]] && continue

# Mount the overlay
mount_overlay "$line"
done < "$list_file"
}

# Only proceed if the underlay base exists
if [ ! -d "$UNDERLAY_BASE" ]; then
echo "Error: Underlay base ($UNDERLAY_BASE) does not exist" >&2
exit 3
fi

process_list "$SYSTEM_LIST"
process_list "$USER_LIST"


/etc/systemd/system/mount-underlays.service

[Unit]
Description=Mount underlay directories
After=local-fs.target
ConditionPathExists=/var/run/qubes/this-is-appvm
ConditionPathExists=/etc/underlay

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=-/usr/bin/mount-underlays.sh
StandardOutput=journal
StandardError=journal
TimeoutSec=12

[Install]
WantedBy=default.target


Example Usage
=============

Example with Doom Emacs:

In the TVM:

1. In the TVM, create /etc/underlays/.config/{doom,emacs}

2. Copy your doom and emacs directories there. In the AppVM:
tar -cf emacs.tar.gz ~/.config/emacs
qvm-copy emacs.tar.gz

3. In the TVM, create /etc/underlays/underlays.list

.config/doom
.config/emacs

4. Power off the TVM.

5. Start a new disposable based on the TVM. In that, run:
systemctl start mount-underlays
emacs

--
Thanks for reading,

Zaz

[unman]

On Sat, Sep 20, 2025 at 10:52:45AM +0000, 'Zaz Brown' via qubes-devel wrote:
>
> bind-dirs allows an AppVM to persist changes outside user directories.
> But what if a TVM wants to persist changes inside user directories?
>
> I'm seeking feedback on a system I set up to solve this problem: Is there a better way to achieve this goal? Or on the contrary, is this something that might be useful to merge in to Qubes?

In the cases where I have wanted to do this:

I Set the required changes in /etc/skel in the template.
Every qube created from that template will have the set up and further
changes will persist in each qube.
Of course, this requires some forethought and planning.

Where I want to do the same for *existing* qubes, I salt the change in
to each qube using a targeted qubesctl.

I dont know if this is better, but it is simple and it works within the
existing framework. (And, of course, it's self documenting.)

[Zaz Brown]

Thank you, unman.

So you copy the directory to dom0 and then use a salt script to copy to
all relevant VMs? Is there any existing solution that doesn't duplicate
data like this? Because in the case of Emacs, it can be a few GB, in the
case of others it could be larger (e.g. pip packages).

While Linux's `overlay` adds a little complexity, I think the core of my
solution is even simpler than SALT: if you want data shared between all
AppVMs, you put it in the TVM.

The script I shared just automates `overlay` this so that
/etc/underlays/ works similarly to /etc/skel/: Just drop your files in
/etc/underlays/ and add an appropriate entry in
/etc/underlays/underlays.list.

--

[unman]

On Sun, Sep 21, 2025 at 03:18:39PM +0000, Zaz Brown wrote:
[quote]

So you copy the directory to dom0 and then use a salt script to copy to
all relevant VMs? Is there any existing solution that doesn't duplicate
data like this? Because in the case of Emacs, it can be a few GB, in the
case of others it could be larger (e.g. pip packages).

[/quote]
Yes, duplication of data can be an issue, but in most of the cases that
I use, the bare configuration is relatively small. In any case, storage
is cheap, as they say.

[quote]

While Linux's `overlay` adds a little complexity, I think the core of my
solution is even simpler than SALT: if you want data shared between all
AppVMs, you put it in the TVM.

[/quote]
Yes, I understand this, and it is a good use case. I'll have to mull
over the code (which is why I didnt comment on it the first time), and
I'm too tired to do it now.

--
I never presume to speak for the Qubes team.
When I comment in the mailing lists I speak for myself.

[qubist]

Hi Zaz,

I also use /etc/skel in the template for certain things.

Example:

I create /etc/skel/.bash_customizations and append this like to
/etc/skel/.bashrc:

[ -f "${HOME}/.bash_customizations" ] && . "${HOME}/.bash_customizations"

So, anything which must be universal to all AppVMs is customized
through the template in /etc/skel/.bash_customizations. Then, each
AppVM can further customize through "${HOME}/.bash_customizations".

Another possible approach might be to use /usr/local/etc for
AppVM-specific config, however the particular app must be made to
support that.

Could that work for you?

[Zaz Brown]

I thought /etc/skel/ copies its files to the VM only when the VM is
first created? After that, if you make updates to files in /etc/skel/,
they are not propagated to the AppVMs, right?

Furthermore, this duplicates storage. So if you wanted to share, for
example, a large program installed via pip or some other non-system
method, to all AppVMs, this could take up a lot of space, and keeping it
up to date across all AppVMs could be a pain.

> Another possible approach might be to use /usr/local/etc for
> AppVM-specific config, however the particular app must be made to
> support that.

Isn't this solving a different problem? Per-AppVM-specific config rather
than config shared across all AppVMs?

--
Thanks for your response,

Zaz

[qubist]

On Mon, 29 Sep 2025 09:57:10 +0000 'Zaz Brown' via qubes-devel wrote:

> I thought /etc/skel/ copies its files to the VM only when the VM is
> first created? After that, if you make updates to files in /etc/skel/,
> they are not propagated to the AppVMs, right?

That's my understanding too.

> Furthermore, this duplicates storage. So if you wanted to share, for
> example, a large program installed via pip or some other non-system
> method, to all AppVMs, this could take up a lot of space, and keeping it
> up to date across all AppVMs could be a pain.

If you want it in all AppVMs, up-to-date and always-synced, the answer
is pretty straightforward: Install it in the template. Even if the
particular installation method installs in /rw, you can still move the
destination to e.g. /root and symlink to it (in the template and in the
AppVMs as well), so nothing will need to be duplicated at any time.

> > Another possible approach might be to use /usr/local/etc for
> > AppVM-specific config, however the particular app must be made to
> > support that.
>
> Isn't this solving a different problem? Per-AppVM-specific config rather
> than config shared across all AppVMs?

I thought you were trying to solve both. In case /usr/local/etc has no
config, the app will look upstream (in /etc).