Setting up bindfs mount in LXC container

[Richard Hector]

Hi all,

I'm using bindfs in my web LXC containers to allow particular users to
write to their site docroot as the correct user.

Getting this to work has been really hacky, and while it does seem to
work, I get log messages saying it didn't ...

In /var/lib/lxc/<containername>/config:
========================================================================
lxc.hook.start-host = /usr/local/bin/fuse.hook
========================================================================

In /usr/local/bin/fuse.hook:
========================================================================
#!/bin/bash
at now + 1 minute <<END 2>>/var/log/lxc/${LXC_NAME}-hook-error.log
/usr/local/bin/fuse.hook.s2
END
========================================================================

In /usr/local/bin/fuse.hook.s2:
========================================================================
lxc-device -n ${LXC_NAME} add /dev/fuse
lxc-attach -n ${LXC_NAME} /usr/local/bin/bindfs_mount
========================================================================

In /usr/local/bin/bindfs_mount (in the container):
========================================================================
#!/bin/bash
file='/usr/local/etc/bindfs_mounts'
while read line; do
mount "${line}"
done < "${file}"
========================================================================

In /usr/local/etc/bindfs_mounts (in the container):
========================================================================
/home/richard/<sitename>/doc_root
========================================================================

In /etc/fstab (in the container) (single line wrapped by MUA):
========================================================================
/srv/<sitename>/doc_root /home/richard/<sitename>/doc_root fuse.bindfs
noauto,--force-user=richard,--force-group=richard,--create-for-user=<sitename>,--create-for-group=<sitename>
0 0
========================================================================

I'm sure shell experts (or LXC experts) will tell me this 2-stage
process is unnecessary, or that there is a better way to do it, but IIRC
it doesn't work if lxc is waiting for the hook to finish; other stuff
needs to happen before the device creation works.


At boot, however, I get these messages emailed from the at job (3 lines,
wrapped by MUA):
========================================================================
lxc-device: <containername>: commands.c: lxc_cmd_add_bpf_device_cgroup:
1185 Message too long - Failed to add new bpf device cgroup rule
lxc-device: <containername>: lxccontainer.c: add_remove_device_node:
4657 set_cgroup_item failed while adding the device node
lxc-device: <containername>: tools/lxc_device.c: main: 153 Failed to add
/dev/fuse to <containername>
========================================================================

The device file is created correctly, and the mount work.

Oh - and interestingly, this only seems to happen when the host boots.
If I just reboot (or shutdown and start) the container, it works fine.

It doesn't matter if I increase the delay on the at job.

If I don't use the at job, but run those commands manually after boot,
it works fine with no error messages.

Any hints?

I suspect my limited understanding of cgroups is contributing to my
problems ...

Cheers,
Richard

[Max Nikulin]

On 17/01/2023 04:06, Richard Hector wrote:
>
> I'm using bindfs in my web LXC containers to allow particular users to
> write to their site docroot as the correct user.

I am not familiar with bindfs, so I may miss something important for
your use case.

First of all I am unsure why you prefer bindfs instead of mapping some
container users to host users using namespaces. With the following
configuration 1000 inside a container and on the host is the same UID:

lxc.idmap = u 0 100000 1000
lxc.idmap = u 1000 1000 1
lxc.idmap = u 1001 101001 64535
lxc.idmap = g 0 100000 1000
lxc.idmap = g 1000 1000 1
lxc.idmap = g 1001 101001 64535

lxc.mount.entry = /home/richard/sitename/doc_root /srv/sitename/doc_root
none bind,optional,create=dir

> In /usr/local/bin/fuse.hook:

I would look into lxcfs hook for inspiration

> In /usr/local/bin/fuse.hook.s2:
> ========================================================================
> lxc-device -n ${LXC_NAME} add /dev/fuse

Is there any reason why it can not be done using lxc.mount.entry in the
container config?

> lxc-attach -n ${LXC_NAME} /usr/local/bin/bindfs_mount

I would consider adding a systemd unit inside container. Unsure if could
be done using an udev rule.

> If I don't use the at job, but run those commands manually after boot,
> it works fine with no error messages.

Unsure if it is relevant, but it is better to run lxc-start and
lxc-attach as a systemd unit with Delegate=yes configuration, either a
temporary one (systemd-run) or configured as a service. It ensures
proper cgroup and scope. Otherwise some cryptic errors may happen.

[Max Nikulin]

On 17/01/2023 17:52, Max Nikulin wrote:
> lxc.mount.entry = /home/richard/sitename/doc_root /srv/sitename/doc_root
> none bind,optional,create=dir

Sorry, path inside the container should be without the leading slash.

lxc.mount.entry = /home/richard/sitename/doc_root srv/sitename/doc_root
none bind,optional,create=dir

[Richard Hector]

On 17/01/23 23:52, Max Nikulin wrote:
> On 17/01/2023 04:06, Richard Hector wrote:
>>
>> I'm using bindfs in my web LXC containers to allow particular users to
>> write to their site docroot as the correct user.
>
> I am not familiar with bindfs, so I may miss something important for
> your use case.
>
> First of all I am unsure why you prefer bindfs instead of mapping some
> container users to host users using namespaces. With the following
> configuration 1000 inside a container and on the host is the same UID:
>
> lxc.idmap = u 0 100000 1000
> lxc.idmap = u 1000 1000 1
> lxc.idmap = u 1001 101001 64535
> lxc.idmap = g 0 100000 1000
> lxc.idmap = g 1000 1000 1
> lxc.idmap = g 1001 101001 64535
>
> lxc.mount.entry = /home/richard/sitename/doc_root /srv/sitename/doc_root
> none bind,optional,create=dir

Disclaimer - I haven't actually tried any of your suggestions yet.

My goal is not to map container users to host users, but to allow a
container user (human user) to access a directory as another container
user (non-human owner of files). This should also be doable for multiple
human users for the same site.

>> In /usr/local/bin/fuse.hook:
>
> I would look into lxcfs hook for inspiration

Interesting; will do. Not sure exactly where to start, but will get there.

>> In /usr/local/bin/fuse.hook.s2:
>> ========================================================================
>> lxc-device -n ${LXC_NAME} add /dev/fuse
>
> Is there any reason why it can not be done using lxc.mount.entry in the
> container config?

Is that usable for adding a device file? The only way I found to do that
is using lxc-device from outside the container. mknod inside doesn't work.

>> lxc-attach -n ${LXC_NAME} /usr/local/bin/bindfs_mount
>
> I would consider adding a systemd unit inside container. Unsure if could
> be done using an udev rule.

That might be better, but it does need to rely on the device existing first.

>> If I don't use the at job, but run those commands manually after boot,
>> it works fine with no error messages.
>
> Unsure if it is relevant, but it is better to run lxc-start and
> lxc-attach as a systemd unit with Delegate=yes configuration, either a
> temporary one (systemd-run) or configured as a service. It ensures
> proper cgroup and scope. Otherwise some cryptic errors may happen.

So even for running stuff manually, run it from systemd? Interesting,
will investigate further. I wasn't aware of systemd-run.

Thanks,
Richard

[Max Nikulin]

On 18/01/2023 03:52, Richard Hector wrote:
> On 17/01/23 23:52, Max Nikulin wrote:
>>
>> lxc.idmap = u 0 100000 1000
>> lxc.idmap = u 1000 1000 1
>>

>> lxc.mount.entry = /home/richard/sitename/doc_root
>> srv/sitename/doc_root none bind,optional,create=dir
>

> My goal is not to map container users to host users, but to allow a
> container user (human user) to access a directory as another container
> user (non-human owner of files). This should also be doable for multiple
> human users for the same site.

Do you mean mapping several users (human and service ones) from a single
container to the same host UID? The approach I suggested works for 1:1
mapping. Another technique is group permissions and ACLs, but I would
not call it straightforward. A user may create a file that belongs to
wrong group or inaccessible by another user.

>> I would look into lxcfs hook for inspiration
>
> Interesting; will do. Not sure exactly where to start, but will get there.

/usr/share/lxcfs/lxc.mount.hook (lxcfs package)

>>> lxc-device -n ${LXC_NAME} add /dev/fuse
>>
>> Is there any reason why it can not be done using lxc.mount.entry in
>> the container config?
>
> Is that usable for adding a device file? The only way I found to do that
> is using lxc-device from outside the container. mknod inside doesn't work.

I use it to pass /dev/dri and /dev/video0 to a container. The only issue
is that the webcam is a USB device, so permissions become wrong after
suspend to RAM and the following resume.

>>> lxc-attach -n ${LXC_NAME} /usr/local/bin/bindfs_mount
>>
>> I would consider adding a systemd unit inside container. Unsure if
>> could be done using an udev rule.
>
> That might be better, but it does need to rely on the device existing
> first.

I see no difference since in your approach device presence is ensured by
a hook. With a lxc.mount.entry it should be even more reliable.

> So even for running stuff manually, run it from systemd? Interesting,
> will investigate further. I wasn't aware of systemd-run.

I use unprivileged containers to experiment with applications packaged
for other linux distributions or just in relatively isolated
environment. My experience that even after searching for particular
errors it is not obvious what should be done to fix it.

Delegate=yes is mandatory for lxc-start. To attach:

systemd-run --user --scope --property=Delegate=yes -- \
lxc-attach --clear-env --keep-var TERM ...

At first I tried --pty instead of --scope for interactive sessions, but
I noticed some issues as well.

-
https://discuss.linuxcontainers.org/t/pam-cgfs-how-to-setup-with-cgroup-v2/10741/
"Pam_cgfs How to setup with cgroup v2?"
- https://github.com/lxc/lxc/issues/3221#issuecomment-648829685
"Unable to start an unprivileged container on fresh install of Fedora
31 · Issue #3221 · lxc/lxc"

[Richard Hector]

On 18/01/23 16:38, Max Nikulin wrote:
> On 18/01/2023 03:52, Richard Hector wrote:
>> On 17/01/23 23:52, Max Nikulin wrote:
>>>
>>> lxc.idmap = u 0 100000 1000
>>> lxc.idmap = u 1000 1000 1
>>>
>>> lxc.mount.entry = /home/richard/sitename/doc_root
>>> srv/sitename/doc_root none bind,optional,create=dir
>>
>> My goal is not to map container users to host users, but to allow a
>> container user (human user) to access a directory as another container
>> user (non-human owner of files). This should also be doable for
>> multiple human users for the same site.
>
> Do you mean mapping several users (human and service ones) from a single
> container to the same host UID? The approach I suggested works for 1:1
> mapping. Another technique is group permissions and ACLs, but I would
> not call it straightforward. A user may create a file that belongs to
> wrong group or inaccessible by another user.

I'll use more detail :-)

I have a Wordpress site. The directory /srv/sitename/doc_root, and most
of the directories under it, are owned by user 'sitename'.

PHP runs as 'sitename-run', which has access (via group 'sitename') to
read all of that, but not write it. Some subdirectories, eg
.../doc_root/wp-content/uploads, are group-writeable so that it can save
things there.

An authorised site maintainer, eg me ('richard') (but there may be any
number of others), needs to be able to write under /srv/sitename, so I
use bindfs to mount /srv/sitename under /home/richard/sitename, which
presents it as owned by me, and translates the ownership back to
'sitename' when I write to it. So each human user sees the site as owned
by them, but it's all mapped to 'sitename' on the fly.

These users I guess map to host users, but I'm not particularly
interested in that ... actually I should care more, because it actually
maps to a real but unrelated user id on the host, which could have bad
implications - but I think that's a separate issue.

I'm not ignoring the rest of your message; I'll look at that separately :-)

Cheers,
Richard

[Max Nikulin]

On 18/01/2023 13:02, Richard Hector wrote:
> I have a Wordpress site. The directory /srv/sitename/doc_root, and most
> of the directories under it, are owned by user 'sitename'.
>
> PHP runs as 'sitename-run', which has access (via group 'sitename') to
> read all of that, but not write it. Some subdirectories, eg
> .../doc_root/wp-content/uploads, are group-writeable so that it can save
> things there.
>
> An authorised site maintainer, eg me ('richard') (but there may be any
> number of others), needs to be able to write under /srv/sitename,

Just an idea (likely should be applied recursively)

setfacl -m user:richard:rwx /srv/sitename
setfacl -m default:user:richard:rwx /srv/sitename

the same for PHP read-only access. However likely it is impossible to
set default: x for directories only. Another issue is that a directory
moved from outside will not inherit default ACL. So despite it looks
like standard problem that may be solved by ACLs I do not have a
complete recipe.

Are there users that have access to container only using ssh? If you
have access to host, you can run any command to access filesystem
outside of container using lxc-usernsexec (perhaps with modified uidmap)
or inside container using lxc-attach ... su --login ...

If have not read it yet, some general LXC ideas are described in
https://stgraber.org/2013/12/20/lxc-1-0-blog-post-series/