Bind-mounting special devices into a unprivileged container

[Johannes Kastl]

Hi all,

sorry if this is an old hat, but everything that I found on this does
not work...

Setting:
- proxmox host
- lxc 4.0.6
- unprivileged container
- I need the null and zero special devices inside a container

I have an older setup where the container was running privileged, and
I could just use the lxc.cgroup.devices.allow settings.

Apparently this does not work with unprivileged containers.

I thought bind-mounting should do the trick, but it fails:

lxc.mount.entry = /dev/null var/lib/named/dev/null none bind 0 0
lxc.mount.entry = /dev/zero var/lib/named/dev/zero none bind 0

This is the output:

# pct start 102
__safe_mount_beneath_at: 1106 Function not implemented - Failed to
open 51(dev)
mount_entry: 1953 No such file or directory - Failed to mount
"/dev/null" on
"/usr/lib/x86_64-linux-gnu/lxc/rootfs/var/lib/named/dev/null"
lxc_setup: 3353 Failed to setup mount entries
do_start: 1218 Failed to setup container "102"
__sync_wait: 36 An error occurred in another process (expected
sequence number 5)
__lxc_start: 1999 Failed to spawn container "102"
startup for container '102' failed


Not sure if this is proxmox specific, or if I am missing something here?

Any help is highly appreciated, even if it is just pointing out my
mistakes... :-)

Kind Regards,
Johannes

[Serge E. Hallyn]

On Mon, Jun 07, 2021 at 01:09:48PM +0200, Johannes Kastl wrote:
> Hi all,
>
> sorry if this is an old hat, but everything that I found on this does
> not work...
>
> Setting:
> - proxmox host
> - lxc 4.0.6
> - unprivileged container
> - I need the null and zero special devices inside a container
>
> I have an older setup where the container was running privileged, and
> I could just use the lxc.cgroup.devices.allow settings.
>
> Apparently this does not work with unprivileged containers.
>
> I thought bind-mounting should do the trick, but it fails:
>
> lxc.mount.entry = /dev/null var/lib/named/dev/null none bind 0 0

When you don't put a leading '/' in the mount target, you're
asking for the path to be relative to the mounted container rootfs.

So you probably wanted

lxc.mount.entry = /dev/null dev/null none bind 0 0

> lxc.mount.entry = /dev/zero var/lib/named/dev/zero none bind 0
>
> This is the output:
>
> # pct start 102
> __safe_mount_beneath_at: 1106 Function not implemented - Failed to
> open 51(dev)
> mount_entry: 1953 No such file or directory - Failed to mount
> "/dev/null" on
> "/usr/lib/x86_64-linux-gnu/lxc/rootfs/var/lib/named/dev/null"
> lxc_setup: 3353 Failed to setup mount entries
> do_start: 1218 Failed to setup container "102"
> __sync_wait: 36 An error occurred in another process (expected
> sequence number 5)
> __lxc_start: 1999 Failed to spawn container "102"
> startup for container '102' failed
>
>
> Not sure if this is proxmox specific, or if I am missing something here?
>
> Any help is highly appreciated, even if it is just pointing out my
> mistakes... :-)
>
> Kind Regards,
> Johannes
>

> --
> You received this message because you are subscribed to the Google Groups "lxc-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lxc-users+...@lists.linuxcontainers.org.
> To view this discussion on the web visit https://groups.google.com/a/lists.linuxcontainers.org/d/msgid/lxc-users/37fe9dda-d059-378d-b092-8e543f136a21%40ojkastl.de.

pub RSA 4096/AF399D68 2017-02-28 Johannes Kastl <ma...@ojkastl.de>
> sub RSA 4096/1AA68976 2017-02-28
> sub RSA 4096/F1ED4237 2017-02-28
>

[Wolfgang Bumiller]

> On 06/07/2021 1:09 PM Johannes Kastl <ma...@ojkastl.de> wrote:
>
>
> Hi all,
>
> sorry if this is an old hat, but everything that I found on this does
> not work...
>
> Setting:
> - proxmox host
> - lxc 4.0.6
> - unprivileged container
> - I need the null and zero special devices inside a container

Version sounds new enough that you should be able to set the 'mknod' feature flag on PVE

>
> I have an older setup where the container was running privileged, and
> I could just use the lxc.cgroup.devices.allow settings.
>
> Apparently this does not work with unprivileged containers.

You should be able to allow mknod in the container via the Options -> Features settings
in PVE for this.

[Johannes Kastl]

Hi Serge,

On 07.06.21 19:21 Serge E. Hallyn wrote:
> On Mon, Jun 07, 2021 at 01:09:48PM +0200, Johannes Kastl wrote:

>> lxc.mount.entry = /dev/null var/lib/named/dev/null none bind 0 0
>
> When you don't put a leading '/' in the mount target, you're
> asking for the path to be relative to the mounted container rootfs.
>
> So you probably wanted
>
> lxc.mount.entry = /dev/null dev/null none bind 0 0

I wanted the host's /dev/null bind-mounted to /var/lib/named/dev/null
inside the container. So I removed the leading slash, to make the
target be inside the container.

Or did I misunderstand this feature?

Johannes

[Johannes Kastl]

Hi Wolfang,

thanks for the fast answer!

On 08.06.21 09:16 Wolfgang Bumiller wrote:
>> On 06/07/2021 1:09 PM Johannes Kastl <ma...@ojkastl.de> wrote:

> Version sounds new enough that you should be able to set the 'mknod' feature flag on PVE

> You should be able to allow mknod in the container via the Options -> Features settings
> in PVE for this.

I'll try that, I did not discover that there is a feature to enabled
this from proxmox.

I'll report back what I find.

Kind Regards,
Johannes

[Johannes Kastl]

On 08.06.21 16:07 Johannes Kastl wrote:
> Hi Wolfang,

Sorry for mistyping your name...

> I'll report back what I find.

I found the "Create devices" Feature, which apparently adds "features:
mknod=1" to the pct configuration.

And I can successfully create nodes inside the container.

I have not found which exact setting in the config file in
/var/lib/lxc/ID/config this relates to. Wolfgang, do you have a link
to this feature's documentation?

Kind Regards,
Johannes

[Serge E. Hallyn]

No, then that is correct. It's possible you need to add create
to the options, i.e.

lxc.mount.entry = /dev/null var/lib/named/dev/null none bind,create=file 0 0

[Wolfgang Bumiller]

> On 06/08/2021 4:35 PM Johannes Kastl <ma...@ojkastl.de> wrote:
> I found the "Create devices" Feature, which apparently adds "features:
> mknod=1" to the pct configuration.
>
> And I can successfully create nodes inside the container.
>
> I have not found which exact setting in the config file in
> /var/lib/lxc/ID/config this relates to. Wolfgang, do you have a link
> to this feature's documentation?

There's no user-friendly documentation as of yet.

It sets `lxc.seccomp.notify.proxy`[1][2] and modifies the seccomp profile to have mknod syscalls be handled by a userspace daemon, pve-lxc-syscalld[3], since the kernel rejects device mknod calls from user namespaces.

Only a handful of device nodes are allowed via a hardcoded list[4] and changing the devices cgroup will not enable more of them to work.

pve-lxc-syscalld is basically the first tool in pve using the rust language, particularly its 'async' feature, which was very new at the time and async-bugs in it or its dependencies had the potential to lead to weird hangs, which is part of the reason why it's still marked as experimental. (Though by now it uses tokio's (the runtime) stable 1.0 version[5] and seems to work fine.)

[1] https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html
[2] https://git.proxmox.com/?p=pve-container.git;a=blob;f=src/PVE/LXC.pm;h=bb1cbdbd18fd507669d7b4786b9b5e7ce4f99983;hb=HEAD#l454
[3] https://git.proxmox.com/?p=pve-lxc-syscalld.git;a=summary
[4] https://git.proxmox.com/?p=pve-lxc-syscalld.git;a=blob;f=src/sys_mknod.rs;h=a498b779dfa841561510aa8eac42cf4c52828d6f;hb=a8a175ea0baa24136fb0fe4f15468d6cb80fe975#l47
[5] https://git.proxmox.com/?p=pve-lxc-syscalld.git;a=commit;h=7a1ab2b21eaf49e8d89f57397f097f7fd7632058

[Johannes Kastl]

Hi Wolfgang,

On 09.06.21 08:46 Wolfgang Bumiller wrote:
>> On 06/08/2021 4:35 PM Johannes Kastl <ma...@ojkastl.de> wrote:

>> I have not found which exact setting in the config file in
>> /var/lib/lxc/ID/config this relates to. Wolfgang, do you have a link
>> to this feature's documentation?
>
> There's no user-friendly documentation as of yet.
>
> It sets `lxc.seccomp.notify.proxy`[1][2] and modifies the seccomp profile to have mknod syscalls be handled by a userspace daemon, pve-lxc-syscalld[3], since the kernel rejects device mknod calls from user namespaces.
>
> Only a handful of device nodes are allowed via a hardcoded list[4] and changing the devices cgroup will not enable more of them to work.
>
> pve-lxc-syscalld is basically the first tool in pve using the rust language, particularly its 'async' feature, which was very new at the time and async-bugs in it or its dependencies had the potential to lead to weird hangs, which is part of the reason why it's still marked as experimental. (Though by now it uses tokio's (the runtime) stable 1.0 version[5] and seems to work fine.)
>

Thank you very much for the very nice and detailled description. And
thanks very much for putting so much work regarding lxc containers
into Proxmox. I really like it!

Kind Regards,
Johannes

[Johannes Kastl]

Hi Serge,

On 08.06.21 18:05 Serge E. Hallyn wrote:

> No, then that is correct. It's possible you need to add create
> to the options, i.e.
>
> lxc.mount.entry = /dev/null var/lib/named/dev/null none bind,create=file 0 0
>

I'll give that a try. Thanks!

Kind Regards,
Johannes

[Serge E. Hallyn]

On Wed, Jun 09, 2021 at 08:46:43AM +0200, Wolfgang Bumiller wrote:
>
> > On 06/08/2021 4:35 PM Johannes Kastl <ma...@ojkastl.de> wrote:
> > I found the "Create devices" Feature, which apparently adds "features:
> > mknod=1" to the pct configuration.
> >
> > And I can successfully create nodes inside the container.
> >
> > I have not found which exact setting in the config file in
> > /var/lib/lxc/ID/config this relates to. Wolfgang, do you have a link
> > to this feature's documentation?
>
> There's no user-friendly documentation as of yet.
>
> It sets `lxc.seccomp.notify.proxy`[1][2] and modifies the seccomp profile to have mknod syscalls be handled by a userspace daemon, pve-lxc-syscalld[3], since the kernel rejects device mknod calls from user namespaces.

That's cool :)

> Only a handful of device nodes are allowed via a hardcoded list[4] and changing the devices cgroup will not enable more of them to work.
>
> pve-lxc-syscalld is basically the first tool in pve using the rust language, particularly its 'async' feature, which was very new at the time and async-bugs in it or its dependencies had the potential to lead to weird hangs, which is part of the reason why it's still marked as experimental. (Though by now it uses tokio's (the runtime) stable 1.0 version[5] and seems to work fine.)
>
> [1] https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html
> [2] https://git.proxmox.com/?p=pve-container.git;a=blob;f=src/PVE/LXC.pm;h=bb1cbdbd18fd507669d7b4786b9b5e7ce4f99983;hb=HEAD#l454
> [3] https://git.proxmox.com/?p=pve-lxc-syscalld.git;a=summary
> [4] https://git.proxmox.com/?p=pve-lxc-syscalld.git;a=blob;f=src/sys_mknod.rs;h=a498b779dfa841561510aa8eac42cf4c52828d6f;hb=a8a175ea0baa24136fb0fe4f15468d6cb80fe975#l47
> [5] https://git.proxmox.com/?p=pve-lxc-syscalld.git;a=commit;h=7a1ab2b21eaf49e8d89f57397f097f7fd7632058
>

> --
> You received this message because you are subscribed to the Google Groups "lxc-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lxc-users+...@lists.linuxcontainers.org.

> To view this discussion on the web visit https://groups.google.com/a/lists.linuxcontainers.org/d/msgid/lxc-users/919428530.914.1623221203605%40webmail.proxmox.com.