UNIX domain sockets on nullfs still broken?
xorqu...@googlemail.com
in it. Clients connect to this socket to communicate with the server.
$ jackd -d oss -r 44100 -p 128
$ ls -alF /tmp/jack-11001/default
total 4
drwx------ 2 xw wheel 512 30 Nov 14:19 ./
drwx------ 3 xw wheel 512 30 Nov 14:19 ../
prw-r--r-- 1 xw wheel 0 30 Nov 14:19 jack-ack-fifo-54211-0|
prw-r--r-- 1 xw wheel 0 30 Nov 14:19 jack-ack-fifo-54211-1|
prw-r--r-- 1 xw wheel 0 30 Nov 14:19 jack-ack-fifo-54211-2|
srwxr-xr-x 1 xw wheel 0 30 Nov 14:19 jack_0=
srwxr-xr-x 1 xw wheel 0 30 Nov 14:19 jack_ack_0=
$ sudo mount_nullfs /tmp/ /jail/k4m/tmp
In the jail:
k4m$ ls -alF /tmp/jack-11001/default
drwx------ 2 xw wheel 512 30 Nov 14:19 ./
drwx------ 3 xw wheel 512 30 Nov 14:19 ../
prw-r--r-- 1 xw wheel 0 30 Nov 14:19 jack-ack-fifo-54211-0|
prw-r--r-- 1 xw wheel 0 30 Nov 14:19 jack-ack-fifo-54211-1|
prw-r--r-- 1 xw wheel 0 30 Nov 14:19 jack-ack-fifo-54211-2|
srwxr-xr-x 1 xw wheel 0 30 Nov 14:19 jack_0=
srwxr-xr-x 1 xw wheel 0 30 Nov 14:19 jack_ack_0=
k4m$ ktrace jack_showtime
jack server not running?
k4m$ kdump | grep '/tmp/jack-11001'
76030 initial thread STRU struct sockaddr { AF_LOCAL, /tmp/jack-11001/default/jack_0 }
76030 initial thread NAMI "/tmp/jack-11001/default/jack_0"
76030 initial thread RET connect -1 errno 61 Connection refused
$ uname -a
FreeBSD viper.internal.network 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:02:08 UTC 2009 ro...@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
xw
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
Ivan Voras
I would expect to see this result from the jail since it's obviously a
Bad Idea, but does it work from the same (host) machine without the jail
in between (i.e. just the nullfs, no jails)?
Ivan Voras
> On 2009-11-30 15:43:01, Ivan Voras wrote:
>> xorqu...@googlemail.com wrote:
>>> 76030 initial thread STRU struct sockaddr { AF_LOCAL, /tmp/jack-11001/default/jack_0 }
>>> 76030 initial thread NAMI "/tmp/jack-11001/default/jack_0"
>>> 76030 initial thread RET connect -1 errno 61 Connection refused
>> I would expect to see this result from the jail since it's obviously a
>> Bad Idea, but does it work from the same (host) machine without the jail
>> in between (i.e. just the nullfs, no jails)?
>
>
> What's the sane solution, then, when the only method of communication
> is unix domain sockets?
It is a security problem. I think the long-term solution would be to add
a sysctl analogous to security.jail.param.securelevel to handle this.
I don't think there is a workaround right now.
Vlad Galu
> On 2009-11-30 15:43:01, Ivan Voras wrote:
>> xorqu...@googlemail.com wrote:
>> > �76030 initial thread NAMI �"/tmp/jack-11001/default/jack_0"
>> > �76030 initial thread RET � connect -1 errno 61 Connection refused
>>
>> I would expect to see this result from the jail since it's obviously a
>> Bad Idea, but does it work from the same (host) machine without the jail
>> in between (i.e. just the nullfs, no jails)?
>
>
> What's the sane solution, then, when the only method of communication
> is unix domain sockets?
For redirecting a connection to a UNIX socket to a remote host:port,
there's net/unix2tcp. Perhaps you can patch it to go the other way
around as well?
>
> xw
Alexander Leidinger
16:14:40 +0100):
> xorqu...@googlemail.com wrote:
>> On 2009-11-30 15:43:01, Ivan Voras wrote:
>>> xorqu...@googlemail.com wrote:
>>>> 76030 initial thread STRU struct sockaddr { AF_LOCAL,
>>>> /tmp/jack-11001/default/jack_0 }
>>>> 76030 initial thread NAMI "/tmp/jack-11001/default/jack_0"
>>>> 76030 initial thread RET connect -1 errno 61 Connection refused
>>> I would expect to see this result from the jail since it's
>>> obviously a Bad Idea, but does it work from the same (host) machine
It is not a bad idea, at least not if we talk about mounting something
from JailA to JailB. Think about the MySQL socket. I have a jail with
MySQL, and I have a jail which wants to connect to it. I do not want
to allow network connections between those jails (be it for
performance reasons, or that I do not want to involve a network
connection, or that I do not want to give the MySQL jail an IP at all
or whatever).
Solution: give access to the socket via the FS. Ideally by putting the
socket in its own directory and mounting this directory over to the
jail. A workaround for this scenario is below.
>>> without the jail in between (i.e. just the nullfs, no jails)?
>>
>> Hm, yes, you're right. It does work without a jail involved.
>>
>> What's the sane solution, then, when the only method of communication
>> is unix domain sockets?
>
> It is a security problem. I think the long-term solution would be to
It is a risk-management problem, and as such not the responsability of
FreeBSD to enforce it. If the sysadmin wants to shoot in his foot, it
is his decision.
> add a sysctl analogous to security.jail.param.securelevel to handle this.
Do you know the code which is responsible for the reject of access to
the socket? If yes I can provide a patch regarding jail.param.something.
> I don't think there is a workaround right now.
My workaround with MySQL is to have the jail and the socket in the
same FS (I would prefer to have them on separate FS). Then you can do
a hardlink of the socket into the jail (obviously after each restart
of the software, but this can be scripted). This works for me.
Bye,
Alexander.
--
You are capable of planning your future.
http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
xorqu...@googlemail.com
>
> My workaround with MySQL is to have the jail and the socket in the
> same FS (I would prefer to have them on separate FS). Then you can do
> a hardlink of the socket into the jail (obviously after each restart
> of the software, but this can be scripted). This works for me.
>
Interesting. I'll try it. Thanks.
Regards,
xw
Linda Messerschmidt
>> What's the sane solution, then, when the only method of communication
>> is unix domain sockets?
>
> It is a security problem. I think the long-term solution would be to add a
> sysctl analogous to security.jail.param.securelevel to handle this.
Out of curiosity, why is allowing accessing to a Unix domain socket in
a filesystem to which a jail has explicitly been allowed access more
or less secure than allowing access to a file or a devfs node in a
filesystem to which a jail has explicitly been allowed access?
Alexander Leidinger
1 Dec 2009 10:22:02 -0500):
> On Mon, Nov 30, 2009 at 10:14 AM, Ivan Voras <ivo...@freebsd.org> wrote:
>>> What's the sane solution, then, when the only method of communication
>>> is unix domain sockets?
>>
>> It is a security problem. I think the long-term solution would be to add a
>> sysctl analogous to security.jail.param.securelevel to handle this.
>
> Out of curiosity, why is allowing accessing to a Unix domain socket in
> a filesystem to which a jail has explicitly been allowed access more
> or less secure than allowing access to a file or a devfs node in a
> filesystem to which a jail has explicitly been allowed access?
Answer A: There is no difference.
Answer B: You open up a direct communication channel between two
systems, which may not have been able to communicate before (firewall
rules, ...). With files you can do something similar too, but having a
socket there makes it more easy and you do not need to write extra
code. It is similar to enabling SHM access in jails (currently all
jails share the same SHM area). And depending on the application with
the socket, you may be able to change files on the other side, to
which you do not have access to otherwise (think about a daemon which
changes passwords...).
Answer A is good if you control what is run where and how, and if you
use jails for easy data migration and program separation (lightweight
virtualization).
Answer B is valid if you are an ISP which rents jails (in this case
you do not share a FS read-write anyway (at leat you shouldn't) and
the point does not really matter).
Pick the answer depending on your viewpoint / security requirements
and the software you are using.
As both points are valid, we should provide the possibility to have
both situations working.
Bye,
Alexander.
--
Is death legally binding?
http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
Julian Elischer
> Quoting Linda Messerschmidt <linda.mes...@gmail.com> (from Tue, 1
> Dec 2009 10:22:02 -0500):
>
>> On Mon, Nov 30, 2009 at 10:14 AM, Ivan Voras <ivo...@freebsd.org> wrote:
>>>> What's the sane solution, then, when the only method of communication
>>>> is unix domain sockets?
>>>
>>> It is a security problem. I think the long-term solution would be to
>>> add a
>>> sysctl analogous to security.jail.param.securelevel to handle this.
>>
>> Out of curiosity, why is allowing accessing to a Unix domain socket in
>> a filesystem to which a jail has explicitly been allowed access more
>> or less secure than allowing access to a file or a devfs node in a
>> filesystem to which a jail has explicitly been allowed access?
>
> Answer A: There is no difference.
>
> Answer B: You open up a direct communication channel between two
> systems, which may not have been able to communicate before (firewall
> rules, ...). With files you can do something similar too, but having a
> socket there makes it more easy and you do not need to write extra code.
> It is similar to enabling SHM access in jails (currently all jails share
> the same SHM area). And depending on the application with the socket,
> you may be able to change files on the other side, to which you do not
> have access to otherwise (think about a daemon which changes passwords...).
I have used chroots and jails in a way that relies on the ability of a
shared unix domain pipe being usable to communicate between them, and
I also see why it may not be good.
I suggest that the ability to do so might be somehow controllable by
the jail creator in some way.
>
> Answer A is good if you control what is run where and how, and if you
> use jails for easy data migration and program separation (lightweight
> virtualization).
>
> Answer B is valid if you are an ISP which rents jails (in this case you
> do not share a FS read-write anyway (at leat you shouldn't) and the
> point does not really matter).
>
> Pick the answer depending on your viewpoint / security requirements and
> the software you are using.
>
> As both points are valid, we should provide the possibility to have both
> situations working.
yes please.
A sysctl would do at a pinch, but maybe a per-jail setting might be
possible too.
>
> Bye,
> Alexander.
Alexander Leidinger
09:43:25 -0800):
What worries me is, that it seems from comments in this thread, that
nullfs is having a tighter security regarding jails than UFS/ZFS. I
think all should work consistently in this regard (which would mean
there will be a regression for some people if we switch UFS/ZFS to
work in the same way).
> I suggest that the ability to do so might be somehow controllable by
> the jail creator in some way.
>
>>
>> Answer A is good if you control what is run where and how, and if
>> you use jails for easy data migration and program separation
>> (lightweight virtualization).
>>
>> Answer B is valid if you are an ISP which rents jails (in this case
>> you do not share a FS read-write anyway (at leat you shouldn't) and
>> the point does not really matter).
>>
>> Pick the answer depending on your viewpoint / security requirements
>> and the software you are using.
>>
>> As both points are valid, we should provide the possibility to have
>> both situations working.
>
> yes please.
> A sysctl would do at a pinch, but maybe a per-jail setting might be
> possible too.
Per-Jail is not a problem, I just need to know where the priv check is
which is causing this behavior (so far I thought it is some limitation
of nullfs and not a priv check). So far I hadn't the time to search
for it, I want to finish the import of v4l in the linuxulator first.
Bye,
Alexander.
--
BOFH excuse #102:
Power company testing new voltage spike (creation) equipment
http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137
Robert Watson
> jackd (audio/jack) creates a directory in /tmp with a UNIX domain socket in
> it. Clients connect to this socket to communicate with the server.
We currently support the sharing of UNIX domain sockets between file system
layers on either nullfs or unionfs. In the former case, this is a bug, and in
the latter case, it is a feature.
The specific nature of the bug is that you can't just copy the socket pointer
between layers in the vnode stack without additional reference counting (and
other similar state propagation), so if we allowed inter-layer access it would
lead to use-after-free panics and similar sorts of problems.
This occurs, BTW, because the socket pointer is directly in struct vnode, and
not queried by a VOP, which could be forwarded by nullfs down a layer. The
fixes here aren't easy, so I would anticipate UNIX domain sockets not working
across nullfs layers for some time to come. It's not immediately clear to me
which approach is the best way to fix it, since it likely requires UNIX domain
sockets to learn about stacked file systems in some form, which will
significantly complicate an already complicated relationship.
Robert N M Watson
Computer Laboratory
University of Cambridge
Robert Watson
>> What's the sane solution, then, when the only method of communication is
>> unix domain sockets?
>
> It is a security problem. I think the long-term solution would be to add a
> sysctl analogous to security.jail.param.securelevel to handle this.
>
> I don't think there is a workaround right now.
I'm not sure I agree on the above, hence my comments about nullfs and unionfs.
I see nullfs as intended to provide references (possibly masked to read-only)
to the same fundamental object, and unionfs to provide independence between
different consumers that see objects via different file system mounts. As
such, I'd expect UNIX domain sockets to "work" for inter-jail communication
when using nullfs, and "not work" when using unionfs. It's simply a property
of the implementation of the linkage between VFS and UNIX domain sockets that
they are currently both broken (in fact, someone tried to "fix" it with union
mounts recenty, running into the use-after-free bugs I mentioned, but also
breaking the semantics in my view).
Robert N M Watson
Computer Laboratory
University of Cambridge
Robert Watson
On Tue, 1 Dec 2009, Linda Messerschmidt wrote:
> On Mon, Nov 30, 2009 at 10:14 AM, Ivan Voras <ivo...@freebsd.org> wrote:
>>> What's the sane solution, then, when the only method of communication
>>> is unix domain sockets?
>>
>> It is a security problem. I think the long-term solution would be to add a
>> sysctl analogous to security.jail.param.securelevel to handle this.
>
> Out of curiosity, why is allowing accessing to a Unix domain socket in a
> filesystem to which a jail has explicitly been allowed access more or less
> secure than allowing access to a file or a devfs node in a filesystem to
> which a jail has explicitly been allowed access?
(I seem to have caught this thread rather late in the game due to being on
travel) -- Ivan is wrong about nullfs, it's broken due to a bug, not a
feature, and that bug is not present when using a single file system. He's
thinking of unionfs semantics, where if it worked it would be a bug. :-)
Robert N M Watson
Computer Laboratory
University of Cambridge
Robert Watson
> On Mon, 30 Nov 2009, xorqu...@googlemail.com wrote:
>
>> jackd (audio/jack) creates a directory in /tmp with a UNIX domain socket in
>> it. Clients connect to this socket to communicate with the server.
>
> We currently support the sharing of UNIX domain sockets between file system
> layers on either nullfs or unionfs. In the former case, this is a bug, and
Should read "neither ... nor".
Ivan Voras
>
> On Tue, 1 Dec 2009, Linda Messerschmidt wrote:
>
>> On Mon, Nov 30, 2009 at 10:14 AM, Ivan Voras <ivo...@freebsd.org> wrote:
>>>>
>>>> What's the sane solution, then, when the only method of communication
>>>> is unix domain sockets?
>>>
>>> It is a security problem. I think the long-term solution would be to add
>>> a
>>> sysctl analogous to security.jail.param.securelevel to handle this.
>>
>> Out of curiosity, why is allowing accessing to a Unix domain socket in a
>> filesystem to which a jail has explicitly been allowed access more or less
>> secure than allowing access to a file or a devfs node in a filesystem to
>> which a jail has explicitly been allowed access?
>
> (I seem to have caught this thread rather late in the game due to being on
> travel) -- Ivan is wrong about nullfs, it's broken due to a bug, not a
> feature, and that bug is not present when using a single file system. He's
> thinking of unionfs semantics, where if it worked it would be a bug. :-)
You have a point there. I was actually thinking more of sysvshm -
which doesn't have anything to do with any of the issues here - but
has some of the same properties (and is also used by databases - e.g.
postgresql, which I'm using daily so it sort of cross-linked). The
reason I'd like the nullfs barrier kept is that it (like shm) is used
for IPC, and in this case, IPC across different jails (though a file
system itself also be used so...). It's not a big issue - I'll also
accept that it's the operator's fault if he doesn't know sharing file
systems will also share sockets and fifos on it...