Bug#638322: nfs-common: rpc.statd binds to udp port 631 preventing cups startup
Ben Hutchings
> Package: nfs-common
> Version: 1:1.2.4-1
> Severity: normal
>
> Hi,
>
> sometimes rpc.statd binds to port 631 udp which is used by cups. Therefore cups is unable to bind to its port and no printers get discovered.
>
> Rebooting the system helps as rpc.statd uses another port afterwards.
This is a fundamental problem of the bindresvport() function, and
not specific to rpc.statd. Reassigning to general.
The 'portreserve' package provides a kluge to avoid this, but it
requires other packages to register the ports that must be reserved.
It also won't work reliably, because insserv runs init scripts in
parallel and there is thus a race condition in the way services claim
their ports from the portreserve daemon.
A proper fix probably involves using systemd's socket-activation.
Yes, I said systemd - which presumably means we'll have to wait
another 5 years for this to be fixed.
Ben.
--
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
- Albert Camus
--
To UNSUBSCRIBE, email to debian-bugs-...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
brian m. carlson
> On Thu, Aug 18, 2011 at 05:39:16PM +0200, Jan Möbius wrote:
> > sometimes rpc.statd binds to port 631 udp which is used by cups. Therefore cups is unable to bind to its port and no printers get discovered.
> >
> > Rebooting the system helps as rpc.statd uses another port afterwards.
>
> This is a fundamental problem of the bindresvport() function, and
> not specific to rpc.statd. Reassigning to general.
Actually, according to the manpage:
Unlike some bindresvport() implementations, the glibc implementation
ignores any value that the caller supplies in sin->sin_port.
Fixing this might be a useful way around the problem. I'd code up a
patch, but eglibc won't take it without copyright assignment.
--
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187
Ben Hutchings
> On Thu, Aug 18, 2011 at 05:41:22PM +0100, Ben Hutchings wrote:
> > On Thu, Aug 18, 2011 at 05:39:16PM +0200, Jan Möbius wrote:
> > > sometimes rpc.statd binds to port 631 udp which is used by cups. Therefore cups is unable to bind to its port and no printers get discovered.
> > >
> > > Rebooting the system helps as rpc.statd uses another port afterwards.
> >
> > This is a fundamental problem of the bindresvport() function, and
> > not specific to rpc.statd. Reassigning to general.
>
> Actually, according to the manpage:
>
> Unlike some bindresvport() implementations, the glibc implementation
> ignores any value that the caller supplies in sin->sin_port.
>
> Fixing this might be a useful way around the problem. I'd code up a
> patch, but eglibc won't take it without copyright assignment.
You can't fix that, because it can't rely on existing callers to
initialise the field at all.
Ben.
Russell Coker
daemons which use RPC services are not permitted to bind to any of the ports
that are reserved for services and therefore such a bind attempt fails with
EPERM, glibc will just decrement the port number and try again when this
happens.
http://etbe.coker.com.au/2007/11/06/squid-and-se-linux/
I mentioned this in the above blog post, I think it was in about 2002 that I
wrote the policy to do this.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/201108191013....@coker.com.au
Guus Sliepen
> Systems running SE Linux tend not to have this problem. In most cases the
> daemons which use RPC services are not permitted to bind to any of the ports
> that are reserved for services and therefore such a bind attempt fails with
> EPERM, glibc will just decrement the port number and try again when this
> happens.
>
> http://etbe.coker.com.au/2007/11/06/squid-and-se-linux/
>
> I mentioned this in the above blog post, I think it was in about 2002 that I
> wrote the policy to do this.
We could also patch bindresvport() to skip all ports mentioned in
/etc/services, to get similar behaviour as with SE Linux. Or patch the programs
using it to first try to bind to a static port that does not conflict with
those in /etc/services, and if that fails fall back to bindresvport().
--
Met vriendelijke groet / with kind regards,
Guus Sliepen <gu...@debian.org>
Russell Coker
> We could also patch bindresvport() to skip all ports mentioned in
> /etc/services, to get similar behaviour as with SE Linux. Or patch the
> programs using it to first try to bind to a static port that does not
> conflict with those in /etc/services, and if that fails fall back to
> bindresvport().
That would be a viable option. On my system there are 124 TCP ports listed
with numbers <1024 (which seems to be the main problem area). Losing 12% of
the address space seems viable.
One thing to note when comparing this to SE Linux is that the SE Linux policy
labels some ports that aren't in /etc/services but which are in relatively
common use. One example is port 24 for LMTP. Also with SE Linux there is an
easy way of adding new port labels and as the typical daemon won't be
permitted to bind to an unlabeled port the sysadmin is compelled to do the
correct thing.
Now one could patch bindresvport() to also check /etc/services.local or some
other source of configuration information about which ports are likely to be
used. But getting the users to accept that will take some effort.
Of course most users just don't have enough RPC traffic to generate the
problem.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/201108191920....@coker.com.au
Adam Borowski
> On Fri, Aug 19, 2011 at 10:13:17AM +1000, Russell Coker wrote:
> > Systems running SE Linux tend not to have this problem. In most cases the
> > daemons which use RPC services are not permitted to bind to any of the ports
> > that are reserved for services and therefore such a bind attempt fails with
> > EPERM, glibc will just decrement the port number and try again when this
> > happens.
>
> /etc/services, to get similar behaviour as with SE Linux. Or patch the programs
> using it to first try to bind to a static port that does not conflict with
> those in /etc/services, and if that fails fall back to bindresvport().
Or use a whitelist rather than pretending that /etc/services was complete
anywhere within the last 20 years.
Not to mention bindresvport() removes the freedom of the sysadmin to bind
services to whatever ports she wishes. Or, say, run multiple instances of a
service.
--
1KB // Yo momma uses IPv4!
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Edward Allcutt
> On Thu, Aug 18, 2011 at 05:39:16PM +0200, Jan Möbius wrote:
>> sometimes rpc.statd binds to port 631 udp which is used by cups. Therefore cups is unable to bind to its port and no printers get discovered.
>>
>> Rebooting the system helps as rpc.statd uses another port afterwards.
>
> This is a fundamental problem of the bindresvport() function, and
> not specific to rpc.statd. Reassigning to general.
Sure, bindresvport is archaic, but workarounds already exist. In
particular, Debian already adds /etc/bindresvport.blacklist and the
default already contains port 631. Does the submitter have this
file in place with the default contents?
> The 'portreserve' package provides a kluge to avoid this, but it
> requires other packages to register the ports that must be reserved.
> It also won't work reliably, because insserv runs init scripts in
> parallel and there is thus a race condition in the way services claim
> their ports from the portreserve daemon.
That seems like a much worse kluge than the existing blacklist. Allowing
packages to register reserved ports however seems a useful feature.
Reassign to eglibc as request for supporting /etc/bindresvport.blacklist.d ?
> A proper fix probably involves using systemd's socket-activation.
> Yes, I said systemd - which presumably means we'll have to wait
> another 5 years for this to be fixed.
Irrelevant. Promoting systemd for its side-effect as an amelioration for an
ureliable kluge is not a strong argument. ;) [0]
[0] Not intended as an argument against systemd either.
--
Edward Allcutt
Russell Coker
> Or use a whitelist rather than pretending that /etc/services was complete
> anywhere within the last 20 years.
AFAIK /etc/services has always been a complete list of ports assigned by IANA.
If someone makes a port commonly used without getting IANA approval that's
their problem/mistake.
> Not to mention bindresvport() removes the freedom of the sysadmin to bind
> services to whatever ports she wishes. Or, say, run multiple instances of
> a service.
If you make your program use bindresvport() then it means that you don't care
what the port number is as long as it's in the reserved range. This generally
means that it's a RPC service and the Portmapper will tell everyone which port
to use or that there is some other channel to tell the clients which port to
connect to (maybe a bit like the FTP two-port setup).
If you run multiple instances of a service using RPC then I guess you could
use different names with the Portmapper.
It seems to me that the only problem is if you run multiple instances of a
daemon on different ports and don't use /etc/bindresvport.blacklist, SE Linux,
or some other method of telling bindresvport() to leave your port alone. That
wouldn't be an issue of sysadmin freedom but sysadmin ignorance (and I am one
of the people who was ignorant of bindresvport.blacklist).
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/201108200002....@coker.com.au
Adam Borowski
> On Fri, 19 Aug 2011, Adam Borowski <kilo...@angband.pl> wrote:
> > Not to mention bindresvport() removes the freedom of the sysadmin to bind
> > services to whatever ports she wishes. Or, say, run multiple instances of
> > a service.
>
> If you make your program use bindresvport() then it means that you don't care
> what the port number is as long as it's in the reserved range.
Except that it should not get into ports used by something else.
> It seems to me that the only problem is if you run multiple instances of a
> daemon on different ports and don't use /etc/bindresvport.blacklist, SE Linux,
> or some other method of telling bindresvport() to leave your port alone. That
> wouldn't be an issue of sysadmin freedom but sysadmin ignorance (and I am one
> of the people who was ignorant of bindresvport.blacklist).
You can't blame "sysadmin ignorance". I've just grepped through every
single man page in Debian (ok, amd64 main), and there is not a single
reference to /etc/bindresvport.blacklist. In fact, even bindresvport() is
referenced only from its own manpage and from portreserve which is another
hack to work around this bug. portreserve is neither recommended/suggested,
nor has any data that would allow it to work.
No other daemon I know has this problem. If I install daemon foo, I can
expect it to not touch any ports it hasn't been configured to use. It's
just portmap/SunRPC that uses random scatter-shot that can trample on
something else.
So what about this: let's reserve a number of ports for portmap's exclusive
usage[1]. There's like 900 unused assignments, so there's plenty of space
than could be parcelled off. SunRPC has long since degenerated from
something with a general purpose to a peculiarity of NFS, so not many ports
are needed. Only under a pathological configuration one could exceed any
reasonable static limit, and in that case bindresvport() would revert to the
blacklist+scattershot.
[1]. Unless the sysadmin knowingly takes them for some other purpose; no
different from, say, having sshd listen on port 443.
--
1KB // Yo momma uses IPv4!
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Russell Coker
> > It seems to me that the only problem is if you run multiple instances of
> > a daemon on different ports and don't use /etc/bindresvport.blacklist,
> > SE Linux, or some other method of telling bindresvport() to leave your
> > port alone. That wouldn't be an issue of sysadmin freedom but sysadmin
> > ignorance (and I am one of the people who was ignorant of
> > bindresvport.blacklist).
>
> You can't blame "sysadmin ignorance". I've just grepped through every
> single man page in Debian (ok, amd64 main), and there is not a single
Ignorance means not knowing. Sure there are probably some bug reports about
man pages due, but it's still something you or I could have found out.
apt-get source libc6
> No other daemon I know has this problem. If I install daemon foo, I can
> expect it to not touch any ports it hasn't been configured to use. It's
> just portmap/SunRPC that uses random scatter-shot that can trample on
> something else.
Yes, SunRPC and anything that opens a port for callback.
> So what about this: let's reserve a number of ports for portmap's exclusive
> usage[1]. There's like 900 unused assignments, so there's plenty of space
> than could be parcelled off. SunRPC has long since degenerated from
> something with a general purpose to a peculiarity of NFS, so not many ports
> are needed. Only under a pathological configuration one could exceed any
> reasonable static limit, and in that case bindresvport() would revert to
> the blacklist+scattershot.
The problem with this theory is the fact that the problem that was reported
with CUPS only occurred after bindresvport() had used every port from 1023
down to 631. A casual scan of /etc/services reveals that there are no long
contiguous ranges available without reserved ports. If you start at the top
the common ports pop3s and imaps could be reached quite quickly.
So it seems that some sort of blacklist is the only way to go.
The idea of a .d directory for blacklist files such that every package
installation that is likely to use some ports will automatically have a
reservation is a good one. Of course there's still the corner case of trying
to install CUPS (or some other daemon) after a long-running RPC service has
grabbed the port.
Maybe we should default to having ports such as 631, 993, 995, 873, 587, 636,
546, and 547 reserved at all times. From a quick scan of /etc/services they
seem to be the most likely ports to be used in the 500-1024 range.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/201108202219....@coker.com.au
Henrique de Moraes Holschuh
> On Sat, 20 Aug 2011, Adam Borowski <kilo...@angband.pl> wrote:
> > > It seems to me that the only problem is if you run multiple instances of
> > > a daemon on different ports and don't use /etc/bindresvport.blacklist,
> > > SE Linux, or some other method of telling bindresvport() to leave your
> > > port alone. That wouldn't be an issue of sysadmin freedom but sysadmin
> > > ignorance (and I am one of the people who was ignorant of
> > > bindresvport.blacklist).
> >
> > You can't blame "sysadmin ignorance". I've just grepped through every
> > single man page in Debian (ok, amd64 main), and there is not a single
>
> Ignorance means not knowing. Sure there are probably some bug reports about
> man pages due, but it's still something you or I could have found out.
...
> > No other daemon I know has this problem. If I install daemon foo, I can
> > expect it to not touch any ports it hasn't been configured to use. It's
> > just portmap/SunRPC that uses random scatter-shot that can trample on
> > something else.
>
> Yes, SunRPC and anything that opens a port for callback.
Firewall port blocking can also cause such problems (denial of service).
While it is a different problem, it has the same roots as SunRPC binding to
undesired sockets: applications that use random sockets do not know whether
they're going to get a socket they're supposed to use.
Intelligent use of conntrack can help on single hosts (reducing the problem
to incoming callback connections), but most sites have border policies that
forbid any traffic to flow for some ports. It causes minor issues for DNS
traffic (timeouts on a small fraction of the queries), for example.
> So it seems that some sort of blacklist is the only way to go.
Yes. And we can easily maintain a current one for Debian-packaged software,
although the initial build of such a blacklist will take some work.
> The idea of a .d directory for blacklist files such that every package
> installation that is likely to use some ports will automatically have a
> reservation is a good one. Of course there's still the corner case of trying
> to install CUPS (or some other daemon) after a long-running RPC service has
> grabbed the port.
That's not such a big problem, as it will be noticed immediately and causes
no surprise downtime of a service.
> Maybe we should default to having ports such as 631, 993, 995, 873, 587, 636,
> 546, and 547 reserved at all times. From a quick scan of /etc/services they
> seem to be the most likely ports to be used in the 500-1024 range.
Looks good, and we can take extra ports as bug reports. A mail to d-d-a and
a short article to planet.d.o and LWN may help to raise awareness of such
issues: although this _is_ a longstanding and _well known_ issue, the ways
to avoid the worst problems it can cause are _not_ well known.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20110820123...@khazad-dum.debian.net
Ben Hutchings
> On Thu, 18 Aug 2011, Ben Hutchings wrote:
> > On Thu, Aug 18, 2011 at 05:39:16PM +0200, Jan Möbius wrote:
> >> sometimes rpc.statd binds to port 631 udp which is used by cups. Therefore cups is unable to bind to its port and no printers get discovered.
> >>
> >> Rebooting the system helps as rpc.statd uses another port afterwards.
> >
> > This is a fundamental problem of the bindresvport() function, and
> > not specific to rpc.statd. Reassigning to general.
>
> Sure, bindresvport is archaic, but workarounds already exist. In
> particular, Debian already adds /etc/bindresvport.blacklist and the
> default already contains port 631. Does the submitter have this
> file in place with the default contents?
Oh, I completely missed that.
> > The 'portreserve' package provides a kluge to avoid this, but it
> > requires other packages to register the ports that must be reserved.
> > It also won't work reliably, because insserv runs init scripts in
> > parallel and there is thus a race condition in the way services claim
> > their ports from the portreserve daemon.
>
> That seems like a much worse kluge than the existing blacklist. Allowing
> packages to register reserved ports however seems a useful feature.
>
> Reassign to eglibc as request for supporting /etc/bindresvport.blacklist.d ?
[...]
That seems like it would be necessary in the general case. However, if
port 631 is already on the list then it has nothing to do with the
current bug report.
In fact, the problem seems to be that bindresvport() supports IPv4 only
and therefore libtirpc (the new SunRPC client library) does not use it
(for either IPv4 or IPv6). glibc declares bindresvport6() for IPv6
addresses, but it doesn't appear to define it. So it seems like we need
to:
1. Add bindresvport6() to glibc
2. Use glibc's bindresvport{,6}() in libtirpc
3. Add configuration directory for reserving more ports (not for this
bug)
Ben.
Andreas Barth
> Yes. And we can easily maintain a current one for Debian-packaged software,
> although the initial build of such a blacklist will take some work.
Actually, the existing interface net.ipv4.ip_local_port_range seems to
work quite well. And there are so many ports that for most servers it
seems acceptable to limit the outgoing ports to only a tiny portion of
port numbers (like 1/4th or so).
Andi
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20110820141...@mails.so.argh.org
Ben Hutchings
> * Henrique de Moraes Holschuh (h...@debian.org) [110820 14:39]:
> > Yes. And we can easily maintain a current one for Debian-packaged software,
> > although the initial build of such a blacklist will take some work.
>
> Actually, the existing interface net.ipv4.ip_local_port_range seems to
> work quite well. And there are so many ports that for most servers it
> seems acceptable to limit the outgoing ports to only a tiny portion of
> port numbers (like 1/4th or so).
This has nothing to do with bindresvport().
Ben.
Henrique de Moraes Holschuh
> * Henrique de Moraes Holschuh (h...@debian.org) [110820 14:39]:
> > Yes. And we can easily maintain a current one for Debian-packaged software,
> > although the initial build of such a blacklist will take some work.
>
> Actually, the existing interface net.ipv4.ip_local_port_range seems to
> work quite well. And there are so many ports that for most servers it
No, it doesn't. And we have at least one extremely important protocol that
needs as many ports as we can give it (DNS).
A blacklist is the way to go, and we already have it. We just need to fill
it, make it easier to extend (.d directory), tell people about it, and teach
stuff other than SunRPC to use it when necessary.
--
"One disk to rule them all, One disk to find them. One disk to bring
them all and in the darkness grind them. In the Land of Redmond
where the shadows lie." -- The Silicon Valley Tarot
Henrique Holschuh
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20110821035...@khazad-dum.debian.net
Russell Coker
> On Sat, 20 Aug 2011, Andreas Barth wrote:
> > * Henrique de Moraes Holschuh (h...@debian.org) [110820 14:39]:
> > > Yes. And we can easily maintain a current one for Debian-packaged
> > > software, although the initial build of such a blacklist will take
> > > some work.
> >
> > Actually, the existing interface net.ipv4.ip_local_port_range seems to
> > work quite well. And there are so many ports that for most servers it
# cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
The above is from one of my systems. This isn't used for RPC, presumably
because they want the special <1024 port numbers that imply root ownership.
> No, it doesn't. And we have at least one extremely important protocol that
> needs as many ports as we can give it (DNS).
Aug 21 11:42:48 ns named[2382]: using default UDP/IPv4 port range: [1024,
65535]
Aug 21 11:42:48 ns named[2382]: using default UDP/IPv6 port range: [1024,
65535]
BIND seems to use ports >1024 as well, again this is different from the
typical RPC issues but does have the potential to cause problems (there are
more than a few UDP ports >1024 in /etc/services). Maybe BIND should be
patched to use the same port reservation procedure as RPC.
> A blacklist is the way to go, and we already have it. We just need to fill
> it, make it easier to extend (.d directory), tell people about it, and
> teach stuff other than SunRPC to use it when necessary.
Yes.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/201108212238....@coker.com.au