Bind to port 22 on 0.0.0.0 failed: Address already in use
Ignoramus26630
On one of them, sshd could not start.
Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
My question is WTF, what could it be that keeps the port during boot
time?
For now, I added a command that attempts to restart sshd after 3
minutes, but I am a little concerned.
Googling suggests that it is because ipv4 and ipv6 comes up at a
diff. time, but still, this is kind of bad.
i
Pascal Hambourg
Be aware that crossposting to multiple groups without a follow-up
reduces your chances to be read, as some servers reject such articles.
Ignoramus26630 a ï¿œcrit :
>
> Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
>
> My question is WTF, what could it be that keeps the port during boot
> time?
Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
the IPv6 unspecified address), that sshd opened just before. If the
socket was opened without the V6ONLY option or if sysctl
net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
connections and thus prevents opening an IPv4 socket on the same port.
But it should be fine, as the IPv6 socket accepts both IPv4 and IPv6
connections. This was intended to ease transition from IPv4 to IPv6 in
applications. However I don't understand the SIGHUP.
> For now, I added a command that attempts to restart sshd after 3
> minutes, but I am a little concerned.
What does this change ?
Ignoramus26630
> Hello,
>
> Be aware that crossposting to multiple groups without a follow-up
> reduces your chances to be read, as some servers reject such articles.
Not in my experience.
> Ignoramus26630 a ?crit :
>>
>> Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
>>
>> My question is WTF, what could it be that keeps the port during boot
>> time?
>
> Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
> the IPv6 unspecified address), that sshd opened just before. If the
> socket was opened without the V6ONLY option or if sysctl
> net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
> connections and thus prevents opening an IPv4 socket on the same port.
> But it should be fine, as the IPv6 socket accepts both IPv4 and IPv6
> connections. This was intended to ease transition from IPv4 to IPv6 in
> applications. However I don't understand the SIGHUP.
>
>> For now, I added a command that attempts to restart sshd after 3
>> minutes, but I am a little concerned.
>
> What does this change ?
I am fairly positive that it is a very temporary, transient
issue. That's why it does not happen all the time on all servers. In
only (so far) occurred once on one server out of very many ones that
reboot every week.
Starting sshd again after 3 minutes should cover me pretty well.
However, I am disappointed that such an essential (for accessing
remote computers) function like ssh, could not initialize and gave
up.
I was lucky that I could hack (kind of) my way into this server using
its autoupdate scripts.
If, say, httpd could not start, I would not be as disappointed,
because it can be fixed remotely, but if, say, sshd cannot start then
you end up with a dangerous brick.
i
Greg Russell
Ignoramus26630 <ignoram...@NOSPAM.26630.invalid> typed:
> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22
> on 0.0.0.0 failed: Address already in use. Feb 28 00:04:52
...
> My question is WTF, what could it be that keeps the port during boot
> time?
A previously started process that uses port 22.
Use "lsof -i :22" or "netstat -antp | egrep ':22|:ssh' " to see what's there
akready.
Pascal Hambourg
> On 2010-02-28, Pascal Hambourg <boite-...@plouf.fr.eu.org> wrote:
>> Hello,
>>
>> Be aware that crossposting to multiple groups without a follow-up
>> reduces your chances to be read, as some servers reject such articles.
>
> Not in my experience.
Out of the three NNTP services I use to read Usenet groups, only one
shows this thread.
>> Ignoramus26630 a ?crit :
>>> Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
>>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
>>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
>>>
>>> My question is WTF, what could it be that keeps the port during boot
>>> time?
>> Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
>> the IPv6 unspecified address), that sshd opened just before. If the
>> socket was opened without the V6ONLY option or if sysctl
>> net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
>> connections and thus prevents opening an IPv4 socket on the same port.
>> But it should be fine, as the IPv6 socket accepts both IPv4 and IPv6
>> connections. This was intended to ease transition from IPv4 to IPv6 in
>> applications. However I don't understand the SIGHUP.
The "address already in use" issue was never a problem for me (anyway it
can be solved with -4 or -6 options on the sshd commandline or
ListenAddress directives in sshd_config according to one's needs).
Actually the real problem seems to be caused by the SIGHUP, which is
totally unrelated AFAICS.
I found this Debian bug report that may be related to your problem :
<http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=502444>
A script sends a SIGHUP to sshd and creates a race condition.
Dag-Erling Smørgrav
> Probably the IPv6 socket listening on :: (short form of 0:0:0:0:0:0:0:0,
> the IPv6 unspecified address), that sshd opened just before. If the
> socket was opened without the V6ONLY option or if sysctl
> net.ipv6.bindv6only=0 (kernel default), such a socket also accepts IPv4
> connections and thus prevents opening an IPv4 socket on the same port.
OpenSSH 4.9p1 and newer is aware of this issue and sets the V6ONLY
option on the socket before binding it to the listen address (normally
0:22).
Hardy still has 4.7p1, but that doesn't matter. In the default
configuration, sshd will use getaddrinfo() with AF_UNSPEC and AI_PASSIVE
to obtain a list of all available addresses. It will then attempt to
bind to each of them in order. It will warn about those that fail, but
as long as at least one of them worked, it will eun. Hence, the only
impact is that if 0.0.0.0 was listed first (which was clearly not the
case here), :: will fail and sshd will only listen for IPv4 connections.
In either case - whether 0.0.0.0 or :: was listed first - it will warn
about failing to bind to the second address, but it will still run.
I would concentrate on finding the cause of the SIGHUP.
DES
--
Dag-Erling Smørgrav - d...@des.no
Doug Freyburger
> Pascal Hambourg <boite-...@plouf.fr.eu.org> wrote:
>
>> Be aware that crossposting to multiple groups without a follow-up
>> reduces your chances to be read, as some servers reject such articles.
>
> Not in my experience.
It's not just the servers. It's the killfiles. In my experpience the
chances that a post is from a troll grows exponentially with the number
of groups cross-posted. I automatically kill all posts to 4+ groups in
my global filter list and it dramatically improved my UseNet experience.
One more group and I would never have seen it. For high noise groups I
filter out all cross-posted messages at all.
>>> Feb 28 00:04:51 WORK_SERVER_NAME sshd[5465]: Server listening on :: port 22.
>>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
>>> Feb 28 00:04:52 WORK_SERVER_NAME sshd[5465]: Received SIGHUP; restarting.
>
>>> My question is WTF, what could it be that keeps the port during boot
>>> time?
For the address to be in use some process already had to be using it.
Is sshd listed in the xinetd configs to run automatically? Is it also
in the rc scripts? If it is in both and there is a high number of ssh
connections happening on your network there's a reace condition - It is
possible for an incoming connection to happen before the rc scripts
reach that point. Since it's listed by xinetd it gets run.
> I am fairly positive that it is a very temporary, transient
> issue. That's why it does not happen all the time on all servers. In
> only (so far) occurred once on one server out of very many ones that
> reboot every week.
>
> Starting sshd again after 3 minutes should cover me pretty well.
>
> However, I am disappointed that such an essential (for accessing
> remote computers) function like ssh, could not initialize and gave
> up.
Try moving the sshd start to an earlier number and/or to a lower
numbered init state. Or leave it in xinetd and depend on that to launch
it to remove the race condition. Or common it out of xinetd and depend
on the rc script to run it at boot time.
If you comment it out of xinetd I suggest you use a meta-demon script.
Make a small script that runs sshd in forground in an infinite loop.
Then each time sshd exists the script runs a new one. It no longer
matters if the daemon dies because it tries again immediately. I
learned this trick from the O'Reily DNS book - It was used on one of the
root servers to keep BIND service running at all times.
> I was lucky that I could hack (kind of) my way into this server using
> its autoupdate scripts.
Console access with ILO or similar rules.
Nico Kadel-Garcia
> Ignoramus26630 wrote:
> If you comment it out of xinetd I suggest you use a meta-demon script.
> Make a small script that runs sshd in forground in an infinite loop.
> Then each time sshd exists the script runs a new one. It no longer
> matters if the daemon dies because it tries again immediately. I
> learned this trick from the O'Reily DNS book - It was used on one of the
> root servers to keep BIND service running at all times.
This sort of behavior is built into "daemontools", if you want to
bother to migrate SSH to running under that service.