ssh_exchange_identification: Connection closed by remote host

[John Almberg]

Hi guys,

Woke up this morning and discovered that one of my FreeBSD 7.2 servers
was down. When I try to SSH into the box, I get this:

~ 510 $ ssh m...@my.example.com
ssh_exchange_identification: Connection closed by remote host

FTP doesn't work, either, but the DNS server on the machine responds to
queries, and I can ping the box.

Any ideas on what might be the problem?

Thanks: John

_______________________________________________
freebsd-...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questi...@freebsd.org"

[Christer Solskogen]

On Fri, Jul 2, 2010 at 12:52 PM, John Almberg <jalm...@identry.com> wrote:

> ~ 510 $ ssh m...@my.example.com
> ssh_exchange_identification: Connection closed by remote host
>

Locked account, maybe?

--
chs,

[John Almberg]

Christer Solskogen wrote:
> On Fri, Jul 2, 2010 at 12:52 PM, John Almberg<jalm...@identry.com> wrote:
>
>
>> ~ 510 $ ssh m...@my.example.com
>> ssh_exchange_identification: Connection closed by remote host
>>
>>
>
> Locked account, maybe?
>

I've tried several accounts and they all give the same result. There's
also the fact that FTP and Apache seem to be broken, as well.

[Bas Smeelen]

On 07/02/2010 01:28 PM, John Almberg wrote:
> Christer Solskogen wrote:
>> On Fri, Jul 2, 2010 at 12:52 PM, John Almberg<jalm...@identry.com>
>> wrote:
>>
>>
>>> ~ 510 $ ssh m...@my.example.com
>>> ssh_exchange_identification: Connection closed by remote host
>>>
>>>
>>
>> Locked account, maybe?
>>
> I've tried several accounts and they all give the same result. There's
> also the fact that FTP and Apache seem to be broken, as well.

It could be that your /var filesystem filled up

DISCLAIMER:
This e-mail is for the intended recipient(s) only.
Access, disclosure, copying, distribution or reliance on any of it by anyone else is prohibited.
If you have received it by mistake please let us know by reply and then delete it from your system.

[John Almberg]

Bas Smeelen wrote:
> On 07/02/2010 01:28 PM, John Almberg wrote:
>
>> Christer Solskogen wrote:
>>
>>> On Fri, Jul 2, 2010 at 12:52 PM, John Almberg<jalm...@identry.com>
>>> wrote:
>>>
>>>
>>>
>>>> ~ 510 $ ssh m...@my.example.com
>>>> ssh_exchange_identification: Connection closed by remote host
>>>>
>>>>
>>>>
>>> Locked account, maybe?
>>>
>>>
>> I've tried several accounts and they all give the same result. There's
>> also the fact that FTP and Apache seem to be broken, as well.
>>
> It could be that your /var filesystem filled up
>

I'm on the console, now. Looks like a swapspace problem...

The first terminal is scrolling by the swapspace messages really fast (it kills httpd, but then starts again).
I tried logging in on the 2nd and 3rd virtual console, but hangs after I type root - never prompts for password.

Is there anything I can do besides rebooting?

On that subject... does Ctrl-Alt-Del initiate an orderly shutdown?

-- John

[Bas Smeelen]

>>>> Locked account, maybe?
>>>>
>>>>
>>> I've tried several accounts and they all give the same result. There's
>>> also the fact that FTP and Apache seem to be broken, as well.
>>>
>> It could be that your /var filesystem filled up
>>
> I'm on the console, now. Looks like a swapspace problem...
>
> The first terminal is scrolling by the swapspace messages really fast
> (it kills httpd, but then starts again).
> I tried logging in on the 2nd and 3rd virtual console, but hangs after
> I type root - never prompts for password.
>
> Is there anything I can do besides rebooting?
>
> On that subject... does Ctrl-Alt-Del initiate an orderly shutdown?
>

You can type shutdown -r now on the console, you probably won't see it
because of the messages but it will reboot your machine or you can use
ctrl+alt+del
Check your messages log when it's up again to see what has gone wrong
Good luck
bas

DISCLAIMER: This e-mail is for the intended recipient(s) only. Access, disclosure, copying,
distribution or reliance on any of it by anyone else is prohibited. If you have received it by
mistake please let us know by reply and then delete it from your system.

_______________________________________________

[Matthew Seaman]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 02/07/2010 12:44:48, John Almberg wrote:
> I'm on the console, now. Looks like a swapspace problem...
>
> The first terminal is scrolling by the swapspace messages really fast
> (it kills httpd, but then starts again).
> I tried logging in on the 2nd and 3rd virtual console, but hangs after I
> type root - never prompts for password.
>
> Is there anything I can do besides rebooting?
>
> On that subject... does Ctrl-Alt-Del initiate an orderly shutdown?

If you can't log in -- even on the console -- then rebooting is really
your only option. Ctrl-Alt-Del should bring the system down cleanly if
you haven't disabled that functionality. Otherwise, just toggle the power.

The symptoms you're seeing could well be due to filesystem problems or
to some filesystem filling up (/tmp is a prime suspect) or due to
running out of memory+swap. Some sort of memory leak sounds pretty
likely actually.

Probably best to bring the system up in single user mode and run fsck on
all the filesystems manually -- that will show if you've got h/w
problems with drives and possibly with disk controllers or cabling too.
Then check for overfull filesystems. You may not find any -- rebooting
can clear a number of conditions where disk space is not released back
to the OS properly after use. You may or may not find any clues as to
what went wrong in the system logs. In the absence of any other clues,
the only option is to monitor the server closely and wait for something
similar to happen again. Hopefully if there is a next time, you'll be
able to catch it and fix the underlying problem before it takes the
machine out a second time.

Cheers,

Matthew

- --
Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard
Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
JID: mat...@infracaninophile.co.uk Kent, CT11 9PW
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkwt1ZkACgkQ8Mjk52CukIwRVgCfXTHymgNMOwMN69H5NxwdTUsV
OjwAn2TPAgiHgW94+4swodm4mQbKhYIg
=iWlM
-----END PGP SIGNATURE-----

[John Almberg]

>> On that subject... does Ctrl-Alt-Del initiate an orderly shutdown?
>>
>
> If you can't log in -- even on the console -- then rebooting is really
> your only option. Ctrl-Alt-Del should bring the system down cleanly if
> you haven't disabled that functionality. Otherwise, just toggle the power.
>
> The symptoms you're seeing could well be due to filesystem problems or
> to some filesystem filling up (/tmp is a prime suspect) or due to
> running out of memory+swap. Some sort of memory leak sounds pretty
> likely actually.
>
> Probably best to bring the system up in single user mode and run fsck on
> all the filesystems manually -- that will show if you've got h/w
> problems with drives and possibly with disk controllers or cabling too.
> Then check for overfull filesystems. You may not find any -- rebooting
> can clear a number of conditions where disk space is not released back
> to the OS properly after use. You may or may not find any clues as to
> what went wrong in the system logs. In the absence of any other clues,
> the only option is to monitor the server closely and wait for something
> similar to happen again. Hopefully if there is a next time, you'll be
> able to catch it and fix the underlying problem before it takes the
> machine out a second time.
>
>

Yes, I can't log in. I get a login prompt, but no password prompt. I'm
going to try ctrl-alt-del and see what happens.

Crossing fingers...

[Bas Smeelen]

On 07/02/2010 02:09 PM, John Almberg wrote:
>

Sorry I missed that you can't login.
Good luck

DISCLAIMER: This e-mail is for the intended recipient(s) only. Access, disclosure, copying,
distribution or reliance on any of it by anyone else is prohibited. If you have received it
by mistake please let us know by reply and then delete it from your system.

_______________________________________________

[John Almberg]

So, ctrl-alt-del did the trick. I was able to log in and actually, the
whole box came up and everything seems to be working.

I thought for sure I'd find that my /var directory was full up, but it's
only at 77% (that's the weak spot on this box... I wish I'd made the
/var partition bigger.)

The message log is full of these messages:

38054 Jul 2 08:13:02 qu kernel: swap_pager_getswapspace(16): failed

If I run swapinfo, I get this:

[master@qu:log]> swapinfo
Device 1K-blocks Used Avail Capacity
/dev/ar0s1b 2055952 208 2055744 0%

I looked back in the log file to see if there were any clues when the
problem began and found this:

Jul 2 03:19:25 qu kernel: swap_pager: out of swap space
Jul 2 03:19:26 qu kernel: swap_pager_getswapspace(7): failed
Jul 2 03:19:26 qu kernel: pid 93543 (mysqld), uid 88, was killed: out
of swap space
Jul 2 03:19:26 qu kernel: pid 85077 (ruby18), uid 1023, was killed: out
of swap space
Jul 2 03:19:25 qu root: Check for bad ssh behavior
Jul 2 03:20:05 qu root: Check for bad ssh behavior
Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed
Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(15): failed
Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(14): failed
Jul 2 03:20:49 qu kernel: swap_pager_getswapspace(16): failed
Jul 2 03:20:49 qu last message repeated 2 times

It pretty much goes on forever from there.

Is there any other place I can look for clues as to why I ran out of
swap space? This machine is basically a webserver, running apache/mysql
and ruby on rails. It's been running for over a year with no problems.
No new software introduced on the box, recently.

-- John