- Jamis
- Jamis
- Jamis
On Jul 14, 2008, at 8:53 PM, matt wrote:
>
> Nope,
>
> ClientAliveInterval 15
> ClientAliveCountMax 15
>
> gives the following stack trace every time in the same place (when it
> tries to restart the 3rd of 3 app server instances)
>
> /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/buffered_io.rb:
> 98:in `send': closed stream (IOError)
> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
> buffered_io.rb:98:in `send_pending'
> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
> connection/
> session.rb:208:in `postprocess'
> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
> connection/
> session.rb:207:in `each'
> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
> connection/
> session.rb:207:in `postprocess'
> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/capistrano/
> processable.rb:31:in `process_iteration'
> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/capistrano/
> processable.rb:43:in `ensure_each_session'
> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/capistrano/
> processable.rb:41:in `each'
> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/capistrano/
> processable.rb:41:in `ensure_each_session'
> ... 75 levels...
> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/capistrano/cli/
> execute.rb:14:in `execute'
> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/bin/cap:4
> from /usr/bin/cap:19:in `load'
> from /usr/bin/cap:19
>
>
> or sometimes this trace earlier in the process:
>
> /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/connection/
> session.rb:523:in `channel_request': undefined method `do_request' for
> nil:NilClass (NoMethodError)
> And when I comment out these settings, things start working again (at
> least now during non-peak hours), so it seems these settings break cap
> (or net:ssh). In the man page for sshd_config, it states that these
> settings are for ssh protocol v2 only - is net::ssh v2? If it is,
> then maybe the async nature of the server polling back to the client
> is exposing a bug somewhere?
Net::SSH is v2, yeah. I suspect the absence of those settings just
means that the connection remains open until the underlying connection
times out, which is longer than the 60 seconds that you had originally
set.
> Also, the net:ssh docs mention have this intriguing example for
> Net::SSH::Connection:Session::send_global_request
> ssh.send_global_request("keep-...@openssh.com")
> If I can't get server side keep alives working, any tips on using this
> as a keep-alive mechanism on the client side?
I'd need to see your full deploy.rb to answer that one. Basically, at
certain points during your local processing, you'd need to do
something like:
sessions.values.each do |session|
session.send_global_request("keep-...@openssh.com")
end
That may not be possible, though, if your local processing involves
something like waiting for a text editor to finish (which is where
I've run into this in the past).
- Jamis
>
>
> Matt
>
>
> On Jul 14, 10:53 pm, matt <m...@mobicious.com> wrote:
>> Nope,
>>
>> ClientAliveInterval 15
>> ClientAliveCountMax 15
>>
>> gives the following stack trace every time in the same place (when it
>> tries to restart the 3rd of 3 app server instances)
>>
>> /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/buffered_io.rb:
>> 98:in `send': closed stream (IOError)
>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>> buffered_io.rb:98:in `send_pending'
>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>> connection/
>> session.rb:208:in `postprocess'
>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>> connection/
>> session.rb:207:in `each'
>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>> connection/
>> session.rb:207:in `postprocess'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>> processable.rb:31:in `process_iteration'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>> processable.rb:43:in `ensure_each_session'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>> processable.rb:41:in `each'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>> processable.rb:41:in `ensure_each_session'
>> ... 75 levels...
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/cli/
>> execute.rb:14:in `execute'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/bin/cap:4
>> from /usr/bin/cap:19:in `load'
>> from /usr/bin/cap:19
>>
>> or sometimes this trace earlier in the process:
>>
>> /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/connection/
>> session.rb:523:in `channel_request': undefined method `do_request'
>> for
>> nil:NilClass (NoMethodError)
>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>> connection/
>> session.rb:428:in `send'
>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>> connection/
>> session.rb:428:in `dispatch_incoming_packets'
>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>> connection/
>> session.rb:185:in `preprocess'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>> processable.rb:17:in `process_iteration'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>> processable.rb:43:in `ensure_each_session'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>> processable.rb:41:in `each'
>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>> capistrano/
>>>> processable.rb:31:in `process_iteration'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:43:in `ensure_each_session'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:41:in `each'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:41:in `ensure_each_session'
>>>> ... 75 levels...
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/cli/
>>>> execute.rb:14:in `execute'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/bin/cap:4
>>>> from /usr/bin/cap:19:in `load'
>>>> from /usr/bin/cap:19
>>
>>>> /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/connection/
>>>> session.rb:523:in `channel_request': undefined method
>>>> `do_request' for
>>>> nil:NilClass (NoMethodError)
>>>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>>>> connection/
>>>> session.rb:428:in `send'
>>>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>>>> connection/
>>>> session.rb:428:in `dispatch_incoming_packets'
>>>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>>>> connection/
>>>> session.rb:185:in `preprocess'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:17:in `process_iteration'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:43:in `ensure_each_session'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:41:in `each'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:41:in `ensure_each_session'
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/
>>>> processable.rb:17:in `process_iteration'
>>>> ... 74 levels...
>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>> capistrano/cli/
We had to deal with this in one of our recipes, where we throw up a
text editor and ask the deployer to describe the changes being
deployed. While the text editor was open, the connections were all
being starved, so if the deployer took too long composing their novel,
things would blow up. :(
The work around is this. Close all sessions before you start serially
restarting servers. After each one finishes restarting, close the
connection again. The worst that happens here is that cap just needs
to reconnect to those servers multiple times, which takes a few seconds.
It should work to just put the following at the top of serial_restart:
teardown_connections_to(sessions.keys)
Let me know if that makes any difference,
Jamis
>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>> capistrano/cli/
>>> execute.rb:14:in `execute'
>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/bin/cap:4
>>> from /usr/bin/cap:19:in `load'
>>> from /usr/bin/cap:19
>>
>>>>> processable.rb:31:in `process_iteration'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/
>>>>> processable.rb:43:in `ensure_each_session'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/
>>>>> processable.rb:41:in `each'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/
>>>>> processable.rb:41:in `ensure_each_session'
>>>>> ... 75 levels...
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/cli/
>>>>> execute.rb:14:in `execute'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/bin/cap:4
>>>>> from /usr/bin/cap:19:in `load'
>>>>> from /usr/bin/cap:19
>>
>>>>> /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/connection/
>>>>> session.rb:523:in `channel_request': undefined method `do_request'
>>>>> for
>>>>> nil:NilClass (NoMethodError)
>>>>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>>>>> connection/
>>>>> session.rb:428:in `send'
>>>>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>>>>> connection/
>>>>> session.rb:428:in `dispatch_incoming_packets'
>>>>> from /Library/Ruby/Gems/1.8/gems/net-ssh-2.0.3/lib/net/ssh/
>>>>> connection/
>>>>> session.rb:185:in `preprocess'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/
>>>>> processable.rb:17:in `process_iteration'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/
>>>>> processable.rb:43:in `ensure_each_session'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/
>>>>> processable.rb:41:in `each'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/
>>>>> processable.rb:41:in `ensure_each_session'
>>>>> from /Library/Ruby/Gems/1.8/gems/capistrano-2.4.3/lib/
>>>>> capistrano/