Capistrano 3, ssh gateways and ssh timeouts, oh my

1,235 views

Skip to first unread message

Graham Ballantyne

unread,

Aug 28, 2014, 8:20:07 PM8/28/14

to capis...@googlegroups.com

(cross-post from http://stackoverflow.com/questions/25559870/capistrano-3-ssh-gateways-and-ssh-timeouts-oh-my)

I'm converting a project from Capistrano 2 to 3. The servers we deploy to are on a different network from our workstations and aren't directly accessible. We have a gateway server with legs into both network that we use. This set up worked fine in Capistrano 2 with the built-in gateway support.

I'm having issues with this in Capistrano 3. We have several environments (production/test/stage/etc) with varying numbers of servers (anywhere from 1 to 20). In the environment with only one server, my deploy works fine. When I try it in an environment with four servers, I get SSH timeout errors.

Configuration

In config/deploy.rb:

if (ENV.has_key?('gateway') && ENV['gateway'].downcase == "true")

require 'net/ssh/proxy/command'

gateway_user = ENV['gateway_user'] || ENV['USER']

set :ssh_options, {

proxy: Net::SSH::Proxy::Command.new("ssh #{gateway_user}@gatewayserver.externalnetwork -W %h:%p"),

forward_agent: true,

keys: [File.join(ENV["HOME"], ".ssh", "id_rsa_canvas")],

user: 'canvasuser'

}

end

In config/deploy/testing.rb:

role :app, %w{appserver1.internalnetwork appserver2.internalnetwork appserver3.internalnetwork}

role :db, %w{managementserver.internalnetwork}

I have a simple task defined that should print the hostname of each machine:

namespace :wtf do

task :hostname do

on roles(:all) do

execute :hostname

end

When I run that task, I get the following:

$ gateway=true bin/cap --trace testing wtf:hostname

Ruby 2.0 support is untested

** Invoke testing (first_time)

** Execute testing

** Invoke load:defaults (first_time)

** Execute load:defaults

** Invoke bundler:map_bins (first_time)

** Execute bundler:map_bins

** Invoke wtf:hostname (first_time)

** Execute wtf:hostname

INFO[d437644c] Running /usr/bin/env hostname on appserver1.internalnetwork

DEBUG[d437644c] Command: ( PATH=/usr/pgsql-9.1/bin:$PATH /usr/bin/env hostname )

INFO[43196a4b] Running /usr/bin/env hostname on appserver2.internalnetwork

DEBUG[43196a4b] Command: ( PATH=/usr/pgsql-9.1/bin:$PATH /usr/bin/env hostname )

INFO[ee99de0e] Running /usr/bin/env hostname on appserver3.internalnetwork

DEBUG[ee99de0e] Command: ( PATH=/usr/pgsql-9.1/bin:$PATH /usr/bin/env hostname )

INFO[abe3e148] Running /usr/bin/env hostname on managementserver.internalnetwork

DEBUG[abe3e148] Command: ( PATH=/usr/pgsql-9.1/bin:$PATH /usr/bin/env hostname )

INFO[43196a4b] Finished in 0.605 seconds with exit status 0 (successful).

DEBUG[43196a4b] appserver2.internalnetwork

INFO[43196a4b] Finished in 0.605 seconds with exit status 0 (successful).

INFO[d437644c] Finished in 10.524 seconds with exit status 0 (successful).

DEBUG[d437644c] appserver1.internalnetwork

INFO[d437644c] Finished in 10.524 seconds with exit status 0 (successful).

ssh: connect to host gatewayserver.externalnetwork port 22: Operation timed out

cap aborted!

SSHKit::Runner::ExecuteError: Exception while executing on host appserver3.internalnetwork: connection closed by remote host

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/server_version.rb:50:in `rescue in block (2 levels) in negotiate!'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/server_version.rb:46:in `block (2 levels) in negotiate!'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/server_version.rb:45:in `loop'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/server_version.rb:45:in `block in negotiate!'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/server_version.rb:43:in `loop'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/server_version.rb:43:in `negotiate!'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/server_version.rb:32:in `initialize'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:84:in `new'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:84:in `block in initialize'

/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/timeout.rb:52:in `timeout'

/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/timeout.rb:97:in `timeout'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:84:in `initialize'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh.rb:202:in `new'

vendor/bundle/ruby/2.0.0/gems/net-ssh-2.9.1/lib/net/ssh.rb:202:in `start'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:50:in `call'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:50:in `create_new_entry'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:22:in `checkout'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:179:in `with_ssh'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:131:in `block in _execute'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:128:in `tap'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:128:in `_execute'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:66:in `execute'

lib/capistrano/tasks/canvas.rake:65:in `block (3 levels) in <top (required)>'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:54:in `instance_exec'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:54:in `run'

vendor/bundle/ruby/2.0.0/gems/sshkit-1.5.1/lib/sshkit/runners/parallel.rb:13:in `block (2 levels) in execute'

Net::SSH::Disconnect: connection closed by remote host