SSH to BOSH VM instance fails

[animesh nandanwar]

Hi All,

After I do " bosh micro deploy /var/vcap/stemcells/micro-bosh-stemcell-openstack-0.6.4.tgz" , the deployment gets stuck at "waiting for agent" stage. I checked the "bosh_micro_deploy.log" below the messages from it. SSH fails for the created VM and I tried manually doing ssh vc...@10.0.0.4 to created vm but it still fails.  I am not able to get over it :-(. 

Any help will be highly appreciated. 

Thanks,

Animesh

 

LOGS:

I, [2012-08-27T20:41:54.239875 #11884] [create_vm(bm-999d4c6b-195d-41c6-bd22-96797b239764, ...)]  INFO -- : Configuring network for `3c899b3d-2a72-4cee-8b78-f763ce791369'

I, [2012-08-27T20:41:54.273320 #11884] [create_vm(bm-999d4c6b-195d-41c6-bd22-96797b239764, ...)]  INFO -- : Updating server settings for `3c899b3d-2a72-4cee-8b78-f763ce791369'

I, [2012-08-27T20:41:54.406014 #11884] [0x932eac]  INFO -- : discovered bosh ip=10.0.0.4

D, [2012-08-27T20:41:56.074089 #11884] [0x932eac] DEBUG -- : tcp socket 10.0.0.4:22 SystemCallError: #<Errno::ECONNREFUSED: Connection refused - connect(2)>

D, [2012-08-27T20:41:57.112989 #11884] [0x932eac] DEBUG -- : tcp socket 10.0.0.4:22 is readable

I, [2012-08-27T20:42:57.113436 #11884] [0x932eac]  INFO -- : Preparing for ssh tunnel: ssh -R 25889:127.0.0.1:25889vcap@10.0.0.4

D, [2012-08-27T20:42:57.309720 #11884] [0x932eac] DEBUG -- : ssh start vc...@10.0.0.4 failed: #<Net::SSH::AuthenticationFailed: vcap>

D, [2012-08-27T20:42:58.444499 #11884] [0x932eac] DEBUG -- : ssh start vc...@10.0.0.4 failed: #<Net::SSH::AuthenticationFailed: vcap>

D, [2012-08-27T20:42:59.564189 #11884] [0x932eac] DEBUG -- : ssh start vc...@10.0.0.4 failed: #<Net::SSH::AuthenticationFailed: vcap>

D, [2012-08-27T20:43:00.681202 #11884] [0x932eac] DEBUG -- : ssh start vc...@10.0.0.4 failed: #<Net::SSH::AuthenticationFailed: vcap>

Thanks,

Animesh

[吴镇乐]

囧 I get the same error...

2012/8/29 animesh nandanwar <animeshn...@gmail.com>

[animesh nandanwar]

Then I am not the only one to face this !! 

I remember that it worked previously.

Thanks,

Animesh

[Vadim Spivak]

Which CPI is using?

Thanks,

Vadim

[animesh nandanwar]

Oh Sorry Vadim,  forgot to mention that !!

I am using Openstack-bosh-cpi and ssh fails after creation of VM during "bosh micro deploy /var/vcap/stemcells/micro-bosh-stemcell-openstack-0.6.4.tgz" i.e. deployment does not get hold of agent.

I am using "https://github.com/drnic/bosh-getting-started/blob/master/create-a-bosh/creating-a-micro-bosh-from-stemcell-openstack.md" this tutorial for deploying Micro-Bosh.

Thanks,

Animesh

[Vadim Spivak]

Can you try to login to the Micro-Bosh VM using the password used to create the SALTED_PASSWORD? 

Then verify if the public key was properly installed in /home/vcap/.ssh/authorized_keys?

Thanks,

Vadim

[吴镇乐]

I try to deploy again,then it works.I don't know why this heppen,but thanks gad.

2012/8/30 Vadim Spivak <vsp...@vmware.com>

[animesh nandanwar]

Hi Ferdy,
Still I can not ssh using salted password.

I tried following 

1. ssh vc...@10.0.0.4 -i /home/vcap/.ssh/id_rsa.pub

2. ssh vc...@10.0.0.4 -i /home/vcap/.ssh/inception.pem

Everytime it asks for password, for which I used salted password.

Thanks,

Animesh

[Vadim Spivak]

The default password is cloudc0w, can you see if that works?

Thanks,
Vadim

[animesh nandanwar]

Vadim, That also does not work.

How exactly user been setup at BOSH VM ? and how inception.pem is used in authenticating vcap user ?

Thanks,

Animesh

[gabi sweda]

Did you verify the networking? I have had this happen when the VLAN was not routed correctly. Are you able to get on console and try login with the c1oudc0w password?

gabi

> I, [2012-08-27T20:42:57.113436 #11884] [0x932eac] INFO -- : Preparing for ssh tunnel: ssh -R 25889:127.0.0.1:25889vc...@10.0.0.4

[animesh nandanwar]

Yes, I think vlan has been been properly setup. The funny thing is when I do "root@inception-vm: ssh 10.0.0.4 -i /home/vcap/.ssh/inception.pem" it works.

Thanks,

Animesh

[Vadim Spivak]

If that works then net-ssh should also succeed since it should be using the same private key. 

https://github.com/cloudfoundry/bosh/blob/master/deployer/lib/deployer/instance_manager_helpers.rb#L54

Can you try to run that snippet by itself?

e.g.:

require 'rubygems'

require 'net/ssh'

ip = "X.Y.Z"

ssh_user = "vcap"

ssh_key = "/path/to/private/key.pem"

loop do

  begin

    Net::SSH.start(ip, ssh_user, :keys => [ssh_key], :paranoid => false)

    logger.debug("ssh #{ssh_user}@#{ip}: ESTABLISHED")

    break

  rescue => e

    logger.debug("ssh start #{ssh_user}@#{ip} failed: #{e.inspect}")

    sleep 1

  end

end

(I haven't tested the snippet, so there might be some typos)

[animesh nandanwar]

Hi Vadim, 

SSH still fails after this snippet. But I can ssh with default password "c1oudc0w". 

I am not sure what is going wrong.

Do you suggest to use default password as a salted password as a workaround ?

Thanks,

Animesh 

[Vadim Spivak]

That's bizzarre, so you can on the same machine use ssh from the command line with the private key only, however when you run the snippet above it fails?

-Vadim

[animesh nandanwar]

This amazes me as well.

This way it succeeds "ssh -R 25889:127.0.0.1:25889 vc...@10.0.0.2" with password "c1oudc0w" but below snippet fails.

Something is really going wrong here.  

require 'rubygems'

require 'net/ssh'

ip = "10.0.0.2"

ssh_user = "vcap"

ssh_key = "/home/vcap/.ssh/inception.pem"

loop do

  begin

    Net::SSH.start(ip, ssh_user, :keys => [ssh_key], :paranoid => false)

    puts "ssh #{ssh_user}@#{ip}: ESTABLISHED"

    break

  rescue => e

    puts "ssh start #{ssh_user}@#{ip} failed: #{e.inspect}"

    sleep 1

  end

end

[Vadim Spivak]

Wait, does it succeed with just "ssh 10.0.0.4 -i /home/vcap/.ssh/inception.pem" without any password?

Thanks,

Vadim

[animesh nandanwar]

Yes, it does succeed without any password.

[Doug MacEachern]

Looks like that command is using root as the login user, need to test
with user vcap:
ssh vc...@10.0.0.4 -i /home/vcap/.ssh/inception.pem

If that doesn't work, check the permissions on
10.0.0.4:/home/vcap/.ssh (should be 0700) and contents of
/home/vcap/.ssh/authorized_keys

If this is the problem, looks like it is fixed here:
https://github.com/cloudfoundry/bosh/commit/45a22e320d63fd2419b6486ae6d8e6d4726755f8

But, this fix came after the current bosh-release that's included in
stemcell 0.6.4

On Thu, Aug 30, 2012 at 2:23 PM, animesh nandanwar
<animeshn...@gmail.com> wrote:

[animesh nandanwar]

Thanks Doug,

I checked permissions on 10.0.0.4:/home/vcap/.ssh  and it is not 0700.

Also "ssh vc...@10.0.0.4 -i /home/vcap/.ssh/inception.pem" succeeds but only with salted password.

Thanks,

Animesh

[吴镇乐]

chmod vcap:vcap /home/vcap/.ssh on 10.0.0.4 solve the problem

2012/9/1 animesh nandanwar <animeshn...@gmail.com>

[吴镇乐]

Sorry , It's command " chown vcap:vcap /home/vcap/.ssh on 10.0.0.4"

2012/9/4 吴镇乐 <wuzh...@gmail.com>

[animesh nandanwar]

"chown vcap:vcap /home/vcap/.ssh" on 10.0.0.4 solves the issue and I can now ssh to 10.0.0.4.

But right now I am manually doing this on Bosh instance while deployment is waiting for agent. And during this,  deployment does not get hold of an agent. 

root@inception-vm:/var/vcap/deployments# bosh micro deployment microbosh-openstack

WARNING! Your target has been changed to `http://microbosh-openstack:25555'!

Deployment set to '/var/vcap/deployments/microbosh-openstack/micro_bosh.yml'

root@inception-vm:/var/vcap/deployments# bosh micro deploy /var/vcap/stemcells/micro-bosh-stemcell-openstack-0.6.4.tgz

Deploying new micro BOSH instance `microbosh-openstack/micro_bosh.yml' to `http://microbosh-openstack:25555' (type 'yes' to continue): yes

Verifying stemcell...

File exists and readable                                     OK

Using cached manifest...

Stemcell properties                                          OK

Stemcell info

-------------

Name:    micro-bosh-stemcell

Version: 0.6.4

Deploy Micro BOSH

  unpacking stemcell (00:00:05)

  uploading stemcell (00:02:01)

  creating VM from c25be723-05fd-4131-a325-557542b6ebbe (00:00:52)

Waiting for the agent        |ooooo              | 3/11 00:13:42  ETA: --:--:--BOSH CLI Error: cannot access agent (Connection refused - connect(2) (http://10.0.0.3:6868))

Error information saved in /root/.bosh_error

root@inception-vm:/var/vcap/deployments#