Unable to bosh ssh in AWS - VPC environment

[asif.m...@gmail.com]

Hi Team,

I followed this doc to set up the Production cluster of CF in AWS VPC
http://docs.cloudfoundry.org/deploying/ec2/bootstrap-aws-vpc.html

The setup was successful and when i do the bosh vms , I could see that all the instances are running.
But when i try to bosh ssh into any of the instances it fails with connection timed out

[root@ip-10-7-140-111 micro]# bundle exec bosh ssh
1. nats_z2/0
2. uaa_z1/0
3. api_z1/0
4. clock_global/0
5. api_worker_z1/0
6. etcd_z1/0
7. etcd_z1/1
8. hm9000_z1/0
9. runner_z1/0
10. loggregator_z1/0
11. loggregator_trafficcontroller_z1/0
12. router_z1/0
13. acceptance_tests/0
14. smoke_tests/0
Choose an instance: 1
Enter password (use it to sudo on remote host): *****
Target deployment is `cf'

Setting up ssh artifacts

Director task 12

Task 12 done
Starting interactive shell on job nats_z2/0
ssh: connect to host 10.10.80.11 port 22: Connection timed out

Cleaning up ssh artifacts

Director task 13

Task 13 done

 I tried CF with EC2 Server alone, where I was able to ssh without any issues, but when i set it up in VPC it fails.
Am I missing any other parameter ? Could someone help this out?

Asif

[Sabha]

What if you provide the gateway details [--gateway_host HOST] [--gateway_user USER] when doing the ssh?

-Sabha

[asif.m...@gmail.com]

Hi Sabha,

I am not sure what to enter for the gateway_host and gateway_user details.
Is this available anywhere in AWS console or yml file ?

Asif

[Sabha]

If you are already have direct access to the network (since you are getting timeouts, I suspect not) then there is no need for specifying the gateway.

But if the cf related vms are running on a different network but you can communicate with them only via the bosh director (micro of full bosh) , then the bosh director becomes your gateway/jumpbox.

-Sabha

[asif.m...@gmail.com]

I used the following values for gateway host which i got it from route 53 configuration and the username I created during microbosh creation

bosh ssh --gateway_host micro.cf.cloud.com --gateway_user cloudadmin

It fails with 'Authentication failed with gateway micro.cf.cloud.com and user cloudadmin'

Asif

[Sabha]

if you had already logged in to the micro-bosh instance, then authentication should not have failed.

Check "bosh status" and provide its ip as the gateway_host and the login you used as the gateway_user.

-Sabha

[asif.m...@gmail.com]

The micro-bosh instance was created as part of the bootstrap command. wherein it created a 'bosh' key pair.
So currently i dont have the bosh pem file to login to the micro bosh instance.
Also, I used the same IP and login as mentioned in the bosh status when I tried before where it failed..

Asif

[sven...@gopivotal.com]

Hi Asif,

Can you try bundle exec bosh vms should list of all vms and ipaddress, from that directory you can ssh vcap@ipaddressofvm ? Can you try and let me know.

Thank you,
Sridhar

[asif.m...@gmail.com]

Hi Sridhar,

I thought i could use my own pem file by moving to the micro bosh instance - but I wasn't able to succeed.
In the process, the micro bosh instance stopped working so I had to tear down and start from scratch again.
i have been continuously tearing down and creating the vpc instance for past few days because of this issue.

Why do you think this issue happens?

I was able to target the URL, create org, space and user but was not able to push an application.
To debug, I wasn't able to ssh into any of the VMs.

I will try the ssh vcap@ipaddressofvm  once the cf is deployed.

Thanks for your help.
Asif

[asif.m...@gmail.com]

I tried with ssh vcap@10.xx.xx.18 .. it failed with connection timed out

Asif

[Daniel Lavine]

Can you try using the public ip address (or resolvable hostname) of the microbosh vm as the gateway_host and "vcap" as the gateway_user when you invoke bosh ssh.

bosh ssh --gateway_host micro.cf.cloud.com --gateway_user vcap nats_z1/0

You may need to add the ssh key to your keychain for this to work - if the bootstrap process created it for you, it should be in ~/.ssh/id_rsa_bosh and you can add it with the command

ssh-add ~/.ssh/id_rsa_bosh

--Dave and Dan

CF Community Pair

[asif.m...@gmail.com]

Great ! That works!

So I was not adding the id_rsa_bosh key to my keychain.. also during ssh-add i faced the below
'could not open a connection to your authentication agent'  issue
But then followed the instruction from here .. https://coderwall.com/p/rdi_wq  (for any poor souls like me)

Thanks a lot Dave and Dan

[asif.m...@gmail.com]

Now, I am unable to push any apps using cf push.
It fails with the following error message

Upload failed. Try again with 'cf push'.
CFoundry::BadResponse: :

With the -t trace message I could see that it starts - 'queued' - 'running' - 'failed'
Very similar to this issue -> https://github.com/cloudfoundry-community/bosh-cloudfoundry/issues/246
I tried their fix like restarting the api node but still the issue exists.. They also talk about restarting data node, which doesnt exists in aws vpc since its using rds

Asif

[Daniel Lavine]

This may be a wild guess . . . Check the values of

buildpack_directory_key

droplet_directory_key

app_package_directory_key

resource_directory_key

in your cf manifest. If they have '.' in the values, try changing all '.' to '-', redeploy, and try pushing your app again.

Dan && Dave

CF Community Pair

[asif.m...@gmail.com]

It works !
But how come you got through it.. I got the manifest file using spiff tool and I thought it will give the right configuration for the deployment.
I also checked the log files in CC but they looked good with out any errors.
How should i trouble-shoot such problems in future ?

Thanks so much for your help !

[ng....@gmail.com]

Hi asif & Dave and Dan,

I can't ssh to job vm too. And I don't see id_rsa_bosh in my .ssh folder. How should I do? 

Thanks for help,