"cannot access agent" error while deploying micro-bosh

[animesh nandanwar]

Hi All, 

While deploying a micro bosh instance, I  get a "cannot access agent" error. I did netstat on the microbosh instance, it shows that connection gets established on port 25889 and then eventually gets closed.

below are the logs 

root@inception-vm:/var/vcap/deployments# bosh micro deployment microbosh-openstack

WARNING! Your target has been changed to `http://microbosh-openstack:25555'!

Deployment set to '/var/vcap/deployments/microbosh-openstack/micro_bosh.yml'

root@inception-vm:/var/vcap/deployments# bosh micro deploy /var/vcap/stemcells/micro-bosh-stemcell-openstack-0.6.4.tgz

Deploying new micro BOSH instance `microbosh-openstack/micro_bosh.yml' to `http://microbosh-openstack:25555' (type 'yes' to continue): yes

Verifying stemcell...

File exists and readable                                     OK

Using cached manifest...

Stemcell properties                                          OK

Stemcell info

-------------

Name:    micro-bosh-stemcell

Version: 0.6.4

Deploy Micro BOSH

  unpacking stemcell (00:00:05)

  uploading stemcell (00:02:04)

  creating VM from 2944aac0-077f-4b92-833d-dff5232e5ff7 (00:01:00)

Waiting for the agent        |ooooo              | 3/11 00:19:14  ETA: --:--:--BOSH CLI Error: cannot access agent (Connection refused - connect(2) (http://10.0.0.3:6868))

Error information saved in /root/.bosh_error

On Bosh VM:

vcap@bm-6add4350-4fc5-4fe6-a538-6ed0c0b93b5d:~$ netstat

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      0 vm-1e12c276-8828-:45101 169.254.169.254:www     TIME_WAIT

tcp        0      0 vm-1e12c276-8828-:45102 169.254.169.254:www     TIME_WAIT

tcp        0      0 vm-1e12c276-8828-4b:ssh inception-vm.nova:48635 ESTABLISHED

tcp        0      0 vm-1e12c276-8828-:45103 169.254.169.254:www     TIME_WAIT

tcp        0      0 vm-1e12c276-8828-4b:ssh inception-vm.nova:35952 ESTABLISHED

tcp        0      0 vm-1e12c276-8828-:45107 169.254.169.254:www     ESTABLISHED

tcp        0      0 vm-1e12c276-8828-:45106 169.254.169.254:www     ESTABLISHED

tcp        0      0 vm-1e12c276-8828-:45105 169.254.169.254:www     ESTABLISHED

tcp6       0      0 localhost:55047         localhost:25889         ESTABLISHED

tcp6       0      0 localhost:25889         localhost:55047         ESTABLISHED

vcap@bm-6add4350-4fc5-4fe6-a538-6ed0c0b93b5d:~$ netstat

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      0 vm-1e12c276-8828-4b:ssh inception-vm.nova:48635 ESTABLISHED

tcp        0      0 vm-1e12c276-8828-4b:ssh inception-vm.nova:35952 ESTABLISHED

tcp        0      0 vm-1e12c276-8828-:45107 169.254.169.254:www     ESTABLISHED

tcp        0      0 vm-1e12c276-8828-:45106 169.254.169.254:www     ESTABLISHED

tcp        0      0 vm-1e12c276-8828-:45105 169.254.169.254:www     ESTABLISHED

tcp6       1      0 localhost:55047         localhost:25889         CLOSE_WAIT

tcp6       0      0 localhost:25889         localhost:55047         FIN_WAIT2

Thanks,

Animesh

[Martin Englund]

Animesh,

how is the security group you are using configured? You need the
following ports open from your where you run "bosh micro deploy":
- 22
- 6868 (agent http interface)
- 25555 (bosh director)

cheers,
/Martin

--
cheers,
/Martin
--
Martin Englund, Staff Engineer, Cloud Foundry, VMware Inc.
"The question is not if you are paranoid, it is if you are paranoid enough."

[animesh nandanwar]

Hi Martin,

The required ports are opened on inception vm as well as on BOSH vm. 

Thanks,

Animesh

[animesh nandanwar]

I did netstat on both inception VM and micro bosh vm. It shows that tcp connection first gets established on port 25889 and then it is closed down. It has no entry for port 6868. 

on Inception VM:

root@inception-vm:/var/vcap/deployments/microbosh-openstack# netstat

tcp        0      0 localhost:33497         localhost:25889         ESTABLISHED

tcp        0      0 localhost:25889         localhost:33497         ESTABLISHED

on microbosh vm:

vcap@bm-cafb0248-5c08-4da6-9aca-bd95b6068b86:~$ netstat

Active Internet connections (w/o servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        1      0 localhost:54149         localhost:25889         CLOSE_WAIT

tcp        0      0 localhost:25889         localhost:54149         FIN_WAIT2

Thanks,

Animesh

[nasa...@gmail.com]

Hi Animesh,

    Is this issue solved for you.  If so please share the solution.  I am getting the same error when trying to deploy following the instructions in the deploying to aws using cloud foundry bosh blog post. (http://blog.cloudfoundry.org/2012/09/06/deploying-to-aws-using-cloud-foundry-bosh/).  The console information is as follows

anbarasan@ubuntu:~/deployments$ bosh micro deploy ami-64d59436
Deploying new micro BOSH instance `aws/micro_bosh.yml' to `http://aws:25555' (type 'yes' to continue): yes
Deploy Micro BOSH
  using existing stemcell (00:00:00)                                            
  creating VM from ami-64d59436 (00:00:36)                                      
Waiting for the agent        |ooo                | 2/11 00:07:03  ETA: --:--:--BOSH CLI Error: cannot access agent (Connection refused - connect(2) (http://54.251.33.181:6868))
Error information saved in /home/anbarasan/.bosh_error

[animesh nandanwar]

No, Issue still persists. 

One more observation is this issue is seen with folsom release and not with Essex.

Thanks,

Animesh

[Ferran Rodenas]

Can you please paste your .bosh_error file?

- Ferdy

2012/9/17 animesh nandanwar <animeshn...@gmail.com>

[Martin Englund]

Can you ssh onto the VM and check the agent log for error messages:
/var/vcap/bosh/log/current

Cheers,
/Martin

On Mon, Sep 17, 2012 at 8:30 AM, animesh nandanwar

[animesh nandanwar]

Hi Ferdy,

I am currently on Essex so can not provide you logs. Will update you today EOD about required information.

May be  nasa...@gmail.com  can update, if he has system in failed state.

Thanks,

Animesh

[Anbarasan Gangadaran]

Hi Ferdy,

    find attached the .bosh_error file.

Thanks,

Anbarasan