Failure on "cf bootstrap aws"

[David Welch]

Hey all, 

I'm following the relatively new instructions for getting CF2 running on AWS (http://docs.cloudfoundry.com/docs/running/deploying-cf/ec2/) and working my way through some of the kinks. Other than a few hiccups for things like naming collisions, all has been pretty smooth until now. 

When I reach the Deploy Cloud Foundry stage, I end up getting sunk on the actual deploy. The command that fails (bundle exec cf bootstrap aws) runs through a number of tasks but eventually ends up failing every time.

I've tried debugging the commands being run and gleaning what's going on, but am not sure what to check / where to look. Originally I failed on Task 5 (looks like the actual deploy step) with the following: Error 400007: `service_gateways/0' is not running after update

After poking around a bit, reading the docs & googling, I decided to try running the install again. This time it was much faster (as it didn't have to download, compile & push the 1.6GB platform again) but failed on step 8 with the same error. Another subsequent run got to step 11, but then the same error.

So, with all that in mind, any ideas on what I can do to fix the issue & deploy CF to AWS? Running the install over and over might get me there, but I'm definitely grinding gears and would like to know what / why I'm having issues.

I've read most the docs I've been able ti dig up, but if I need to just rtfm again I'm happy to do so, but a little direction on what I'm looking for would go a long way :)

Thanks! 

[rick.m...@gmail.com]

Hi David,

You are not alone. I followed the same instructions and I am getting the same error. I have also run it multiple times and the step keeps moving. My latest one was step 12.

I am still looking for the root cause.

[David Welch]

Hey Rick,

So, I've realized that (I think) it's the same step that is running over and over, but the task / step number keeps incrementing. I've rerun it a ton of times and haven't seen progress, so that probably won't get you much farther.

I came across a CF2 + AWS video today from Dr Nic (practically infamous in these parts) talking about building your own Heroku on CloudFoundry deployed to AWS. It looks like it's the base work for this guide (and hopefully has more details)

I haven't finished the whole video yet, but figure I'll follow the steps & advice given in it to see if there's better results. You may want to check it out at http://www.youtube.com/watch?v=e0EprkBamvQ -- and if you get things working, let me know what deity I need to send my sacrifices to :)  

Good luck!

[James Bayer]

I'm sorry everyone is having so much trouble with the bootstrap the last few days. Our early warning system that something is broken, which is typically CI, isn't doing an end-to-end test for AWS bootstrap clearly.  We have it for most other parts of our build process for ongoing deployments. I haven't completed the steps myself lately, but typically when something goes wrong using BOSH, you can use the "bosh vms" and "bosh ssh" commands to see which instances are running and navigate to them for further troubleshooting. 

To clear Dr Nic's name, his bosh-bootstrap tool was an inspiration for this tool, but his name shouldn't be muddied by the core CF team's dropped ball.

We'll provide an update on this thread soon.

--

Thank you,

James Bayer

[Dr Nic Williams]

David, I've been rewriting bosh-bootstrap (into inception-server and bosh-bootstrap v0.11) recently; and I'm now ready to upgrade bosh-cloudfoundry for the v2 components. Hope to have this done over the next week or so. If you can wait patiently :)

I'm not sure how the Pivotal team are managing their CF/VPC production deployment with the bootstrap tools they created for themselves. I think they are one-time tools I think so perhaps they aren't using them daily which makes motivation for maintenance hard.

They aren't cutting final releases of cf-release either, which makes it hard on everyone to talk about versions that work vs versions with critical bugs etc. Versioned releases of things (cf-release for example) is an important tool for sharing libraries/tools; so I assume it applies to sharing CF with those of us on this mailing list too. I may need to create a parallel cf-release with final releases published. Not sure yet. Hopefully pivotal start doing it asap; but I can't do it for them sadly.

--
Dr Nic Williams
Stark & Wayne LLC - the consultancy for Cloud Foundry
http://starkandwayne.com
+1 415 860 2185
twitter: drnic

On Wed, May 22, 2013 at 7:48 PM, David Welch <da...@davidwelch.co> wrote:

[Dr Nic Williams]

I've been thinking more and more recently that the future of bosh releases isn't to "cut a release" within the source repo itself. Rather you cut a release and its uploaded/stored somewhere else outside the source repo. Just like a RubyGem is stored in a gem server; not inside the git repo.

This means you only need the gem; not the source repo; to use the bosh release.

It also means that anyone can create their own releases; not just one person with credentials.

--
Dr Nic Williams
Stark & Wayne LLC - the consultancy for Cloud Foundry
http://starkandwayne.com
+1 415 860 2185
twitter: drnic

[rick.m...@gmail.com]

David - 

I'll definitely check the video out. I just saw how long it is so I will have to watch it in pieces. I'll let you know if I make any progress.

Dr Nic - 

I'll be happy to try out bosh-bootstrap again when you have it upgraded. Until then, I need to keep trying other things as waiting patiently is frowned upon in my company :) I also like where you are going with your bosh release ideas.

Thanks.

[Dr Nic Williams]

bosh-bootstrap itself is now upgraded; you can use it for all your bosh needs :)

​Next is the bosh-cloudfoundry project to make working with cf-release fun forever :)

--
Dr Nic Williams
Stark & Wayne LLC - the consultancy for Cloud Foundry
http://starkandwayne.com
+1 415 860 2185
twitter: drnic

[rick.m...@gmail.com]

Dr Nic,

I gave bosh-boostrap a try and it broke right after downloading the stemcell.

/Users/rmendes/.microbosh/deployments/firstbosh/last_successful_micro-bosh-stemcell-aws.tgz

/usr/local/var/rbenv/versions/1.9.3-p392/lib/ruby/1.9.1/rake/ext/module.rb:36:in `const_missing': uninitialized constant Bosh::Bootstrap::Cli::Helpers::Bundle::Bundler (NameError)

        from /usr/local/var/rbenv/versions/1.9.3-p392/gemsets/bosh-bootstrap/gems/bosh-bootstrap-0.11.3/lib/bosh-bootstrap/cli/helpers/bundle.rb:7:in `bundle'

[Dr Nic Williams]

Ahh a regression perhaps from removing bosh_cli as a dependency. Can you please create a shiny issue for this and I'll investigate the stemcell downloading aspect when I get home to real Internet.

​As an aside, does AWS/us-east-1 work (using an AMI) as expected?

​I'll double check the scope of the CI integration tests too when I get back.

--
Dr Nic Williams
Stark & Wayne LLC - the consultancy for Cloud Foundry
http://starkandwayne.com
+1 415 860 2185
twitter: drnic

[Ferran Rodenas]

Rick, seems a bundler dependency error. Can you please check if you have the bundler gem installed? If not just do a "gem install bundler" and try again.

- Ferdy

2013/5/23 Dr Nic Williams <drnicw...@gmail.com>

[rick.m...@gmail.com]

Ferdy -

$ gem list

*** LOCAL GEMS ***

bosh-bootstrap (0.11.3)

builder (3.2.0)

bundler (1.3.5)

...

Dr Nic -

I'll retry with us-east-1 and open an issue with what I see with that and my initial try with us-west-1.

Thanks.

[rick.m...@gmail.com]

Dr Nic

Issue #201 is open for this and I was able to reproduce the problem with us-east-1.

Thanks.

[Dr Nic Williams]

For anyone else with this issue, it has been patched in v0.11.4

--
Dr Nic Williams
Stark & Wayne LLC - the consultancy for Cloud Foundry
http://starkandwayne.com
+1 415 860 2185
twitter: drnic

[Christian Dupuis]

James, any chance for an update? This is stopping everyone from following the published steps and getting a system up and running.

Thanks, Christian

[David Stevenson]

Hey guys,

I successfully ran "cf bootstrap aws" yesterday.  The service_gateways/0 not running issue is usually that they can't communicate with CloudController.  That can be a result of DNS not being set up correctly for your VPC domain and subdomain, or maybe a problem with CC itself.  See if you can curl the CC API by going to api.SUBDOMAIN.DOMAIN (for example, http://api.david.cf-app.com/info).  If you can't, check that DNS is wired such that *.SUBDOMAIN.DOMAIN routes to your amazon ELB and ensure that DOMAIN delegates all nameserving for SUBDOMAIN to amazon route53.  The domain delegation step is a manual step, and I think it's where people usually make a mistake.

Can someone let me know if you get this working?

-David Stevenson

Director of Engineering, CF Services

[pjpe...@gmail.com]

Hi David,

We are getting the same error two: 

Error 400007: `service_gateways/0' is not running after update

I've checked DNS and cloud controller api and it seems to work:

curl http://api.hackathoncloud.zks.org/info

{"name":"vcap","build":"2222","support":"http://support.cloudfoundry.com","version":2,"description":"Cloud Foundry sponsored by Pivotal","authorization_endpoint":"http://login.hackathoncloud.zks.org","token_endpoint":"http://uaa.hackathoncloud.zks.org","allow_debug":true}

Peng

[David Welch]

Hi David,

​That sounds like it could have been the cause. I followed the instructions on the route 53 setup but wasn't sure if I made the changes correctly in the GoDaddy DNS manager. 

I'm out of town for the weekend but will try it again ASAP. 

As I'm not great with the networking side of things, if anyone could give me a bit more detail on the DNS setup I'd appreciate it. I believe I added NS records for "@" with the values created in the route 53 zone. Does that Sound correct?

—
Sent from Mailbox for iPhone

[rick.m...@gmail.com]

Hi David Stevenson,

You solved our problem! Our network setup was incorrect. I fixed that, which got data coming back from the api call. Better, that got me a clean run of 'bundle exec cf bootstrap aws'.

Hi David Welch,

Our networking is pretty simple. In the AWS account we are deploying to, I added a hosted zone for our subdomain. Nothing fancy there. I gave it the full domain name we wanted - mysubdomain.mydomain.com. It picked its own set of name servers.

In another AWS account where we manage the top-level domain, I selected the domain and went into its record sets. I added a new record set there of type NS, the full domain to match the other account, the same name servers that were picked in the other account and simple routing.

Once all of that was saved, the curl command worked and the bosh command worked. 

[Matt Reider]

Rick / David Welch,

Would you mind giving feedback or editing the instructions based on what you learned. I would like to make this clearer.

Assuming of course that you arrived at the point where you could target and push an app. I hope so.

Sent from my iPhone

[rick.m...@gmail.com]

I'll post what I have learned so far at some point today. Once I get one of our sample apps deployed I will post anything I learn with that too.

Thanks.

[rick.m...@gmail.com]

mreider -

I opened a PR with changes to the instructions based on what I learned.

Thanks.

[James Bayer]

Awesome, thanks for doing that!

[Matt Reider]

Thanks so much!

Sent from my iPhone