Running "bosh aws create" ended with AWS::S3::Errors::BucketAlreadyOwnedByYou

139 views
Skip to first unread message

Евгений Костенко

unread,
Jul 9, 2014, 12:45:47 PM7/9/14
to bosh-...@cloudfoundry.org
Hello, All!

I'm trying to create AWS infrastructure via running "bosh aws create". Env variables are loaded from shell script:

$ cat env.sh 
export BOSH_VPC_DOMAIN=domain.com
export BOSH_VPC_SUBDOMAIN=cf # Pick something more unique than 'cloud' to work around a temporary shortcoming of the tool
export BOSH_AWS_ACCESS_KEY_ID=<CUT>
export BOSH_AWS_SECRET_ACCESS_KEY=<CUT>
export BOSH_AWS_REGION=eu-west-1
export BOSH_VPC_PRIMARY_AZ=eu-west-1a   # see note below
export BOSH_VPC_SECONDARY_AZ=eu-west-1c # see note below

Here is bosh gem's:

$ gem list bosh 

*** LOCAL GEMS ***

bosh-director-core (1.2624.0)
bosh-registry (1.2624.0)
bosh-stemcell (1.2624.0)
bosh_aws_cpi (1.2624.0)
bosh_cli (1.2624.0, 1.2611.0)
bosh_cli_plugin_aws (1.2624.0)
bosh_cli_plugin_micro (1.2624.0)
bosh_common (1.2624.0, 1.2611.0)
bosh_cpi (1.2624.0)
bosh_openstack_cpi (1.2624.0)
bosh_vcloud_cpi (0.5.4)
bosh_vsphere_cpi (1.2624.0)

Every retry in Ireland (us-west-1)  ended with such error:

<SKIPPED>
Executing migration CreateMoreUniqueS3Buckets
creating bucket cf-domain-com-bosh-blobstore
creating bucket cf-domain-com-bosh-artifacts
/cloud/.rvm/gems/ruby-1.9.3-p547/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:368:in `return_or_raise': Your previous request to create the named bucket succeeded and you already own it. (AWS::S3::Errors::BucketAlreadyOwnedByYou)
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/aws-sdk-1.32.0/lib/aws/core/client.rb:469:in `client_request'
        from (eval):3:in `create_bucket'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/aws-sdk-1.32.0/lib/aws/s3/bucket_collection.rb:109:in `create'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/lib/bosh_cli_plugin_aws/s3.rb:9:in `create_bucket'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/migrations/20130529212130_create_more_unique_s3_buckets.rb:22:in `block in execute'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/migrations/20130529212130_create_more_unique_s3_buckets.rb:20:in `each_key'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/migrations/20130529212130_create_more_unique_s3_buckets.rb:20:in `execute'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/lib/bosh_cli_plugin_aws/migration.rb:21:in `run'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/lib/bosh_cli_plugin_aws/migrator.rb:75:in `block in run_migrations'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/lib/bosh_cli_plugin_aws/migrator.rb:74:in `each'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/lib/bosh_cli_plugin_aws/migrator.rb:74:in `run_migrations'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/lib/bosh_cli_plugin_aws/migrator.rb:13:in `migrate'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli_plugin_aws-1.2624.0/lib/bosh/cli/commands/aws.rb:130:in `create'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli-1.2624.0/lib/cli/command_handler.rb:57:in `run'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli-1.2624.0/lib/cli/runner.rb:56:in `run'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli-1.2624.0/lib/cli/runner.rb:16:in `run'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/gems/bosh_cli-1.2624.0/bin/bosh:7:in `<top (required)>'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/bin/bosh:23:in `load'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/bin/bosh:23:in `<main>'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/bin/ruby_executable_hooks:15:in `eval'
        from /cloud/.rvm/gems/ruby-1.9.3-p547/bin/ruby_executable_hooks:15:in `<main>'

As i understand it's from bosh_cli_plugin_aws-1.2624.0/migrations/20130529212130_create_more_unique_s3_buckets.rb.
But i'm sure, that this buckets doesn't exists when running bosh aws create (i've delete it previously and rechecked it twice O_o).

Same command run successfully in N.Virginia (us-east-1) region and bosh plus whole CF was deployed after this.
I've tried to update bosh gems and dependencies, but it's doesn't helped.

This is known bug or i'm miss something?

Dr Nic Williams

unread,
Jul 9, 2014, 12:47:25 PM7/9/14
to bosh-...@cloudfoundry.org, bosh-...@cloudfoundry.org
Did you literally use domain.com? If so, replace it with a hostname that you own that can have *.hostname wildcard DNS.

To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

Евгений Костенко

unread,
Jul 9, 2014, 12:52:38 PM7/9/14
to bosh-...@cloudfoundry.org
Nope, i've just replaced our domain by security considerations. I mean, i've used cf.OurDomainNameHere.com in bosh env variables.

среда, 9 июля 2014 г., 20:47:25 UTC+4 пользователь Dr Nic Williams написал:

Dr Nic Williams

unread,
Jul 9, 2014, 12:57:37 PM7/9/14
to bosh-...@cloudfoundry.org, bosh-...@cloudfoundry.org
Sorry :) I just wanted to sanity check :)

Евгений Костенко

unread,
Jul 10, 2014, 7:23:34 AM7/10/14
to bosh-...@cloudfoundry.org
Yep, but any other suggestions? 
Except "bosh aws destroy" and retry "bosh aws create".

среда, 9 июля 2014 г., 20:57:37 UTC+4 пользователь Dr Nic Williams написал:

Dr Nic Williams

unread,
Jul 10, 2014, 9:39:18 AM7/10/14
to bosh-...@cloudfoundry.org, bosh-...@cloudfoundry.org
No sorry. It's not a tool I use nor track changes for.

Евгений Костенко

unread,
Jul 10, 2014, 9:56:34 AM7/10/14
to bosh-...@cloudfoundry.org
I guess, that it's issue from deeps of aws-sdk. Somewere around gems/bosh_cli_plugin_aws-1.2624.0/migrations/20130529212130_create_more_unique_s3_buckets.rb:22

    buckets.each_key do |bucket|
      say "creating bucket #{bucket}"
      s3.create_bucket(bucket) # <<< here is a problem
    end

Because of BucketAlreadyOwnedByYou description from AWS docs (http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html):

Your previous request to create the named bucket succeeded and you already own it. 
You get this error in all AWS regions except US Standard, us-east-1. 
In us-east-1 region, you will get 200 OK, but it is no-op (if bucket exists it Amazon S3 will not do anything).

Does "bosh aws create" was tested any other region except us-east-1? 
There is any mentions about such limitations in CF community docs?

Such unexpected and unexplained behaviour broke my brain 8(

четверг, 10 июля 2014 г., 17:39:18 UTC+4 пользователь Dr Nic Williams написал:

Евгений Костенко

unread,
Jul 11, 2014, 7:43:12 AM7/11/14
to bosh-...@cloudfoundry.org
Reported this issue to upstream - https://github.com/cloudfoundry/bosh/issues/614. I'm hope, it's will be fixed someday. 
Because there is big list of major issues, which you'll be faced if you are using CF in AWS within any region except us-east-1.
Starting with "there is no fresh version of stemcells", ending with "bosh aws bootstrap micro" which is trying to spawn an EC2 instance using non-existing AMI.

четверг, 10 июля 2014 г., 17:39:18 UTC+4 пользователь Dr Nic Williams написал:

Greg Oehmen

unread,
Jul 11, 2014, 6:55:30 PM7/11/14
to bosh-users
My understanding is that us-east-1 is hardcoded in `bosh aws create`.  At this time, there are no plans to modify that to include other AZs.  The decisions surrounding this predate me and I've never pursued it.  I'll do some exploration and reply with what I find out.

Best 

Greg

Greg Oehmen
Cloud Foundry Product Manager - Bosh
Pivotal

Евгений Костенко

unread,
Jul 12, 2014, 5:12:32 AM7/12/14
to bosh-...@cloudfoundry.org
I've didn't found any "hardcoded" id's of AWS regions related in my issue. 
I'm talking about changing one line, which is responsible for creating S3 bucket when migrations is running under "bosh aws create".

May be it's caused by inattention of dev, who is possibly didn't read details about create_bucket() in AWS S3.
But honestly saying, i'm too was very surprised, when found such unexpected behaviour of simple "create" API call ;)

PS: And this is not end of story "it's not works outside of us-east-1". For example, when you running "bosh aws bootstrap micro" within eu-west-1 (i.e. Ireland), bosh is tried to use AMI, which doesn't exists nowhere outside of N.Virginia. 

суббота, 12 июля 2014 г., 2:55:30 UTC+4 пользователь Greg Oehmen написал:

James Masson

unread,
Jul 14, 2014, 4:10:13 AM7/14/14
to bosh-...@cloudfoundry.org

Pretty much only US-East-1 is supported in BOSH/CF at the moment. It is
possible to use other regions, but you have to be aware of some of the
quirks.

eg. no region-local AMIs and various Blobstore handling oddities -
https://github.com/cloudfoundry/bosh/issues/463


James M
> <https://github.com/cloudfoundry/bosh/issues/614>. I'm hope,
> it's will be fixed someday.
> Because there is big list of major issues, which you'll be faced
> if you are using CF in AWS within any region except us-east-1.
> Starting with "there is no fresh version of stemcells", ending
> with "bosh aws bootstrap micro" which is trying to spawn an EC2
> instance using non-existing AMI.
>
> четверг, 10 июля 2014 г., 17:39:18 UTC+4 пользователь Dr Nic
> Williams написал:
>
> No sorry. It's not a tool I use nor track changes for.
>
> On Thu, Jul 10, 2014 at 4:23 AM, Евгений Костенко
> <nobod...@gmail.com> wrote:
>
> Yep, but any other suggestions?
> Except "bosh aws destroy" and retry "bosh aws create".
>
> среда, 9 июля 2014 г., 20:57:37 UTC+4 пользователь Dr
> Nic Williams написал:
>
> Sorry :) I just wanted to sanity check :)
>
>
>
> On Wed, Jul 9, 2014 at 9:52 AM, Евгений Костенко
> <nobod...@gmail.com> wrote:
>
> Nope, i've just replaced our domain by security
> considerations. I mean, i've used
> cf.OurDomainNameHere.com
> <http://cf.OurDomainNameHere.com> in bosh env
> variables.
>
> среда, 9 июля 2014 г., 20:47:25 UTC+4
> пользователь Dr Nic Williams написал:
>
> Did you literally use domain.com
> <http://domain.com>? If so, replace it with
> a hostname that you own that can have
> *.hostname wildcard DNS.
>
> On Wed, Jul 9, 2014 at 9:45 AM, Евгений
> Костенко <nobod...@gmail.com> wrote:
>
> Hello, All!
>
> I'm trying to create AWS infrastructure
> via running "bosh aws create". Env
> variables are loaded from shell script:
>
> $ cat env.sh
> export BOSH_VPC_DOMAIN=domain.com
> <http://domain.com>
> export BOSH_VPC_SUBDOMAIN=cf # Pick
> something more unique than 'cloud' to
> work around a temporary shortcoming of
> the tool
> export BOSH_AWS_ACCESS_KEY_ID=<CUT>
> export BOSH_AWS_SECRET_ACCESS_KEY=<__CUT>
> export BOSH_AWS_REGION=eu-west-1
> export BOSH_VPC_PRIMARY_AZ=eu-west-1a
> # see note below
> export
> BOSH_VPC_SECONDARY_AZ=eu-west-__1c # see
> note below
>
> Here is bosh gem's:
>
> $ gem list bosh
>
> *** LOCAL GEMS ***
>
> bosh-director-core (1.2624.0)
> bosh-registry (1.2624.0)
> bosh-stemcell (1.2624.0)
> bosh_aws_cpi (1.2624.0)
> bosh_cli (1.2624.0, 1.2611.0)
> bosh_cli_plugin_aws (1.2624.0)
> bosh_cli_plugin_micro (1.2624.0)
> bosh_common (1.2624.0, 1.2611.0)
> bosh_cpi (1.2624.0)
> bosh_openstack_cpi (1.2624.0)
> bosh_vcloud_cpi (0.5.4)
> bosh_vsphere_cpi (1.2624.0)
>
> Every retry in Ireland (us-west-1)
> ended with such error:
>
> <SKIPPED>
> Executing migration
> CreateMoreUniqueS3Buckets
> creating bucket cf-domain-com-bosh-blobstore
> creating bucket cf-domain-com-bosh-artifacts
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/aws-sdk-1.32.0/lib/__aws/core/client.rb:368:in
> `return_or_raise': Your previous request
> to create the named bucket succeeded and
> you already own it.
> (AWS::S3::Errors::__BucketAlreadyOwnedByYou)
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/aws-sdk-1.32.0/lib/__aws/core/client.rb:469:in
> `client_request'
> from (eval):3:in `create_bucket'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/aws-sdk-1.32.0/lib/__aws/s3/bucket_collection.rb:__109:in
> `create'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/lib/bosh_cli_plugin___aws/s3.rb:9:in
> `create_bucket'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/migrations/__20130529212130_create_more___unique_s3_buckets.rb:22:in
> `block in execute'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/migrations/__20130529212130_create_more___unique_s3_buckets.rb:20:in
> `each_key'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/migrations/__20130529212130_create_more___unique_s3_buckets.rb:20:in
> `execute'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/lib/bosh_cli_plugin___aws/migration.rb:21:in
> `run'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/lib/bosh_cli_plugin___aws/migrator.rb:75:in
> `block in run_migrations'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/lib/bosh_cli_plugin___aws/migrator.rb:74:in
> `each'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/lib/bosh_cli_plugin___aws/migrator.rb:74:in
> `run_migrations'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/lib/bosh_cli_plugin___aws/migrator.rb:13:in
> `migrate'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli_plugin_aws-__1.2624.0/lib/bosh/cli/__commands/aws.rb:130:in
> `create'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli-1.2624.0/__lib/cli/command_handler.rb:57:__in
> `run'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli-1.2624.0/__lib/cli/runner.rb:56:in
> `run'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli-1.2624.0/__lib/cli/runner.rb:16:in
> `run'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/gems/bosh_cli-1.2624.0/__bin/bosh:7:in
> `<top (required)>'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/bin/bosh:23:in
> `load'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/bin/bosh:23:in
> `<main>'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/bin/ruby_executable___hooks:15:in
> `eval'
> from
> /cloud/.rvm/gems/ruby-1.9.3-__p547/bin/ruby_executable___hooks:15:in
> `<main>'
>
> As i understand it's
> from bosh_cli_plugin_aws-1.__2624.0/migrations/__20130529212130_create_more___unique_s3_buckets.rb.
> But i'm sure, that this buckets doesn't
> exists when running bosh aws create
> (i've delete it previously and rechecked
> it twice O_o).
>
> Same command run successfully in
> N.Virginia (us-east-1) region and bosh
> plus whole CF was deployed after this.
> I've tried to update bosh gems and
> dependencies, but it's doesn't helped.
>
> This is known bug or i'm miss something?
>
> To unsubscribe from this group and stop
> receiving emails from it, send an email
> to bosh-users+...@cloudfoundry.__org.
>
>
> To unsubscribe from this group and stop
> receiving emails from it, send an email to
> bosh-users+...@cloudfoundry.__org.
>
>
> To unsubscribe from this group and stop receiving emails
> from it, send an email to bosh-users+...@__cloudfoundry.org.
>
>
> To unsubscribe from this group and stop receiving emails from
> it, send an email to bosh-users+...@cloudfoundry.org <javascript:>.
>
>
> To unsubscribe from this group and stop receiving emails from it, send
> an email to bosh-users+...@cloudfoundry.org
> <mailto:bosh-users+...@cloudfoundry.org>.

Евгений Костенко

unread,
Jul 14, 2014, 6:18:32 AM7/14/14
to bosh-...@cloudfoundry.org
Yep, i know that CF not supported any region except us-east-1. I've already figured out it by myself, spending on it too much work days.
And odds with blobs not the biggest problem. My original issue can be easy fixed by adding one new line, but i don't sure, that it will be fixed in upstream)

I'm not alone, whos facing same issues related to "outside of us-east-1". Here is example from Github Gist - https://gist.github.com/danhigham/5804252

CF users (including me) trying to fix "non-existing AMI" issues on their own. It's not a big problem, but it's got even worse because of existing outdated CF docs. And even this docs, as i remembered, didn't warned users about such "known issue". I mean, something like "be warned, this will not work outside of us-east-1. try it on your own risk" in docs related to deploy CF in AWS.

понедельник, 14 июля 2014 г., 12:10:13 UTC+4 пользователь James Masson написал:

Pretty much only US-East-1 is supported in BOSH/CF at the moment. It is
possible to use other regions, but you have to be aware of some of the
quirks.

eg. no region-local AMIs and various Blobstore handling tr -

Adam Stegman

unread,
Jul 15, 2014, 1:49:20 PM7/15/14
to bosh-...@cloudfoundry.org
On Monday, July 14, 2014 3:18:32 AM UTC-7, Евгений Костенко wrote:
CF users (including me) trying to fix "non-existing AMI" issues on their own. It's not a big problem, but it's got even worse because of existing outdated CF docs. And even this docs, as i remembered, didn't warned users about such "known issue". I mean, something like "be warned, this will not work outside of us-east-1. try it on your own risk" in docs related to deploy CF in AWS.

Which outdated docs are you referring to? If we still have docs referencing the AWS plugin and not including region information we should definitely fix that.

Евгений Костенко

unread,
Jul 15, 2014, 2:21:08 PM7/15/14
to bosh-...@cloudfoundry.org
In current CF docs mentioned v143 release. I can assume, that current docs was updated last time in Sep 2013. I guess is can be named as "outdated".
I've re-read docs about deployment in AWS (http://docs.cloudfoundry.org/deploying/ec2/bootstrap-aws-vpc.html) and finally found what i've missed previously:

> Note: Now only deployment to us-east-1 region is supported by the next steps

Maybe I read inattentively, but i think such notes must be more conspicuous.

And i've tried few versions of bosh (from 2579 to 2624) - it's works if you make "quick" fix of issue with existing s3 buckets. I mean that "bosh aws create" will do all as needed - create VPC, RDS, etc.

вторник, 15 июля 2014 г., 21:49:20 UTC+4 пользователь Adam Stegman написал:

Kim Hoffman

unread,
Jul 15, 2014, 2:49:49 PM7/15/14
to bosh-users
It's definitely a problem that our docs are that outdated. We are working to update them all, and I will make sure there is a story in our tracker about this topic specifically. 

In the meantime, I can see how that note would be hard to catch. It's a quick fix for us to style the note to stand out more, so we'll do that right away. Hopefully that will save future readers some pain. 

Thanks for your feedback!
Kim


To unsubscribe from this group and stop receiving emails from it, send an email to bosh-users+...@cloudfoundry.org.

Greg Oehmen

unread,
Jul 21, 2014, 12:56:48 PM7/21/14
to bosh-...@cloudfoundry.org
Thanks Kim.  I'll close the github issue on this & we'll look forward to the doc update.

Greg
Reply all
Reply to author
Forward
0 new messages