Versioning S3 remote state

[Andrew Langhorn]

Hello,

Does anyone use versioning with the S3 remote state backend? We haven't currently got it enabled, and I'd like to understand if anyone sees any negative implications or weird behaviours when using it.

Thanks,

--

Andrew Langhorn
Senior Site Reliability Engineer
Email	andrew....@thoughtworks.com
Telephone	+44 7733 339809

[Joshua Garnett]

Hi Andrew,

I recently switched over to the s3 remote state backend.  We enabled versioning of all assets in the s3 bucket and haven't seen any issues.  There was actually once case where we accidentally had an environment write to the wrong state and I was able to recover the old state by using the aws console tools.

--Josh

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAEpa1D%2Bv4-RVW1gO9HFPR%2Br9zffhJJMKgk7sADvkz4pzvJHpGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Andrew Langhorn]

Hi Josh,

Thanks for the reply! That's exactly what I'd like to avoid if possible, so it's good that it worked a-okay for you.

We'll have a further look at it, and probably enable it shortly, then!

Cheers.

Andrew

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAJAXYdi6q%3DWZDMTq3h%2BZ6X1-vSHFpZJSqBnXuwLQpAza%3DYVW9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[egul...@gmail.com]

Hello,

How are you using s3 buckets for states from multiple environments? I would like to have something like this:


   bucket1/
               dev/dev-state.file
               stg/stg-state.file
               prd/prd-state.file

I'm testing this and 'bucket1' is in different region than dev environment and tf plan fails with error:

Error refreshing state: 1 error(s) occurred:

* data.terraform_remote_state.remote_state: BucketRegionError: incorrect region, the bucket is not in 'us-west-1' region

        status code: 301, request id:

That leads me to think that I would have to create bucket for each state. Have someone encountered similar issue?

Thanks

[David Adams]

You can specify the region the bucket lives in in your config:

    data "terraform_remote_state" "my-remote-state" {

      backend = "s3"

      config {

        bucket = "bucket1"

        region = "us-west-2"

        key    = "dev/dev-state.file"

      }

    }

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/6afb997a-e9af-4bcf-8bb2-64a249871fd7%40googlegroups.com.

[David Adams]

(That said, Terraform could just figure out where the bucket is so long as your credentials can call s3:GetBucketLocation. And in fact it looks like the API is giving a redirect.)

[egul...@gmail.com]

This is exactly what I was looking for.

Thanks

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

[Andrew Langhorn]

Even if you didn't specify a region, it should have still worked in theory: the dev/ stg/ etc. keys are all in the same bucket. We have one state file per environment in the same bucket, versioned, and it works flawlessly.

From memory, we don't specify the region in Terraform because we have CI servers handle that; Terraform thankfully understands the AWS_DEFAULT_REGION environment variable, meaning that it can conform to standards for other AWS tools we use which also respect that environment variable.

Glad you now have it working, though.

[egul...@gmail.com]

yes but the problems I have is when AWS_DEFAULT_REGION is different from the region where bucket was created.

[egul...@gmail.com]

and David's reply solves it. :)

[Andrew Langhorn]

Ah, sorry. In that case, the region parameter is the correct way to fix that. Glad you have it working in any case :)

[egul...@gmail.com]

so after playing with TF code since my last post, today I started working on another TF project and it seems none of these options work for me.. :((((.
Commented code is 2nd option that I've tried without success.


Here's my code:

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "${var.region}"
}

#provider "aws" {
#    region = "us-east-1"
#    alias = "bucket_region"
#}
data "terraform_remote_state" "remote_state" {
  backend = "s3"
  config {
    #"provider" = "aws.bucket_region"
    "bucket" = "my_bucket"
    "region" = "us-east-1"
    "key"    = "tf_states/env1/terraform.tfstate"
    #"region" = "us-east-1"
  }
}

and AWS_DEFAULT_REGION=us-west-1, so I'm stumped. 

tf get fails with:

Error loading Terraform: Error reloading remote state: BucketRegionError: incorrect region, the bucket is not in 'us-west-1' region

        status code: 301, request id:

I even tried manually adding "region" to .terraform/terraform.tfstate but still no success there.

What am I doing wrong here?

Any advice greatly appreciated.

[Ian Duffy]

As the error suggests, the bucket you've specified is not in the region you've specified.

On the AWS console, navigate into the S3 section, open your bucket, click the properties tab, and confirm the region the bucket is placed into.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/4bb76312-7645-433d-8b9d-c2f1d91c654e%40googlegroups.com.

[egul...@gmail.com]

It says 'US Standard' which is us-east-1.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.

[Ian Duffy]

Cool! your error is saying its picking up us-west-1 as the defined region.

"AWS_DEFAULT_REGION=us-west-1" should be AWS_DEFAULT_REGION=us-east-1

and then the terraform config looks fine.

Let me know how you get on. 

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/053327b3-b3c5-4565-a082-733c8ff0c496%40googlegroups.com.

[egul...@gmail.com]

OK,
please help me understand how this works.

Is AWS_DEFAULT_REGION only for remote_state?

I have bucket in east-1 and I'm provisioning infrastructure in west-1. Does that mean we can't have that or since in my templates I'm using

  region = ${var.region}

then tf plan & tf apply knows to spin up resources in different region?

[Ian Duffy]

https://github.com/hashicorp/terraform/blob/master/state/remote/s3.go#L37

If you set a configuration option for region it will use the configuration block. If you don't, it will use AWS_DEFAULT_REGION.

I would suggest leaving AWS_DEFAULT_REGION unset and just specifying the region as required in both your aws provider and remote_state data source.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/218f8184-b821-4713-8844-db261894aa60%40googlegroups.com.

[David Adams]

There is a region specified in his code...

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/CAEJ3w4Xs2iH02uoEoOVeR5Xe9Evu0aDkb95Skq%3D19TyXp%2B0-Gw%40mail.gmail.com.