How to setup profiles and providers when using multiple accounts but using IAM cross account roles

matzuba

unread,

Oct 20, 2017, 10:16:28 AM10/20/17

to Terraform

Hello guys

I have multiple AWS accounts. I want to use 1 master build account with an IAM user/access key combo to push out code to the rest. Each other account will have have a cross account role defined that allows the master account access.

The goal is to not have multiple IAM users per account and only define them in the master account.

I am not sure how to setup the profile as essentially, shouldn't the profile point to the main account where the IAM as that is the account that allows the assumed role in?

How do profiles work with S3 remote state buckets? If i have to use the main profile as the access point, how does Terraform know what account to write the state to? I understand that i have use the provider directive on a resource and it will use that provider and create correctly in the write account.

provider.tf:

# The default provider

provider "aws" {

profile = "build"

region = "ap-southeast-2"

assume_role {

role_arn = "arn:aws:iam::ACCID1:role/CrossAccountAdministration"

}

# test account

provider "aws" {

alias = "test"

profile = "test"

region = "ap-southeast-2"

assume_role {

role_arn = "arn:aws:iam::ACCTID2:role/CrossAccountAdministration"

}

Fernando

unread,

Oct 20, 2017, 10:27:55 AM10/20/17

to Terraform

On Friday, 20 October 2017 15:16:28 UTC+1, matzuba wrote:

How do profiles work with S3 remote state buckets? If i have to use the main profile as the access point, how does Terraform know what account to write the state to? I understand that i have use the provider directive on a resource and it will use that provider and create correctly in the write account.

I'm going over exactly this as we speak.

this is my backend.tf

provider "aws" {
  region = "${var.region}"

  #   assume_role {
  #     role_arn     = "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
  #     session_name = "SESSION_NAME"
  #   }
}

resource "aws_s3_bucket" "state_bucket" {
  bucket = "${var.s3bucket-tfstate}"
  acl    = "private"
  region = "${var.region}"

  versioning {
    enabled = true
  }

  tags {
    Name = "terraform state bucket"
  }
}

resource "aws_s3_bucket_policy" "bucket_policy" {
  bucket = "${var.s3bucket-tfstate}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ftstates3policy20171020",
      "Action": [
        "s3:PutObject",
        "s3:GetObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::MYBUCKET-terraform-state/*",
      "Principal": {
        "AWS": [
          "ACCOUNT_IDS"
        ]
      }
    },
    {
      "Sid": "ftstates3policy20171020",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3::: MYBUCKET-terraform-state",
      "Principal": {
        "AWS": [
          "ACCOUNT_IDS"
        ]
      }
    }
  ]
}
EOF
}

this will allow your other accounts to write to a preexisting bucket.

the issue i'm trying to fix is have a policy into the other accounts so that the written files can be read by other accounts... right now I get denied.\

* data.terraform_remote_state.tfstate-tools: data.terraform_remote_state.tfstate-tools: AccessDenied: Access Denied

so if you get there before me, let me know :D

matzuba

unread,

Oct 21, 2017, 10:34:23 AM10/21/17

to Terraform

Hey

Thanks for sharing.

I have 1 Master account and 2 other accounts which have a cross account role which grants the master role admin access. I authenticate against the master account but i dont understand how assume roles work with multiple S3 buckets.

MY goal was to attempt to have multiple account all accessed from the the "master" account and store the state file in each respective account. I have been trying to do this but i do not think it is supported, I am also getting 403 "Access Denied" when trying to write the state to the other account even though the cross account has admin access. I have played around with making the bucket public just for a quick test and sometimes, this appeared to work but this suggests that the S3 code for allowing cross account role access may not work.

It maybe easier for me just to use the 1 state bucket.

I found this issue i was looking at: https://github.com/terraform-providers/terraform-provider-aws/issues/835

This issue seems to point to an ACL for the bucket user being required which i didnt have.

For my backend config (taken from .terraform state as it is auto generated:

I have tried to set a an assumed role for my s3 config and have set the profile to the master account, this did not seem to make any difference

"bucket": "AccountID-terraform-state",

"key": "environment/Account-id/terraform.tfstate",

"profile": "MasterAccount-ID",

"region": "ap-southeast-2",

"role_arn": "arn:aws:iam::test-account:role/CrossAccountAdministration"

provider "aws" {

profile = "master-account"

region = "ap-southeast-2"

assume_role {

role_arn = "arn:aws:iam::test-account:role/CrossAccountAdministration"

}

This was my bucket policy:

I will test with yours and see if i get anywhere.

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Principal": {

"AWS": "arn:aws:iam::MasterAccoutID:root"

},

"Action": [

"s3:Get*",

"s3:List*",

"s3:Put*"

],

"Resource": "arn:aws:s3:::test-account-bucket-terraform-state/*"

}

]

}

How are you using your profiles? I thought something like below would work (it does for the aws cli) but it doesn't

[master]

aws_access_key_id = XXX

aws_secret_access_key =XXXX

aws_mfa_device = arn:aws:iam::master-acct-id:mfamatzuba

[test]

role_arn = arn:aws:iam::testaccount-id:role/CrossAccountAdministration

source_profile = master

Fernando Miguel

unread,

Oct 21, 2017, 1:19:49 PM10/21/17

to terrafo...@googlegroups.com

I've briefly skimmed your email, so sorry if you already mentioned this.

In our case, we have an IAM account, and users login from there and assume cross account roles to other accounts.

The bucket is on another account not owned by neither really remote accounts that will write the state to the bucket.

So the bucket pre dates the other accounts interaction, and it has a policy to allow those accounts to write there, read, list AND set acl for the objects.

This last bit is important, cause each account writing the state file needs to allow the bucket owner to read the file, which will later reshare access to that object to the other accounts as read.

I was able to achieve this via cli, but I am still trying to find how to have terraform set object acl when writing the state file to the bucket.

Here's aws documentation for it

http://docs.aws.amazon.com/AmazonS3/latest/dev/example-walkthroughs-managing-access-example4.html

I'll share our module once I get this fixed hopefully Monday

--

Fernando Miguel

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-too...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/dfa5cf43-e2de-4ec9-b421-86410913b718%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Fernando Miguel

unread,

Oct 21, 2017, 1:23:24 PM10/21/17

to terrafo...@googlegroups.com

I'm using aws vault to manage my aws IAM roles.

But in the end, an ec2 instance running jenkins will manage the infrastructure.

So that role is allowed to assume roles, like my user is.

You can find more here

https://github.com/FernandoMiguel/AWS-Trust-CrossAccounts

And a sample user config with roles at

https://github.com/FernandoMiguel/kb/blob/master/aws/config

--

Fernando Miguel

On 21 Oct 2017, at 15:34, matzuba <darren...@gmail.com> wrote:

matzuba

unread,

Oct 22, 2017, 9:22:00 AM10/22/17

to Terraform

Hey Fernando

Thanks for the effort you have put in here and sharing your links.

To confirm, you don't have cross account roles working with Terraform?

I have this working with the aws cli, but Terraform needs some more work and the config doesn't seem to work correctly. using the directive source_profiles does not seem to work in the .aws/credentials file

I am going to test your bucket policy and review your link for buckets! thanks for the pointers

⁞ Fernando Miguel

unread,

Oct 22, 2017, 9:37:12 AM10/22/17

to terrafo...@googlegroups.com

I can't say I've set up terraform with roles, cause I'm still invocating each app from cli with aws-shell, and using that to assume the role for each account.

I can try that Monday anyway. I just have to assume the role that allows sts across account from the IAM account, and then have terraform assume the role for each account.

I'll let you know of my progress

--
Fernando Miguel

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.

To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/ca4e631e-18c2-48a3-869f-9ce6d71cbad6%40googlegroups.com.

Brian Lalor

unread,

Oct 22, 2017, 10:06:14 AM10/22/17

to terrafo...@googlegroups.com

This is a limitation of the underlying aws-sdk golang library, unfortunately.

--
Brian Lalor
bla...@bravo5.org

matzuba

unread,

Oct 22, 2017, 11:23:09 PM10/22/17

to Terraform

Thanks for your input Brian. Yes, it is a shame Terraform cant use the same profile definitions in the credentials file.

Fernando

unread,

Oct 23, 2017, 7:49:22 AM10/23/17

to Terraform

So, lets restart this, and i'll share what I've learn.

I've got one IAM account with a couple users. For this example, i'll use my own user.

all users have no privileges other than STS to assume roles.

there are 3 other aws accounts.

one is the tools account which will contain the s3 bucket.

two other accounts for two apps.

each will write the state file to the pre-existing bucket.

each will at some point need to read the other app state file.

so, I login with my iam account, no role, and run the terraform that creates the bucket in the write account, using terraform assume-role

provider "aws" {
  region = "${var.region}"

  assume_role {
    role_arn     = "arn:aws:iam::ACCOUNT1:role/OrganizationAccountAccessRole"
    session_name = "ACCOUNT1"
  }
}

# Multi-account terraform bucket 

resource "aws_s3_bucket" "state_bucket" {
  bucket = "${var.s3bucket-tfstate}"
  acl    = "private"
  region = "${var.region}"

  versioning {
    enabled = true
  }

  tags {
    Name = "terraform state bucket"
  }
}

# Terraform bucket policy granting cross-account to write/read the state file

resource "aws_s3_bucket_policy" "bucket_policy" {
  bucket     = "${var.s3bucket-tfstate}"

  depends_on = ["aws_s3_bucket.state_bucket"]

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {

      "Sid": "YOURCOMPANYftstates3policy20171020",

      "Action": [
        "s3:PutObject",
        "s3:GetObject"
              ],

      "Effect": "Allow",
      "Resource": "arn:aws:s3:::YOURCOMPANY-terraform-state/*",
      "Principal": {
        "AWS": [
          "ACCOUNT2"
        ]
      }
    },
    {
      "Sid": "YOURCOMPANYftstates3policy20171020",
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::YOURCOMPANY-terraform-state",
      "Principal": {
        "AWS": [
          "ACCOUNT2"
        ]
      }
    }
  ]
}
EOF
}

# Creation of state file per cross-account

resource "aws_s3_bucket_object" "poc2statefile" {
  key        = "tools/terraform.tfstate"
  bucket     = "${var.s3bucket-tfstate}"
  source     = "/tmp/empty"
  acl        = "bucket-owner-full-control"
  depends_on = ["aws_s3_bucket.state_bucket"]
}

the bit at the end is me trying to pre-create the state file so the bucket owns the file, but it doesnt work.

using bucket-owner-full-control	in the acl doesnt work either.

anyway, this tf is able to create the bucket and the policy to allow other accounts to write to it, with assume-role. Perfect.

$ aws-vault exec YOURCOMPANY-iam -- terraform apply
aws_s3_bucket.state_bucket: Refreshing state... (ID: YOURCOMPANY-terraform-state)
aws_s3_bucket.state_bucket: Creating...
  acceleration_status:     "" => "<computed>"
  acl:                     "" => "private"
  arn:                     "" => "<computed>"
  bucket:                  "" => "YOURCOMPANY-terraform-state"
  bucket_domain_name:      "" => "<computed>"
  force_destroy:           "" => "false"
  hosted_zone_id:          "" => "<computed>"
  region:                  "" => "eu-west-2"
  request_payer:           "" => "<computed>"
  tags.%:                  "" => "1"
  tags.Name:               "" => "terraform state bucket"
  versioning.#:            "" => "1"
  versioning.0.enabled:    "" => "true"
  versioning.0.mfa_delete: "" => "false"
  website_domain:          "" => "<computed>"
  website_endpoint:        "" => "<computed>"
aws_s3_bucket.state_bucket: Creation complete after 4s (ID: YOURCOMPANY-terraform-state)
aws_s3_bucket_policy.bucket_policy: Creating...
  bucket: "" => "YOURCOMPANY-terraform-state"
  policy: "" => "{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"YOURCOMPANYftstates3policy20171020\",\n      \"Action\": [\n        \"s3:PutObject\",\n        \"s3:GetObject\"\n              ],\n      \"Effect\": \"Allow\",\n      \"Resource\": \"arn:aws:s3:::YOURCOMPANY-terraform-state/*\",\n      \"Principal\": {\n        \"AWS\": [\n          \"ACCOUNT2\"\n        ]\n      }\n    },\n    {\n      \"Sid\": \"YOURCOMPANYftstates3policy20171020\",\n      \"Action\": [\n        \"s3:ListBucket\"\n      ],\n      \"Effect\": \"Allow\",\n      \"Resource\": \"arn:aws:s3:::YOURCOMPANY-terraform-state\",\n      \"Principal\": {\n        \"AWS\": [\n          \"ACCOUNT2\"\n        ]\n      }\n    }\n  ]\n}\n"
aws_s3_bucket_object.poc2statefile: Creating...
  acl:                    "" => "bucket-owner-full-control"
  bucket:                 "" => "YOURCOMPANY-terraform-state"
  content_type:           "" => "<computed>"
  etag:                   "" => "<computed>"
  key:                    "" => "tools/terraform.tfstate"
  server_side_encryption: "" => "<computed>"
  source:                 "" => "/tmp/empty"
  storage_class:          "" => "<computed>"
  version_id:             "" => "<computed>"
aws_s3_bucket_object.poc2statefile: Creation complete after 0s (ID: tools/terraform.tfstate)
aws_s3_bucket_policy.bucket_policy: Creation complete after 0s (ID: YOURCOMPANY-terraform-state)

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Next, another app will run and push the state file:

terraform {
  backend "s3" {
    bucket = "YOURCOMPANY-terraform-state"
    key    = "tools/terraform.tfstate"
    region = "eu-west-2"

  }
}

provider "aws" {
  region = "${var.region}"

  assume_role {
    role_arn     = "arn:aws:iam::ACCOUNT2:role/OrganizationAccountAccessRole"
    session_name = "ACCOUNT2"
  }
}

data "terraform_remote_state" "YOURCOMPANYtfstate" {
  backend = "s3"

  config {
    bucket = "${var.s3bucket-tfstate}"
    key    = "tools/terraform.tfstate"
    region = "${var.region}"
  }
}

module "vpc" {
  source = "git::git@github.com:YOURCOMPANY/terraform-aws-vpc?ref=master"

  name = "tools"

  cidr                 = "${var.cidr-tools}"
  azs                  = ["${var.azs}"]

  enable_nat_gateway = false

  tags = {
    Owner       = "user"
    Environment = "dev"
  }
}

$ aws-vault exec YOURCOMPANY-iam -- terraform apply
Error loading state: AccessDenied: Access Denied status code: 403, request id: 0C664B19XXXX, host id: ZZ6GGN

this fails straight way, cause for some reason, terraform is not assuming the role to read the bucket, and trying to use my iam user, which ofc doesnt have access to it

let's try with assume-role from cli:

$ aws-vault exec YOURCOMPANY-account2 -- terraform apply
data.terraform_remote_state.YOURCOMPANYtfstate: Refreshing state...
data.aws_caller_identity.current: Refreshing state...
data.aws_vpc_endpoint_service.s3: Refreshing state...
module.vpc.aws_vpc.this: Creating...
[...]

Apply complete! Resources: 17 added, 0 changed, 0 destroyed.

yep, that works just fine.

$ aws-vault exec COMPANY-ACCOUNT1 --

aws s3api get-object-acl --bucket COMPANY-terraform-state --key tools/terraform.tfstate An error occurred (AccessDenied) when calling the GetObjectAcl operation: Access Denied

so if anyone knows how to set an acl when writing the state file, please let me know.

also, need to figure out how to have terraform properly assume role when accessing s3.

Fernando

unread,

Oct 23, 2017, 8:43:13 AM10/23/17

to Terraform

Well, that use stupidly easy

just add the ACL to the backend... doh

terraform {
  backend "s3" {
    bucket = "YOUCOMPANY-terraform-state"
    acl    = "bucket-owner-full-control"

now i'm only missing the assume-role...

matzuba

unread,

Oct 23, 2017, 9:46:57 AM10/23/17

to Terraform

This is very positive!

Thank you for the clear overview.

You can set the role to assume on the bucket from looking at the S3 backend directives. Not sure if that helps

I've not been able to go back to my POC today but am keen to get back to it! :)

I've not used AWS Vault but will review it but does this in essence replace what i am doing with my shared_credentials file? ie, storing my ACCESS_KEY and SECRET_KEY? Are you using MFA?

⁞ Fernando Miguel

unread,

Oct 23, 2017, 9:56:38 AM10/23/17

to terrafo...@googlegroups.com

You can set the role to assume on the bucket from looking at the S3 backend directives. Not sure if that helps

Not sure what you mean. can you explain?

I've not used AWS Vault but will review it but does this in essence replace what i am doing with my shared_credentials file? ie, storing my ACCESS_KEY and SECRET_KEY?

I use aws-vault mainly so I dont have my api keys in clear text in my disk. so they are stored in my mac keychain instead.

I linked previously to a sample of my aws config file where you can add all the roles you need.

Are you using MFA?

Yes.

matzuba

unread,

Oct 24, 2017, 9:28:27 AM10/24/17

to Terraform

Hey mate

Sorry, i'm on a workshop and replied without having a chance to review things.

I was referring to setting the "assume role" on the S3 backend state bucket like below:

  backend = "s3"
  config {
    bucket   = "${var.bucket_name}"
    key      = "base/terraform.state"
    region   = "${var.aws_region}"
    role_arn = "arn:aws:iam::XXXXXX:role/Administrator"
  }

}

Yes - storing them on disk is not ideal, good job for MFA though! We are trialing federated authentication with SAML at the moment with the intent that this will replace the IAM user and keys.

Regarding aws-vault, who provides this? Hasibcorp or perhaps you are might be using this project https://github.com/99designs/aws-vault? Sorry, i didn't check your links properly, i can see you are using MFA. I need to understand your setup more as my understanding was that the .aws/config file was not used by terraform, perhaps aws-vault is "handling" the role/source_profile configuration and using it.

I'll be back to testing this soon.

⁞ Fernando Miguel

unread,

Oct 24, 2017, 9:33:20 AM10/24/17

to terrafo...@googlegroups.com

I was referring to setting the "assume role" on the S3 backend state bucket like below:

Ohhhh that's smart... i didnt know we could do that... only have the arn in the provider.... time to change everywhere :D

that will fix my init. thanks

Regarding aws-vault, who provides this? this project https://github.com/99designs/aws-vault?

Correct.

my understanding was that the .aws/config file was not used by terraform, perhaps aws-vault is "handling" the role/source_profile configuration and using it.

terraform still needs credentials.

for AWS CLI those are in .aws/creds

but they can be env vars, EC2 role, or something like aws-vault.

⁞ Fernando Miguel

unread,

Oct 24, 2017, 9:49:23 AM10/24/17

to terrafo...@googlegroups.com

tested and working

terraform {

backend "s3" {

bucket = "COMPANY-terraform-state"

role_arn = "arn:aws:iam::ACCOUNT_ID:role/OrganizationAccountAccessRole"

acl = "bucket-owner-full-control"

key = "terraform.tfstate"

region = "eu-west-2"

# terragrunt to replace this

# bucket = "${var.s3bucket-tfstate}"

# key = "${var.env}/terraform.tfstate"

# region = "${var.region}"

}

provider "aws" {

region = "${var.region}"

assume_role {

role_arn = "arn:aws:iam::ACCOUNT_ID:role/OrganizationAccountAccessRole"

session_name = "DEV"

}

data "terraform_remote_state" "tfstate" {

backend = "s3"

config {

bucket = "${var.s3bucket-tfstate}"

role_arn = "arn:aws:iam::ACCOUNT_ID:role/OrganizationAccountAccessRole"

key = "terraform.tfstate"

region = "${var.region}"

}

Now I can run

$ aws-vault exec iam-account -- terraform init

and it will assume all the right roles :D

thank you so much for helping me figure this out :))

feel free to ping me on slack or hangouts if you have any Qs on vault

matzuba

unread,

Oct 24, 2017, 10:12:37 AM10/24/17

to Terraform

my understanding was that the .aws/config file was not used by terraform, perhaps aws-vault is "handling" the role/source_profile configuration and using it.

terraform still needs credentials.
for AWS CLI those are in .aws/creds
but they can be env vars, EC2 role, or something like aws-vault.

sorry, what i meant is that terraform does not seem able to or support processing the role_arn or source_profile statements when in the credentials file but you having this working with aws-vault which is feeding creds/roles into terraform which is great!

Good stuff!!!!!!

and it will assume all the right roles :D

and thank you for sharing your config and notes. it is a shame a lot of the posts in here go cold!

thank you so much for helping me figure this out :))

Sounds good buddy! will try hangouts!

⁞ Fernando Miguel

unread,

Oct 24, 2017, 10:15:32 AM10/24/17

to terrafo...@googlegroups.com

> and thank you for sharing your config and notes. it is a shame a lot of the posts in here go cold!

my idea is to see if i can opensource (LOL) this to github

and probably make a https://registry.terraform.io/ module.

i'll be using this quite a bit, since all my accounts are assume role, and cross account is bound to happen

Paul Bourdel

unread,

Feb 2, 2018, 9:00:00 AM2/2/18

to Terraform

Thanks for this post! What was your module idea? I know you can pass providers to modules so you can run the same module on different accounts. I think you would have to have a module block for each account. I'm not sure if you can make a module to configure the backend though.

⁞ Fernando Miguel

unread,

Feb 3, 2018, 11:27:22 PM2/3/18

to terrafo...@googlegroups.com

This is an old thread, barely remembered this.

What is your question again?

--
Fernando Miguel

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/terraform/issues
IRC: #terraform-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Terraform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to terraform-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/terraform-tool/cfc7cf61-3ef9-4d88-ae10-5940a0f096d8%40googlegroups.com.

Reply all

Reply to author

Forward