Vault returning an error during AWS ec2 authetnication

[Elizabeth]

Hi,

I'm using Vault 0.9.5 in a container on CoreOS in the AWS public cloud. I've configured the AWS ec2 auth method. The commands used to enable the auth backend and corresponding roles and policies are shown below.

Enable AWS EC2 authentication in Vault:

curl -X POST -H "x-vault-token:<OPS_TOKEN>" "https://$VAULT_SERVER:$VAULT_PORT:8200/v1/sys/auth/aws" -d '{"type":"aws"}'

Create Policies for apps to access Vault:

curl --header "X-Vault-Token:<ROOT_TOKEN>" --request PUT --data  @vault_app1_policy.json https://$VAULT_SERVER:$VAULT_PORT/v1/sys/policy/app1_policy

{

  "policy":"{

    \"path\":{ 

      \"transit\/app1\/*\":{\"capabilities\":[\"create\",\"read\", \"update\", \"list\"]}

      }

    }"

}

Transit  backend has been enabled in Vault.

Create Role for AWS ec2 auth :

curl -X POST -H "x-vault-token:<OPS_TOKEN>"  "https://$VAULT_SERVER:$VAULT_PORT/v1/auth/aws/role/app1_role" -d'{"auth_type": "ec2", "bound_account_id" : "516919419022", "bound_vpc_id" : "vpc-dd9863a6", "policies":"app1_policy","max_ttl":"1h"}'

On attempting to login via:

curl -X POST "https://$VAULT_SERVER:$VAULT_PORT/v1/auth/aws/login" -d  '{"role":"app1_role","pkcs7":"'$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/pkcs7 | tr -d '\n')'","nonce":"<nonce>"}'

I get the error:

"failed to verify instance ID: unable to fetch current caller: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

On the same EC2 instance from where I'm trying to login to the Vault server, I have manually verified the PKCS7 signature successfully.  I checked with AWS but could not help saying it was on Vault. 

Please let me know what I'm missing.

thanks,

Elizabeth

[Jeff Mitchell]

Hi Elizabeth,

The issue is that you need to configure credentials in the mount for Vault to reach out to the EC2 API. It attempts to use the normal credential chain provided by the AWS SDK (credentials file, env vars, instance metadata) but if those aren't available then you need to manually send in an access key/secret key pair.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/eeb8e0c9-90b1-4091-af27-6fcf7fe64c49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Shanthi Koikkara]

Thanks, Jeff ! Got it ...that's what the config client did. I had misunderstood that. 

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/eeb8e0c9-90b1-4091-af27-6fcf7fe64c49%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---

You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/YzEaCB7JcSY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GGzPda9Ou0UQ9av2MPAVyE78_a1imJmm%3DwDiK7LiDxDuA%40mail.gmail.com.

[Elizabeth]

Hi Jeff,

     I spent some time trying to research my next error - but I'm unable to find what's wrong and hence I need to ask here again ...but on getting AWS credentials and running the config client API as shown below I still get an error when logging in. 

 Got AWS Credentials using the IAM role attached to the EC2 instance:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<Role-NAME>

where the ec2 instance running Vault has access to the action ec2:DescribeInstances. For debugging I updated the IAM role with everything as given in the Vault example :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "iam:GetInstanceProfile",
        "iam:GetUser",
        "iam:GetRole"
      ],
      "Resource": "*"
    }
  ]
}

Then ran config client on Vault:

curl --header "X-Vault-Token:<TOKEN>" --request POST --data '{"access_key": "ASIAJLWLWLFR4BSTY3GA","secret_key": "aSIOU7GoGMiCunkEvBev7Strhns/JD/Q4Xz2DpAt"}' http://$VAULT_SERVER:$VAULT_PORT/v1/auth/aws/config/client

Then ran aws login from the Vault client:

curl -X POST "https://$VAULT_SERVER:$VAULT_PORT/v1/auth/aws/login" -d  '{"role":"app1_role","pkcs7":"'$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/pkcs7 | tr -d '\n')'","nonce":"<nonce>"}'

Got the error:

{"errors":["failed to verify instance ID: unable to fetch current caller: InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: 2e8b61e9-3930-11e8-82c6-0b1fe8b46568"]}

Since I'm on coreOS I cannot verify if ec2 Describe Instances using AWS CLI would work - as I cannot install the CLI.... 

Thanks,

Elizabeth

[Elizabeth]

Duh! I updated the aws  login command to: 

 curl -X POST "http://10.158.27.89:8200/v1/auth/aws/login" -d @payload.json

where payload is:

{

        "role": "app1_role",

        "pkcs7": "'$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/pkcs7 | tr -d '\n')'",

        "signature": "'$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/signature | tr -d '\n')'",

        "nonce": "5defbf9e-a8f9-3063-bdfc-54b7a42a1f95",

        "identity": "'$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | tr -d '\n')'"

}

but I still get:

{"errors":["failed to base64 decode the instance identity document"]}

[Joel Thompson]

Hi Elizabeth,

> Got AWS Credentials using the IAM role attached to the EC2 instance

Can you clarify which instance? Is this the CoreOS instance hosting the Vault container?

I think your first attempt was closer to being correct. However, you shouldn't use the credentials obtained from the IAM instance profile. For one reason, those credentials will expire automatically. For another, they require a separate security token to be valid (as with all temporary AWS credentials), and Vault doesn't support configuring a security token using temporary security credentials. That's why you got this error: "InvalidClientTokenId: The security token included in the request is invalid"

I would suggest just configuring your container networking to allow the Vault container to directly access the 169.254.169.254 instance metadata service, and it will just pick up the credentials automatically, including rotating them.

I believe there are two problems with your second attempt. The first is that you need to specify either 1: the pcks parameter OR 2: the identity and signature parameters. You cannot supply both 1 and 2, which is what your payload.json does. The second is that you're not base64 encoding the identity parameter. You need to base64-encode the identity document before passing it in to Vault. (Also, it looks like you've got single quotes inside double quotes; those would cause base64-decoding errors as well.)

--Joel

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0e63017e-0f2e-4e9a-9f56-a8bc272158e4%40googlegroups.com.

[Elizabeth]

I moved further in that I base64 encoded the signature and identity and also configured the AWS certificate in the auth backend.

 

Also removed pkcs7 as it was now redundant .  The error is:

 

 {"errors":["instance identity verification using SHA256 RSA signature is unsuccessful"]}

  I think the error now is around the part in Vault doc around the AWS public cert config that is unclear -to me - 

• type (string: "pkcs7") - Takes the value of either "pkcs7" or "identity", indicating the type of document which can be verified using the given certificate. The PKCS#7 document will have a DSA digest and the identity signature will have an RSA signature, and accordingly the public certificates to verify those also vary. Defaults to "pkcs7".

[Joel Thompson]

Hi Elizabeth,

Are you in GovCloud or one of the China regions? If not, you shouldn't need to set the certificate. Vault comes preloaded with the correct certs so you shouldn't need to configure the AWS public cert. I don't know what AWS does for GovCloud or the China region and so I don't know what would be needed to make it work in those regions.

--Joel

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/170808aa-182d-41fa-b13c-a6f45fea7921%40googlegroups.com.

[Elizabeth]

Hi Joel,

    Thank you - looks like our mails crossed. 

 I'min the AWS Public cloud in the us-east 1a (N Viriginia) region.  

> I would suggest just configuring your container networking to allow the Vault container to directly access the 169.254.169.254 instance metadata service, and it will just pick up the credentials automatically, including rotating them.

Sure. Right now I was doing it manually but I was ensuring the credentials were current. 

I haven't worked on it as yet today. I will go back to the first attempt and try to pursue that. 

thanks,

Elizabeth

[Elizabeth]

Hi,

    I still get the same error on AWS login. On looking  into Vault code I gleaned that the error is thrown when the ec2 describe-Instances is invoked. after the pkcs7 signature validation.  

    I also used the AWS access Key and secret Key I got for Vault and used it from AWS CLI and Describe-Instances works. 

    So perhaps I'm doing something wrong while configured Vault with the access key and secret key. 

To reiterate the steps I followed to config Vault:

1.  Obtained AWS Credentials using the IAM role attached to the EC2 instance hosting the Vault Server container. 
   curl http://169.254.169.254/latest/meta-data/iam/security-credentials/<Role-NAME>
2. Invoked Vault's AWS Config Client to set the AWS access key id and secret key:                                                                                                                                   

   curl --header "X-Vault-Token:$OPS_TOKEN" --request POST --data'{"access_key": "ASIAJOMERSIAQDY3XPPQ","secret_key": "OLe+GvYdoUGXke2/j3mCPKivN/KxHU2iB/urZ248"}' http://127.0.0.1:8200/v1/auth/aws/config/client

Q: Should the  AWS Session Token be also configured in Vault?  If so how? 

@Joel, you had suggested making Vault  obtain the AWS credentials - how can that be done?   BTW, I'm running Vault as a docker container on a EC2 instance that runs CoreOS.  

Any suggestions would be appreciated.

thanks,

Elizabeth

[Joel Thompson]

Hi Elizabeth,

As I mentioned earlier, using just the access key and secret key from http://169.254.169.254/latest/meta-data/iam/security-credentials/<RoleName> will never be sufficient to authenticate to AWS as those credentials also require the security token to be supplied. You cannot supply the token to Vault using the config/client endpoint.

The best way would be to allow your container access to the http://169.254.169.254 endpoint. You'd need to ensure Docker is configured to allow your container access to the http://169.254.169.254 endpoint (and also remove the access key and secret key from the config/client endpoint). There's a stack overflow post on how to do this at https://stackoverflow.com/q/22409367 -- there's a comment that you need to specify --net=host to enable this access (but I'm not a docker networking expert).

The other option would be to create an IAM user and generate an access key and secret key 

--Joel

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/618e1913-16f6-4af5-a94a-edf8eceeb0ee%40googlegroups.com.

[Elizabeth]

Thanks much, Joel. that worked.. 

However, I'm a bit confused - in the start of this thread I had not run the auth/aws/config and I was getting an error: 

"failed to verify instance ID: unable to fetch current caller: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

then I executed the auth/aws/config and I started getting the error:

{"errors":["failed to verify instance ID: unable to fetch current caller: InvalidClientTokenId: The security token included in the request is invalid.\n\tstatus code: 403, request id: 2e8b61e9-3930-11e8-82c6-0b1fe8b46568"]}

and now I undo the auth/aws/config  (but after trying a bunch of other things ) and it works fine.  The Vault container always had access to the metadata service I did not change anything for that. 

I will try and nail down the exact set of vault  auth/aws commands needed  for the ec2 method and post for the benefit of any one else who runs into this issue. 

Thanks again,

Shanthi

[Elizabeth]

I may have found out what caused my initial error as posted at the start of the thread - AWS_REGION was not known to Vault. I set this as a env variable and the initial error does not occur. I tried this on a new installation of Vault. 

So to summarize - the steps below are sufficient to use the aws ec2 auth method successfully.

Configure following on Vault server (ensuring that the vault server has network access to the metadata service and has an environment variable $AWS_REGION set to the right AWS region)

• Enable aws auth backend
  
• Create an aws role for your Vault client (tied to an existing or new policy)
  

Login from the Vault client as below:

curl -X POST "http://$VAULT_SERVER:$VAULT_PORT/v1/auth/aws/login" -d '{"role":"<role created in step 2 above>","pkcs7":"'$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/pkcs7 | tr -d '\n')'", "nonce" : "<nonce>"}'

- Elizabeth