error deploying to ec2 using salt cloud

[Tim Dunphy]

Hi all,

I'm trying to deploy EC2 servers using salt-cloud. But I'm getting an error when I do. This is the error that I'm seeing:

 

    [root@salt salt]# salt-cloud -p ec2_private_win_r3.xlarge server00009

    [ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided access credentials', 'Code': 'AuthFailure'}}, 'RequestID': '33b43015-518e-4865-88e7-b6432e61b0db'}

    [ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided access credentials', 'Code': 'AuthFailure'}}, 'RequestID': '4b88b080-ad32-4388-a133-4322b1c08c04'}

    [ERROR   ] There was a profile error: 'NoneType' object has no attribute 'copy'

 

I’ve verified the AWS keys that I’m using and I’m able to list and even launch new instances using the aws command line with the keys that I’m using in the cloud provider file:

aws ec2 run-instances --image-id ami-xxxxxx --count 1 --instance-type c4.large --key-name company-timd --security-group-ids sg-xxxxx --subnet-id subnet-xxxxx --profile=company-npgovcloud

615531451610    r-036342b377d062086

INSTANCES       0       x86_64          False   xen     ami-e0daeff7    i-0017b45abfed0001e     c4.large        company-timd       2017-06-15T20:04:53.000Z        ip-10-4-137-7.us.kworld.kpmg.com       10.4.137.7              /dev/xvda       ebs     True            subnet-xxxx hvm     vpc-xxxxx

MONITORING      disabled

NETWORKINTERFACES               0e:b0:18:f1:2f:a8       eni-xxxx    615531451610    10.4.137.7      True    in-use  subnet-3a7a5361 vpc-16532f70

ATTACHMENT      2017-06-15T20:04:53.000Z        eni-attach-41df4a4d     True    0       attaching

GROUPS  sg-xxxxx    npgovapp1-sg-default

PRIVATEIPADDRESSES      True    10.4.137.7

PLACEMENT       us-east-1c              default

SECURITYGROUPS  xxxxx     npgovapp1-sg-default

STATE   0       pending

STATEREASON     pending pending

 

This is my cloud provider definition that uses the same keys:

    company-govcloud-nonprod-us-east-1:

      # Set up the location of the salt master

      

      minion:

      master: 10.0.2.15

     

      # Set up grains information, which will be common for all nodes

      # using this driver

      grains:

      node_type: broker

     

      # Valid options are:

      #     private_ips - The salt-cloud command is run inside the EC2

      #     public_ips - The salt-cloud command is run outside of EC2

      #

      ssh_interface: private_ips

     

      # Optionally configure the Windows credential validation number of

      # t-tdetries and delay between retries.  This defaults to 10 retries

      # with a one second delay betdwee retries

      win_deploy_auth_retries: 10

      win_deploy_auth_retry_delay: 1

     

      # Set the EC2 access credentials (see below)

     

      id: 'AKIAIATLQ4FTDDA6BV7A'

      key: 'asdfasdsfadsadasasdafadsadfafasdasda’

     

      # Make sure this key is owned by root with permissions 0400.

      #

      private_key: /etc/salt/company-timd

      keyname: company-timd

      #securitygroup: core-sg-default

     

      # Optionally configure default region

      # Use salt-cloud --list-locations <driver> to obtain valid regions

      #

      location: us-east-1

      availability_zone: us-east-1c

     

      # Configure which user to use to run the deploy script. This setting is

      # dependent upon the AMI that is used to deploy. It is usually safer to

      # configure this individually in a profile, than globally. Typical users

      # are:

      # Amazon Linux -> ec2-user

      # RHEL         -> ec2-user

      # CentOS       -> ec2-user

      # Ubuntu       -> ubuntu

      #

      ssh_username: root

     

      # Optionally add an IAM profile

      #iam_profile: 'arn:aws:iam::xxxxxxxxxxxx:role/rl-company-admin'

     

      driver: ec2

 

And this is the profile that I’m trying to use:

 

    ## Windows Server 2012 Alteryx & Tableau

    ec2_private_win_r3.xlarge:

      provider: company-govcloud-nonprod-us-east-1

      image: ami-xxxxxxx

      size: r3.xlarge

      network_interfaces:

        - DeviceIndex: 0

          SubnetId: subnet-xxxxxxx

          SecurityGroupId: sg-xxxxxx

          PrivateIpAddresses:

            - Primary: True

          AssociatePublicIpAddress: False

      block_device_mappings:

       - DeviceName: /dev/sda1

         Ebs.VolumeSize: 120

         Ebs.VolumeType: gp2

       - DeviceName: /dev/sdf

         Ebs.VolumeSize: 250

         Ebs.VolumeType: gp2

      tag: {'Engagement': '999999999999', 'Owner': 'Tim', 'Name': 'non-production', 'Environment': 'COMPANY-Grouper'}

 

I tried commenting out the IAM profile in the cloud provider definition. I’ve checked and the AWS credentials I’m using has administrator access in IAM.

I think the problem might have to do with specifying the ec2 driver in the cloud provider definition. 

I've put the debug output into this gist:

https://gist.github.com/anonymous/664f2d4b64d453f03ce0c7c2140d6988

I'd like some advice on how to get past this problem.

Thanks,

Tim

--

GPG me!!

gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B

[Mike Freitas]

I assume this will also fail, because i think you get an error about trying to perform a forbidden operation when you authenticate a don't have the permissions. But try this anyway:

salt-cloud --list-images company-govcloud-nonprod-us-east-1

I also assume these are the file names:

 '/etc/salt/cloud.providers.d/ec2.conf'

 '/etc/salt/cloud.profiles.d/ec2_profiles.conf'

I am pretty sure if these weren't loading you would get an error about no profile or provider. 

One last thought, does this instance have an IAM role assigned? I don't know if salt-cloud tries the role first, but again i would expect a permission denied not unable to validate keys.

Mike

[Tim Dunphy]

Hi Mike,

 Thanks for your response. 

When I try the command you suggested, it does work!

 salt-cloud --list-images company-govcloud-nonprod-us-east-1|head -10

company-govcloud-nonprod-us-east-1:

    ----------

    ec2:

        ----------

        aki-0251b36b:

            ----------

            architecture:

                x86_64

            blockDeviceMapping:

                None

Yes, you are correct, these are the file names that I'm using: 

ls -lh /etc/salt/cloud.providers.d/ec2.conf /etc/salt/cloud.profiles.d/ec2_profiles.conf

-rw-r--r--. 1 salt salt 4.8K Jun 12 19:35 /etc/salt/cloud.profiles.d/ec2_profiles.conf

-rw-r--r--. 1 salt salt  18K Jun 16 14:07 /etc/salt/cloud.providers.d/ec2.conf

The instance shouldn't require an IAM role. But I tried uncommenting out that line and tried again:

# Optionally add an IAM profile

  iam_profile: 'arn:aws:iam::615531451610:role/rl-company-admin'

But I still get the same result:

salt-cloud -p ec2_private_win_r3.xlarge server00009

[ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided access credentials', 'Code': 'AuthFailure'}}, 'RequestID': 'bf714616-9520-4704-ab02-7c26a444bad5'}

[ERROR   ] AWS Response Status Code and Error: [401 401 Client Error: Unauthorized] {'Errors': {'Error': {'Message': 'AWS was not able to validate the provided access credentials', 'Code': 'AuthFailure'}}, 'RequestID': '184139b4-876c-490c-985a-dc6ef0cd4d48'}

[ERROR   ] There was a profile error: 'NoneType' object has no attribute 'copy'

The permissions I have assigned to the keys I'm using in the cloud provider file have AdministratorAccess permissions.

Any advice you may have would be appreciated!

Tim

[Mike Freitas]

Hi Tim,

Well that is interesting now isn't it. It certainly sounds like a permission problem now, but exactly what is the fun part. Sorry for the confusion, the IAM role I was referring to would be assigned to your Salt master. The IAM role you commented out is the one that is assigned to the new instance, not the one that Salt cloud would use for provisioning. I made that mistake myself and realized I was assigning my Salt master role to my test instances...Whooops! 

The part that gets me is that according to the AWS docs you are supposed to get a 403 for a permission denied, and sometimes a coded message with more details:

http://docs.aws.amazon.com/cli/latest/reference/sts/decode-authorization-message.html

This talks about decoding the message a bit more.

http://cloudway.io/post/decode-authorization-message/

When I was setting up my IAM role it was very useful, however I think it was consistently getting the same errors when using CLI and Salt, but you are not.

At the moment I'm at a loss. I think if you put on trace logging you will get more of the AWS API request, possibly something in there will be helpful. 

salt-cloud -p ec2_private_win_r3.xlarge server00009 -l trace

Sorry nothing more is jumping out at the moment.

Mike

[Tim Dunphy]

Hi Mike,

 I tried applying a role to my salt master using these instructions:

https://docs.saltstack.com/en/latest/topics/cloud/aws.html

Not sure if I was doing that correctly. But I'm still getting the same error. I put some trace logs into a gist, hopefully that can provide some more info: https://gist.github.com/anonymous/d97b7ff188689568140b5d551b433d00

I haven't had a chance to look at the decode aws authorizations link you pointed me to. I'll have a look at that this weekend. 

Thanks for your help!

Tim

[Mike Freitas]

Offhand that didn't seem to add anything useful, maybe the garbage level:

salt-cloud -p ec2_private_win_r3.xlarge server00009 -l garbage

Any chance you can post the policy for that user to the gist or here if its not too large. As per usual just make sure you sanitize anything sensitive. Have a good weekend as well.

Mike

[Tim Dunphy]

Hmm,ok. I've produced some 'garbage' level log output from the command I'm trying to use. I hope this can help!

https://gist.github.com/anonymous/056b4c50cd87de444958808f4b484d22

Thanks, I appreciate any advice you have!

Tim

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/crmSADk3AH4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/219907b3-0db3-4592-97d6-1709f159e072%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Mike Freitas]

Hi Tim,

Unfortunately nothing is jumping out at me from the logs. Can you post a sanitized copy of your Policy/Role, etc?

Mike

To unsubscribe from this group and all its topics, send an email to salt-users+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/219907b3-0db3-4592-97d6-1709f159e072%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tim Dunphy]

Hi Mike,

Ok, sorry nothing is obvious to me either about why this isn't working. 

The role has the AWS default policy AdministratorAccess. 

These are the permissions on the AdministratorAccess policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

That should give the role access to everything in AWS. So I'm unclear as to why I'm still getting authentication errors. Especially considering that I can create AWS instances on the AWS command line using the same credentials as I have in the cloud provider definition I put earlier in the thread.