AuthFailure => AWS was not able to validate the provided access credentials -- how to troubleshoot

[Steven Noble]

I'm hoping you can help me troubleshoot the error...

AuthFailure => AWS was not able to validate the provided access credentials

...which I'm getting on...

cap rubber:create_staging

I've posted this as an issue on Rubber's Github page but I think this might be a more appropriate venue to raise the issues. If it is, please let me know and I'll take down the Github issue if it's still unanswered at that point.

I have:

# config/rubber/rubber.yml

app_name: testivate
app_user: app
admin_email: <my email>
timezone: US/Eastern
domain: foo.com
cloud_provider: aws
cloud_providers:
  aws:
    region: us-east-1
    access_key: ...
    secret_access_key: ...
    account: ...
    key_name: gsg-keypair
    key_file: "#{Dir[(File.expand_path('~') rescue '/root') + '/.ec2/*' + cloud_providers.aws.key_name].first}"

This is followed by the unchanged default declarations covering security groups etc.

I've tried to avoid changing the file unnecessarily. I haven't done anything that is not explicitly listed below. For example, as far as I'm aware I don't have to enter 'testivate'/'app'/'foo' into a console on the EC2 website before attempting to deploy, so I have not done so.

I generated the strings for cloud_providers.aws.access_key and cloud_providers.aws.secret_access_key via `https://console.aws.amazon.com/iam/home?#security_credential` > Access Keys (Access Key ID and Secret Access Key) > Create New Access Key.

On that page, the key is listed as Last Used: N/A, Last Used Region: N/A, Last Used Service: N/A, Status: Active.

The number I have entered for cloud_providers.aws.account is the number at `https://console.aws.amazon.com/iam/home?#security_credential` > Account Identifiers > AWS Account ID, with the hyphens removed.

I created my keypair via `https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#KeyPairs:sort=keyName` > Create Key Pair. On that page, it says 'N. Virginia' on the location menu in the top-right corner.

I have saved the keypair file as `~/.ec2/gsg-keypair`. Open it and you see "-----BEGIN RSA PRIVATE KEY----- etc etc". I have created a public version as `~/.ec2/gsg-keypair.pub`. Open it and you see "ssh-rsa etc etc".

When I run `cap rubber:create_staging`, I just hit return at each of the prompts, then I get the error/stacktrace below. How can I troubleshoot this?

$ cap rubber:create_staging
    triggering load callbacks
  * 2016-06-05 12:21:25 executing `rubber:init'
  * 2016-06-05 12:21:26 executing `rubber:create_staging'
Hostname to use for staging instance [production]:
Roles to use for staging instance [apache,app,collectd,common,db:primary=true,elasticsearch,examples,graphite_server,graphite_web,graylog_elasticsearch,graylog_mongodb,graylog_server,graylog_web,haproxy,mongodb,monit,passenger,postgresql,postgresql_master,web,web_tools]:
  * 2016-06-05 12:21:28 executing `rubber:create'
/Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/expects.rb:6:in `response_call': AuthFailure => AWS was not able to validate the provided access credentials (Fog::Compute::AWS::Error)
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/response_parser.rb:8:in `response_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:372:in `response'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:236:in `request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:256:in `rescue in request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:204:in `request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:256:in `rescue in request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:204:in `request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/idempotent.rb:26:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/middlewares/base.rb:10:in `error_call'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:256:in `rescue in request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/excon-0.45.4/lib/excon/connection.rb:204:in `request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/fog-xml-0.1.2/lib/fog/xml/sax_parser_connection.rb:35:in `request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/fog-xml-0.1.2/lib/fog/xml/connection.rb:7:in `request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/fog-aws-0.8.1/lib/fog/aws/compute.rb:525:in `_request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/fog-aws-0.8.1/lib/fog/aws/compute.rb:520:in `request'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/fog-aws-0.8.1/lib/fog/aws/requests/compute/describe_security_groups.rb:38:in `describe_security_groups'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/fog-aws-0.8.1/lib/fog/aws/models/compute/security_groups.rb:63:in `all'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/cloud/aws/classic.rb:27:in `describe_security_groups'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/cloud/aws/classic.rb:125:in `sync_security_groups'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/cloud/aws/classic.rb:19:in `setup_security_groups'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/cloud/aws/base.rb:75:in `before_create_instance'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/thread_safe_proxy.rb:13:in `method_missing'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/recipes/rubber/instances.rb:281:in `block in create_instance'
 from /Users/steven/.rvm/rubies/ruby-2.3.1/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/recipes/rubber/instances.rb:280:in `create_instance'
 from /Users/steven/.rvm/gems/ruby-2.3.1/gems/rubber-3.2.2/lib/rubber/recipes/rubber/instances.rb:230:in `block (2 levels) in create_instances'

[Steven Noble]

UPDATE

The following does not generate the error shown above:

require 'fog'

# create a connection
connection = Fog::Compute.new({
  :provider                 => 'AWS',
  :aws_secret_access_key    => YOUR_SECRET_ACCESS_KEY,
  :aws_access_key_id        => YOUR_SECRET_ACCESS_KEY_ID
})

puts connection.servers.length

It returns:

#<Fog::Compute::AWS::Real:70120599372720 @connection_options={:debug_response=>true, :headers=>{"User-Agent"=>"fog/1.37.0 fog-core/1.35.0"}, :persistent=>false} @region="us-east-1" @instrumentor=nil @instrumentor_name="fog.aws.compute" @version="2014-10-01" @use_iam_profile=nil @aws_access_key_id="AKIAI3RQHDV6KBLBLUAA" @aws_credentials_expire_at=nil @signer=#<Fog::AWS::SignatureV4:0x007f8c72d58698 @region="us-east-1", @service="ec2", @aws_access_key_id="AKIAI3RQHDV6KBLBLUAA", @hmac=#<Fog::HMAC:0x007f8c72d585f8 @key="AWS4/NGjaH7giw0Gd8PtqrTPo+Y5L+AFsRDDs3NdjBYF", @digest=#<OpenSSL::Digest: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>, @signer=#<Proc:0x007f8c72d58558@/Users/steven/.rvm/gems/ruby-2.3.1/gems/fog-core-1.35.0/lib/fog/core/hmac.rb:28 (lambda)>>> @endpoint=nil @host="ec2.us-east-1.amazonaws.com" @path="/" @persistent=false @port=443 @scheme="https" @connection=#<Fog::XML::Connection:0x007f8c72d58210 @excon=#<Excon::Connection:7f8c6953f578 @data={:chunk_size=>1048576, :ciphers=>"HIGH:!SSLv2:!aNULL:!eNULL:!3DES", :connect_timeout=>60, :debug_request=>false, :debug_response=>true, :headers=>{"User-Agent"=>"fog/1.37.0 fog-core/1.35.0"}, :idempotent=>false, :instrumentor_name=>"excon", :middlewares=>[Excon::Middleware::ResponseParser, Excon::Middleware::Expects, Excon::Middleware::Idempotent, Excon::Middleware::Instrumentor, Excon::Middleware::Mock], :mock=>false, :nonblock=>true, :omit_default_port=>false, :persistent=>false, :read_timeout=>60, :retry_limit=>4, :ssl_verify_peer=>true, :tcp_nodelay=>false, :thread_safe_sockets=>true, :uri_parser=>URI, :versions=>"excon/0.45.4 (x86_64-darwin15) ruby/2.3.1", :write_timeout=>60, :host=>"ec2.us-east-1.amazonaws.com", :hostname=>"ec2.us-east-1.amazonaws.com", :path=>"/", :port=>443, :query=>nil, :scheme=>"https"} @socket_key="https://ec2.us-east-1.amazonaws.com:443">>>