Re: DynamoDB authentication error

[John Capello]

I'm experiencing exactly the same problem.  Trevor, any luck in avoiding that error?

Thanks,

John

On Thursday, June 20, 2013 5:22:12 PM UTC-4, Trevor Blackford wrote:

Hi,

I just installed boto and am trying to use it to create tables and load data into DynamoDB. When I try to simply create a table I get a JSONResponseError that the security token included in the request is invalid.

>>> from boto.dynamodb2.table import Table

>>> from boto.dynamodb2.fields import HashKey

>>> users = Table.create('users', schema=[HashKey('username')])

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "boto/dynamodb2/table.py", line 196, in create

    **kwargs

  File "boto/dynamodb2/layer1.py", line 368, in create_table

    body=json.dumps(params))

  File "boto/dynamodb2/layer1.py", line 1479, in make_request

    retry_handler=self._retry_handler)

  File "boto/connection.py", line 854, in _mexe

    status = retry_handler(response, i, next_sleep)

  File "boto/dynamodb2/layer1.py", line 1521, in _retry_handler

    data)

boto.exception.JSONResponseError: JSONResponseError: 400 Bad Request

{u'message': u'The security token included in the request is invalid.', u'__type': u'com.amazon.coral.service#UnrecognizedClientException'}

I verified that my credentials work for other resources, namely S3:

>>> s3 = boto.connect_s3()

>>> bucket = s3.get_bucket('XXXXX')

>>> key = bucket.new_key('new-test-key')

>>> key.set_contents_from_string('This is a test of S3')

20

The IAM policy for my user is pretty comprehensive - 

  {

  "Statement": [

    {

      "Effect": "Allow",

      "Action": "*",

      "Resource": "*"

    }

  ]

}

And I can create/delete tables using the web interface. So I'm wondering what I'm missing in my boto configuration. I've included default region info for DynamoDB in my .boto file

Any help would be appreciated.

Thanks,

Trevor  

[and...@kukido.com]

John and Trevor,
Were you able to resolve the issue?
Cheers,
Andrey

[dd]

You have done something wrong with the syntax of your query/code.

The examples in the documentation are woefully short and incomplete.

Try getting the code in this question to work on your system http://stackoverflow.com/questions/20411475/python-amazon-dynamodb-how-to-reliably-delete-a-database-using-boto

Let me know how you go.

[dd]

Please post your code that creates the table.

[and...@kukido.com]

Dd,

I believe I narrowed it down to the following:

Initially I had AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in boto.cfg:

[ec2-user@ip-172-xx-xx-xx etc]$ pwd
/etc
[ec2-user@ip-172-xx-xx-xx etc]$ ls -ltra |grep boto
-rw-r--r--  1 root root     168 Dec 10 20:16 boto.cfg
[ec2-user@ip-172-xx-xx-xx etc]$ cat boto.cfg
[Credentials]
aws_access_key_id = xxx
aws_secret_access_key = xxx

[DynamoDB]
region = us-west-1

[Boto]
debug = 1

Now, it seem that the configuration file is not being picked up by boto. Only after I specified AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on Elastic Beanstalk Console, the error went away.

Somehow boto cannot read the configuration file. Is there anything I can do to debug it? The funny thing is that boto still connects to us-west-1, although the default region is us-east-1 :-/

[Adrian Klaver]

On 12/10/2013 12:39 PM, and...@kukido.com wrote:
> Dd,
>
> I believe I narrowed it down to the following:
>
> Initially I had AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in boto.cfg:
>
> [ec2-user@ip-172-xx-xx-xx etc]$ pwd
> /etc
> [ec2-user@ip-172-xx-xx-xx etc]$ ls -ltra |grep boto
> -rw-r--r-- 1 root root 168 Dec 10 20:16 boto.cfg
> [ec2-user@ip-172-xx-xx-xx etc]$ cat boto.cfg
> [Credentials]
> aws_access_key_id = xxx
> aws_secret_access_key = xxx
>
> [DynamoDB]
> region = us-west-1
>
> [Boto]
> debug = 1
>
> Now, it seem that the configuration file is not being picked up by boto.
> Only after I specified AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on
> Elastic Beanstalk Console, the error went away.

The above would imply it did work at some time. Any changes of note in
the meantime?

>
> Somehow boto cannot read the configuration file. Is there anything I can
> do to debug it? The funny thing is that boto still connects to
> us-west-1, although the default region is us-east-1 :-/

Have you tried a ~/.boto file to see if its a permissions problem?

Any output from debug?

>


--
Adrian Klaver
adrian...@gmail.com

[and...@kukido.com]

Adrian,

The note still the same:

[Tue Dec 10 21:09:51 2013] [error] [client 216.38.150.34] JSONResponseError: JSONResponseError: 400 Bad Request
[Tue Dec 10 21:09:51 2013] [error] [client 216.38.150.34] {u'message': u'The security token included in the request is invalid.', u'__type': u'com.amazon.coral.service#UnrecognizedClientException'}

As you suggested, I tried copy configuration file to ~/.boto

Here are my commands:

    50-copy-boto-configuration-to-home:
        command: "cp .ebextensions/boto.cfg ~/.boto"

    60-list-home-directory:
        command: ls -ltra ~/

And the output:

2013-12-10 21:04:26,835 [DEBUG] Command 60-list-home-directory output: total 36
-rw-r--r--  1 root root  129 Jan 15  2011 .tcshrc
-rw-r--r--  1 root root  100 Jan 15  2011 .cshrc
-rw-r--r--  1 root root  176 Jan 15  2011 .bashrc
-rw-r--r--  1 root root  176 Jan 15  2011 .bash_profile
-rw-r--r--  1 root root   18 Jan 15  2011 .bash_logout
dr-xr-xr-x 23 root root 4096 Dec 10 19:46 ..
drwx------  2 root root 4096 Dec 10 19:46 .ssh
dr-xr-x---  3 root root 4096 Dec 10 20:58 .
-rw-rw-rw-  1 root root  169 Dec 10 21:04 .boto

Listing for ec2-user home does not contain .boto:

[ec2-user@ip-172-xx-xx-xx ~]$ pwd
/home/ec2-user
[ec2-user@ip-172-xx-xx-xx ~]$ ls -ltra
total 24
-rw-r--r-- 1 ec2-user ec2-user  124 Sep  4 18:23 .bashrc
-rw-r--r-- 1 ec2-user ec2-user  176 Sep  4 18:23 .bash_profile
-rw-r--r-- 1 ec2-user ec2-user   18 Sep  4 18:23 .bash_logout
drwxr-xr-x 3 root     root     4096 Sep 25 01:41 ..
drwx------ 2 ec2-user ec2-user 4096 Dec 10 19:46 .ssh
drwx------ 3 ec2-user ec2-user 4096 Dec 10 19:46 .

Should I copy the configuration file to /home/ec2-user instead? Or I can add commands to update ownership for /etc/boto.cfg. It seems that the installation and configuration being run by root, yet the application code is executed by ec2-user.

Cheers,
Andrey

[Adrian Klaver]

On 12/10/2013 01:15 PM, and...@kukido.com wrote:
> Adrian,
>
> The note still the same:
>
> [Tue Dec 10 21:09:51 2013] [error] [client 216.38.150.34] JSONResponseError: JSONResponseError: 400 Bad Request
> [Tue Dec 10 21:09:51 2013] [error] [client 216.38.150.34] {u'message': u'The security token included in the request is invalid.', u'__type': u'com.amazon.coral.service#UnrecognizedClientException'}
>
>
> As you suggested, I tried copy configuration file to ~/.boto
>
> Here are my commands:
>
> 50-copy-boto-configuration-to-home:
> command: "cp .ebextensions/boto.cfg ~/.boto"
>
> 60-list-home-directory:
> command: ls -ltra ~/

So where are you running these commands from and what are they? In other
words what program are you using to interact with the machine?

>
>
> Should I copy the configuration file to /home/ec2-user instead? Or I can
> add commands to update ownership for /etc/boto.cfg. It seems that the
> installation and configuration being run by root, yet the application
> code is executed by ec2-user.

Yes you just added .boto to what looks to be roots home directory. Can't
you just log into the server and cp from /etc/boto.cfg to the
/home/ec2-user/.boto and update the permissions.



>
> Cheers,
> Andrey
>


--
Adrian Klaver
adrian...@gmail.com

[Andrey Myatlyuk]

Hi Adrian,

> On Dec 10, 2013, at 1:36 PM, Adrian Klaver <adrian...@gmail.com> wrote:
>
>> On 12/10/2013 01:15 PM, and...@kukido.com wrote:
>> Adrian,
>>
>> The note still the same:
>>
>> [Tue Dec 10 21:09:51 2013] [error] [client 216.38.150.34] JSONResponseError: JSONResponseError: 400 Bad Request
>> [Tue Dec 10 21:09:51 2013] [error] [client 216.38.150.34] {u'message': u'The security token included in the request is invalid.', u'__type': u'com.amazon.coral.service#UnrecognizedClientException'}
>>
>>
>> As you suggested, I tried copy configuration file to ~/.boto
>>
>> Here are my commands:
>>
>> 50-copy-boto-configuration-to-home:
>> command: "cp .ebextensions/boto.cfg ~/.boto"
>>
>> 60-list-home-directory:
>> command: ls -ltra ~/
>
> So where are you running these commands from and what are they? In other words what program are you using to interact with the machine?

Ah, I was so deep into my project, that forgot to mention the environment. I'm using boto on AWS Elastic Beanstalk. And those commands are executed by the container initializer. I have ssh access to instances, but the idea to bring maximum automation.

>
>>
>>
>> Should I copy the configuration file to /home/ec2-user instead? Or I can
>> add commands to update ownership for /etc/boto.cfg. It seems that the
>> installation and configuration being run by root, yet the application
>> code is executed by ec2-user.
>
> Yes you just added .boto to what looks to be roots home directory. Can't you just log into the server and cp from /etc/boto.cfg to the /home/ec2-user/.boto and update the permissions.

I can definitely copy boto.cfg to ec2-user home. What permissions should I set on the file?

>
>
>
>>
>> Cheers,
>> Andrey
>
>
> --
> Adrian Klaver
> adrian...@gmail.com
>

> --
> You received this message because you are subscribed to a topic in the Google Groups "boto-users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/boto-users/M0Gnuz5n9qs/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to boto-users+...@googlegroups.com.
> To post to this group, send email to boto-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/boto-users.
> For more options, visit https://groups.google.com/groups/opt_out.

[Adrian Klaver]

On 12/10/2013 01:46 PM, Andrey Myatlyuk wrote:

>>> Here are my commands:
>>>
>>> 50-copy-boto-configuration-to-home:
>>> command: "cp .ebextensions/boto.cfg ~/.boto"
>>>
>>> 60-list-home-directory:
>>> command: ls -ltra ~/
>>
>> So where are you running these commands from and what are they? In other words what program are you using to interact with the machine?
> Ah, I was so deep into my project, that forgot to mention the environment. I'm using boto on AWS Elastic Beanstalk. And those commands are executed by the container initializer. I have ssh access to instances, but the idea to bring maximum automation.

Well my method when troubleshooting is to work from simple to complex.
To that end I strip out as much of the overlying code as possible and
get back down to the basics and make sure they work and then add back
the layers one by one to see if I can identify the issue(s). So in this
case run boto independent of Beanstalk and see what happens.

>
>>
>>>
>>>
>>> Should I copy the configuration file to /home/ec2-user instead? Or I can
>>> add commands to update ownership for /etc/boto.cfg. It seems that the
>>> installation and configuration being run by root, yet the application
>>> code is executed by ec2-user.
>>
>> Yes you just added .boto to what looks to be roots home directory. Can't you just log into the server and cp from /etc/boto.cfg to the /home/ec2-user/.boto and update the permissions.
> I can definitely copy boto.cfg to ec2-user home. What permissions should I set on the file?

I would say the permissions for the home directory user.

>>
>>
>>
>>>
>>> Cheers,
>>> Andrey
>>
>>
>> --
>> Adrian Klaver
>> adrian...@gmail.com
>>



--

Adrian Klaver
adrian...@gmail.com

[and...@kukido.com]

On Tuesday, December 10, 2013 1:54:53 PM UTC-8, aklaver wrote:

On 12/10/2013 01:46 PM, Andrey Myatlyuk wrote:

>>> Here are my commands:
>>>
>>>      50-copy-boto-configuration-to-home:
>>>          command: "cp .ebextensions/boto.cfg ~/.boto"
>>>
>>>      60-list-home-directory:
>>>          command: ls -ltra ~/
>>
>> So where are you running these commands from and what are they? In other words what program are you using to interact with the machine?
> Ah, I was so deep into my project, that forgot to mention the environment. I'm using boto on AWS Elastic Beanstalk. And those commands are executed by the container initializer. I have ssh access to instances, but the idea to bring maximum automation.

Well my method when troubleshooting is to work from simple to complex.
To that end I strip out as much of the overlying code as possible and
get back down to the basics and make sure they work and then add back
the layers one by one to see if I can identify the issue(s). So in this
case run boto independent of Beanstalk and see what happens.

Well, that would be too logical :-)

I tried to run boto from command line, and it works just fine, connecting to the proper region, I'm able to pull the data. Without layers it works great. Yet, within the application it still does not work.

The code is the same:

    from boto.dynamodb2.table import Table
    people = Table('person')
    jane = people.get_item(permalink='jane-doe')
    print 'permalink: %s, inserted_at: %s' % (jane['permalink'], jane['inserted_at'])
 
What makes it even more confusing is that when I dump boto configuration info, it contains the same configuration parameters in both cases:

[ec2-user@ip-172-xx-xx-xx app]$ python
Python 2.6.8 (unknown, Mar 14 2013, 09:31:22) 
[GCC 4.6.2 20111027 (Red Hat 4.6.2-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from boto.pyami.config import Config
>>> config = Config()
>>> config.dump()
[DEFAULT]
debug = 0
working_dir = /mnt/pyami

[DynamoDB]
region = us-west-1

[Credentials]
aws_access_key_id = (proper value from config file) 
aws_secret_access_key = (proper value from config file)
[Boto]
debug = 1


>>> quit()

Now, within my application I added method with exactly the same code:

@application.route('/boto')
def boto():
    from boto.pyami.config import Config
    config = Config()
    config.dump()

The output:

[Tue Dec 10 23:36:22 2013] [error] [DEFAULT]
[Tue Dec 10 23:36:22 2013] [error] debug = 0
[Tue Dec 10 23:36:22 2013] [error] working_dir = /mnt/pyami
[Tue Dec 10 23:36:22 2013] [error] 
[Tue Dec 10 23:36:22 2013] [error] [Credentials]
[Tue Dec 10 23:36:22 2013] [error] aws_access_key_id = (proper value from config file)
[Tue Dec 10 23:36:22 2013] [error] aws_secret_access_key = (proper value from config file)
[Tue Dec 10 23:36:22 2013] [error] 
[Tue Dec 10 23:36:22 2013] [error] [DynamoDB]
[Tue Dec 10 23:36:22 2013] [error] region = us-west-1
[Tue Dec 10 23:36:22 2013] [error] 
[Tue Dec 10 23:36:22 2013] [error] [Boto]
[Tue Dec 10 23:36:22 2013] [error] debug = 1

AWS credentials are not set anywhere else but in the file. If I tweak the credentials in ~/.boto, I get new error:

boto.exception.JSONResponseError: JSONResponseError: 400 Bad Request

{u'Message': u'User: arn:aws:sts::153241250259:assumed-role/aws-elasticbeanstalk-ec2-role/i-3a26aa67 is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-west-1:153241250259:table/person', u'__type': u'com.amazon.coral.service#AccessDeniedException'}

What am I missing here?

[Adrian Klaver]

I am not that familiar with Beanstalk, but my best guess is it has to do
with:

"assumed-role/aws-elasticbeanstalk-ec2-role/i-3a26aa67
is not authorized to perform: dynamodb:GetItem on resource:"

from the exception. It would seem the Beanstalk role is not allowed
access to dynamoDB.

--
Adrian Klaver
adrian...@gmail.com

[and...@kukido.com]

Yes, because that was the error message after I tweaked conf file with incorrect keys to confirm that it is the only source.

Now, with correct keys I still have the same "token" error message:

[Wed Dec 11 00:37:20 2013] [error] [client 216.38.150.34] JSONResponseError: JSONResponseError: 400 Bad Request
[Wed Dec 11 00:37:20 2013] [error] [client 216.38.150.34] {u'message': u'The security token included in the request is invalid.', u'__type': u'com.amazon.coral.service#UnrecognizedClientException'}

 

--
Adrian Klaver
adrian...@gmail.com

[Adrian Klaver]

On 12/10/2013 04:44 PM, and...@kukido.com wrote:

>
> Yes, because that was the error message after I tweaked conf file with
> incorrect keys to confirm that it is the only source.
>
> Now, with correct keys I still have the same "token" error message:
>
> [Wed Dec 11 00:37:20 2013] [error] [client 216.38.150.34] JSONResponseError: JSONResponseError: 400 Bad Request
> [Wed Dec 11 00:37:20 2013] [error] [client 216.38.150.34] {u'message': u'The security token included in the request is invalid.', u'__type': u'com.amazon.coral.service#UnrecognizedClientException'}

The only thing I can think of at the moment is that your application is
creating some sort of encoding issue that is mangling the
aws_access_key_id and/or aws_secret_access_key. Will do some more
pondering to see if anything else comes to mind.

>
> --
> Adrian Klaver
> adrian...@gmail.com <javascript:>
>



--
Adrian Klaver
adrian...@gmail.com

[Adrian Klaver]

On 12/10/2013 04:44 PM, and...@kukido.com wrote:

>
> >

> > Now, within my application I added method with exactly the same
> code:
> >
> > @application.route('/boto')
> > def boto():
> > from boto.pyami.config import Config
> > config = Config()
> > config.dump()

> Yes, because that was the error message after I tweaked conf file with
> incorrect keys to confirm that it is the only source.
>
> Now, with correct keys I still have the same "token" error message:
>
> [Wed Dec 11 00:37:20 2013] [error] [client 216.38.150.34] JSONResponseError: JSONResponseError: 400 Bad Request
> [Wed Dec 11 00:37:20 2013] [error] [client 216.38.150.34] {u'message': u'The security token included in the request is invalid.', u'__type': u'com.amazon.coral.service#UnrecognizedClientException'}

Something else came to mind. How exactly is your application calling the
boto code? Is it possible the error is coming from something other than
the dynamoDB request?



--
Adrian Klaver
adrian...@gmail.com

[and...@kukido.com]

I found the difference between two calls - from command line and from within application.

In command line the request looks like:
send: 'POST /? HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 90\r\nX-Amz-Target: DynamoDB_20120810.GetItem\r\nHost: dynamodb.us-west-1.amazonaws.com\r\nX-Amz-Date: 20131211T052752Z\r\nUser-Agent: Boto/2.19.0 Python/2.7.5 Linux/3.4.62-53.42.amzn1.x86_64\r\nContent-Type: application/x-amz-json-1.0\r\nAuthorization: AWS4-HMAC-SHA256 Credential=<actual-aws-access-key-id>/20131211/us-west-1/dynamodb/aws4_request,SignedHeaders=host;x-amz-date;x-amz-target,Signature=5ee5f67a5ad7478474c0abe0381cedcb5f8d0da923e05ccb7f73a51dd9c54380\r\n\r\n{"ConsistentRead": false, "TableName": "person", "Key": {"permalink": {"S": "jane-doe"}}}'

reply: 'HTTP/1.1 200 OK\r\n'

In the application it looks like:
[Wed Dec 11 05:28:12 2013] [error] send: 'POST /? HTTP/1.1\\r\\nAccept-Encoding: identity\\r\\nContent-Length: 90\\r\\nX-Amz-Target: DynamoDB_20120810.GetItem\\r\\nHost: dynamodb.us-west-1.amazonaws.com\\r\\nX-Amz-Date: 20131211T052812Z\\r\\nUser-Agent: Boto/2.19.0 Python/2.7.5 Linux/3.4.62-53.42.amzn1.x86_64\\r\\nContent-Type: application/x-amz-json-1.0\\r\\nAuthorization: AWS4-HMAC-SHA256 Credential=/20131211/us-west-1/dynamodb/aws4_request,SignedHeaders=host;x-amz-date;x-amz-target,Signature=<long signature>f\\r\\n\\r\\n{"ConsistentRead": false, "TableName": "person", "Key": {"permalink": {"S": "jane-doe"}}}'

[Wed Dec 11 05:28:12 2013] [error] reply: 'HTTP/1.1 400 Bad Request\\r\\n'

So, I added some code to confirm whether the key is passed:

import os
print 'key: %s' % os.getenv('AWS_ACCESS_KEY_ID')

In command line it will print out as
key: None

In the application it will print out as
key:

It means that AWS Elastic Beanstalk environment sets the key to empty string, even though it should not. And I guess that value has higher priority than the value from boto configuration file.

I will follow-up with AWS on why the value is set in the first place.

Side note on the debug log. I had to add the following to my application code:

import logging
logging.getLogger('boto').setLevel(logging.DEBUG)

Only after that I started to get boto debug messages. I did not see it being mentioned anywhere in the documentation. Is that the right thing to do?

--
Adrian Klaver
adrian...@gmail.com

[Adrian Klaver]

On 12/10/2013 09:59 PM, and...@kukido.com wrote:

>
> I found the difference between two calls - from command line and from
> within application.
>
> In command line the request looks like:
> send: 'POST /? HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length:
> 90\r\nX-Amz-Target: DynamoDB_20120810.GetItem\r\nHost:
> dynamodb.us-west-1.amazonaws.com\r\nX-Amz-Date:
> 20131211T052752Z\r\nUser-Agent: Boto/2.19.0 Python/2.7.5
> Linux/3.4.62-53.42.amzn1.x86_64\r\nContent-Type:
> application/x-amz-json-1.0\r\nAuthorization: AWS4-HMAC-SHA256

> Credential=*<actual-aws-access-key-id>*/20131211/us-west-1/dynamodb/aws4_request,SignedHeaders=host;x-amz-date;x-amz-target,Signature=5ee5f67a5ad7478474c0abe0381cedcb5f8d0da923e05ccb7f73a51dd9c54380\r\n\r\n{"ConsistentRead":

From previous posts, it look you where using Flask so:

http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html

Configure AWS Elastic Beanstalk

Looks like the bundle application asks for the id and key on set up. You
might have set them to empty strings. From the docs it seems you can use
init to renter the values.

[and...@kukido.com]

That's the thing. I did not set those settings to empty strings. It seems that even though the documentations says that the values are not set, they are set to empty strings by the container. 

I guess I will settle on setting the keys in the container configuration. "Just when I thought I was out... they pull me back in."

Thank you for your help in the investigation, Adrian!

Cheers,
Andrey

[Adrian Klaver]

On 12/11/2013 11:07 AM, and...@kukido.com wrote:

>
>
> I guess I will settle on setting the keys in the container
> configuration. "Just when I thought I was out... they pull me back in."
>
> Thank you for your help in the investigation, Adrian!

The part that has me puzzled is that given a choice between an env
variable with an empty string and a config variable that is not empty it
chooses the empty string. At any rate, good luck.

[mitch]

Environment variables and config file values take precedence over IAM Role credentials and I think that makes sense.  However, we could add something to boto to check for empty strings or None values and skip if we find them.  Or we could figure out why Beanstalk is setting those env vars to empty values and try to fix that.

Mitch

[Andrey Myatlyuk]

I asked about that on AWS forum, waiting for a response from Beanstalk team:
https://forums.aws.amazon.com/thread.jspa?messageID=509096

It is easy to reproduce the condition locally:

$ export AWS_ACCESS_KEY_ID=

After that any command will result in error, even if you have the values set in the configuration file.

If empty strings and None can be ignored by boto, it would be great. Let me know if I can help in any way. Are there any corner case scenarios, when you positively want to set the value to empty string?

Cheers,
Andrey

--