Cloudwatch: get_metric_statistics() sometimes returns empty list

[Rich Brown]

I have written a script to retrieve Cloudwatch statistics from EC2
that makes this call to Boto 2.0rc1:

data_list = conn.get_metric_statistics(60, start,end,metric,'AWS/
EC2','Average', instance_id)

Much of the time (90% of the time), this returns the expected
information. However, sometimes data_list comes back empty. I have a
(partial) theory that if I re-issue the call frequently (every couple
seconds) it will fail, and then continue to fail until I pause for
many (60 or so) seconds.

Test case:

I see this behavior consistently on OSX 10.6 using Python 2.6.4, using
the following command line. Download the script.awscloudwatch.py from:
http://download.intermapper.com/thirdparty/awscloudwatch.py

echo "aws_access_key_id aws_secret_access_key" | ./awscloudwatch.py
aws_ami_id CPUUtilization 50 75 95

Proper output will show something containing "CPUUtilization is 3.07
Percent"; when the query fails, the script returns "No data returned
for InstanceID "aws_ami_id""

Any ideas why this would happen? Or what additional information I
could collect? Many thanks.

Rich Brown
Hanover, NH

[Mitchell Garnaat]

This has come up before.  I think there is an issue related to this.  I investigated a bit at the time and couldn't really find any explanation but let's give it another try.

One useful thing to add to your test script would be to turn on debug output.  You can do this by changing your script slightly:

def get_metric_data(metric, instance_id, start, end):

# Make a connection to the AWS server system - always seems to return a conn, even if bad keys...

        boto.set_stream_logger('foo')
	conn = boto.connect_cloudwatch(aws_access_key_id, aws_secret_access_key, debug=2)
		

This should allow us to determine exactly what CloudWatch is send in it's responses.  I suspect the response itself is empty which would lead me to believe it's a normal behavior for CloudWatch.  If, on the other hand, data is returned then there must be a bug in boto that causes it fail to process the data properly.

Mitch

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.

[Rich Brown]

Bingo! (What great debugging tools :-)

Here's the output of two successive runs with debug=2 in the
boto.connect_cloudwatch() call. Your suspicion is correct - there's no
data in the response from Cloudwatch. (and thus, boto is working
properly. Whew!).

My question is not really one you can necessarily answer, but how can
I get more info about how to proceed? Are there forums at Amazon that
might talk about this? Thanks.

Rich

======== #1 - Didn't work, and the array is empty ==========

[dhcp-65:~/Documents/src/Assorted Programs] richb% echo "key_id
secret_access_key" | ./awscloudwatch.py i-e6f0cf89 CPUUtilization 50
75 95
These are the arguments passed from InterMapper: ['i-e6f0cf89',
'CPUUtilization', '50', '75', '95']
start time is: 2011-06-17 10:31:11.739478
end time is 2011-06-17 10:32:11.739478
send: 'GET /?
AWSAccessKeyId=xxxxxx&Action=GetMetricStatistics&Dimensions.member.
1.Name=InstanceId&Dimensions.member.1.Value=i-
e6f0cf89&EndTime=2011-06-17T10%3A32%3A11.739478&MetricName=CPUUtilization&Namespace=AWS
%2FEC2&Period=60&SignatureMethod=HmacSHA256&SignatureVersion=2&StartTime=2011-06-17T10%3A31%3A11.739478&Statistics.member.
1=Average&Timestamp=2011-06-17T10%3A32%3A12Z&Version=2010-08-01&Signature=xxxxxxxxx
%3D HTTP/1.1\r\nHost: monitoring.amazonaws.com\r\nAccept-Encoding:
identity\r\nContent-Length: 0\r\nUser-Agent: Boto/2.0rc1 (darwin)\r\n\r
\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: x-amzn-RequestId: 104ca1f6-98cd-11e0-89ea-b9b2d1693c08
header: Content-Type: text/xml
header: Content-Length: 337
header: Date: Fri, 17 Jun 2011 10:32:12 GMT
[]
\{ $CPUUtilization := "[n/a]" }No data returned for InstanceID "i-
e6f0cf89"


======== #2 - about 30 seconds later, worked just fine ==========

[dhcp-65:~/Documents/src/Assorted Programs] richb% echo "key_id
secret_access_key" | ./awscloudwatch.py i-e6f0cf89 CPUUtilization 50
75 95
These are the arguments passed from InterMapper: ['i-e6f0cf89',
'CPUUtilization', '50', '75', '95']
start time is: 2011-06-17 10:31:48.132818
end time is 2011-06-17 10:32:48.132818
send: 'GET /?
AWSAccessKeyId=xxxxxx&Action=GetMetricStatistics&Dimensions.member.
1.Name=InstanceId&Dimensions.member.1.Value=i-
e6f0cf89&EndTime=2011-06-17T10%3A32%3A48.132818&MetricName=CPUUtilization&Namespace=AWS
%2FEC2&Period=60&SignatureMethod=HmacSHA256&SignatureVersion=2&StartTime=2011-06-17T10%3A31%3A48.132818&Statistics.member.
1=Average&Timestamp=2011-06-17T10%3A32%3A48Z&Version=2010-08-01&Signature=xxxxxxxxxx
%3D HTTP/1.1\r\nHost: monitoring.amazonaws.com\r\nAccept-Encoding:
identity\r\nContent-Length: 0\r\nUser-Agent: Boto/2.0rc1 (darwin)\r\n\r
\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: x-amzn-RequestId: 25c379bb-98cd-11e0-afc5-3b252e0fd632
header: Content-Type: text/xml
header: Content-Length: 498
header: Date: Fri, 17 Jun 2011 10:32:48 GMT
[{u'Timestamp': datetime.datetime(2011, 6, 17, 10, 31), u'Average':
2.6299999999999999, u'Unit': u'Percent'}]
The Average for CPUUtilization is 2.63 Percent

CPUUtilization 2.63 is being used
starting method calculate_threshold with data: 2.63
Nagios Response code is 0
\{ $CPUUtilization := "2.63" }CPUUtilization is 2.63 Percent

[Mitchell Garnaat]

Here's a link to the AWS forums:

https://forums.aws.amazon.com/index.jspa

There isn't a specific forum for CloudWatch so just use the EC2 forum.  You might want to do a search first to see if the question has come up before.  In my experience, the forums are quite active and a good place to get answers to AWS-related questions.

Mitch

--

[Rich Brown]

Thanks for the pointer to the AWS forums - I think I found a pretty
good answer. The problem seems to be that the query was asking for a
one-minute slice of data, ending now(). I am guessing that AWS may not
have finished collecting that data sample (because "now" is pretty
recent :-) and as a consequence, it returns an empty response.

To test that theory, I have been running the query a couple ways:

1) One-minute slice, ending 30 seconds ago - this occasionally fails
to return data (5% of the time?)
2) One-minute slice, ending 60 seconds ago - this only failed once in
~1000 requests.

I'm going to continue to experiment with longer-term measurements
(five minute averages) or re-running the query in the empty result
case.

So the answer is that both boto and AWS Cloudwatch pieces work fine -
you just need to understand the parameters of the system you're
working with. (Duh!) Thanks for all your help.

Rich Brown
Hanover, NH

[Mitchell Garnaat]

Thanks for that update.  It's useful information!

Mitch

--

[Midhun Kandoth]

Hi,
I have been the facing the same issue you said. When i try to increase the data points i got an error like

<ErrorResponse xmlns="http://monitoring.amazonaws.com/doc/2010-08-01/">
  <Error>
    <Type>Sender</Type>
    <Code>InvalidParameterCombination</Code>
    <Message>You have requested up to 5,760 datapoints, which exceeds the limit of 1,440. You may reduce the datapoints requested by increasing Period, or decreasing the time range.</Message>
  </Error>
  <RequestId>c6b21634-d83a-11e4-9449-f5b60317b90e</RequestId>
</ErrorResponse>

 So i tried to tried many permutations with perid, start time and end time. But the result was either the above error or empty list. What could be the possible problem. Any hep is appreciated.

Midhun