Polling and consuming messages from Amazon SQS as a service

[tao.hong via StackOverflow]

I am using a SQS as a 'bridge' in my system, which receives tasks from GAE and will be processed on the EC2. Currently, I am able to add tasks to this queue from GAE, but with some difficulties on how to consume these tasks on EC2. So my questions are:

1. What are the recommended ways of building a task consumer (python based) on EC2, which would keep an eye on SQS and assign new inbound jobs to works?
2. Does AWS has this type of SQS monitoring product? If not, is Celery's period task a good candidate?

Please DO NOT REPLY directly to this email but go to StackOverflow:
http://stackoverflow.com/questions/21790592/polling-and-consuming-messages-from-amazon-sqs-as-a-service

[sallie via StackOverflow]

The task consumer is just a (long) poller. The boto Python library has SQS support. AWS provides the SQS service, but they don't make the consumers. I'm not familiar with Celery.

Standard practice is to poll for message (which marks it as 'invisible'), and then perform action at consumer end. At action completion, delete message. If action fails because one of your compute nodes disappeared the message will become visible after a period of time and it'll be picked up in a future poll. If you need something smarter you may want to implement an external ESB or experiment with AWS SWF.

http://boto.readthedocs.org/en/latest/ref/sqs.html

http://aws.amazon.com/swf/

Please DO NOT REPLY directly to this email but go to StackOverflow:

http://stackoverflow.com/questions/21790592/polling-and-consuming-messages-from-amazon-sqs-as-a-service/21790931#21790931