Re: [boto-users] Garbled data when subscribing to SNS topic with SQS?

[Patrick Dunnigan]

I'd suggest postings a subset of your code that reproduces the problem. 

That way the community can see what you are doing and make suggestions. 

On Aug 10, 2012, at 5:49 AM, Csillag wrote:

Hi,

I am new to boto (and AWS in general, as well), and I have encountered a strange bug.

If I create an SNS topic, and an SQS queue, and subscribe the queue to the topic (with boto's subscribe_sqs_queue() API), then messages published on the topic are often corrupted, when they come out of the queue.

If I call message.get_body() on them, I get garbled binary data.

Here is what I know:

 - If I inject messages to the queue directly (not via the topic), then there is no problem, only if I use the topic->queue data path

 - I can receive the messages sent to the topic if I am subscribed via email without any problem, only the queue delivery fails

 - Whether the problem manifests seems to depend on the content I am sending. For example "random text" does not trigger it, but "random text message" does.

 - Although the problem appears to be triggered by the content of the message published to the topic, simply base64-encoding it before transfer does not help.

 - If (instead of using boto) I use Amazon's Java SDK to read the messages from SQS queue, there is no corruption: the same message can be read properly.

What could be the reason of this?

Thank you for your help:

   Csillag

--
You received this message because you are subscribed to the Google Groups "boto-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/boto-users/-/xbvv7MlumjkJ.
To post to this group, send email to boto-...@googlegroups.com.
To unsubscribe from this group, send email to boto-users+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/boto-users?hl=en.

[Mitchell Garnaat]

This blog post (and especially the comments) might prove useful:

http://www.elastician.com/2010/04/subscribing-sqs-queue-to-sns-topic.html

Mitch

[Victor Trac]

I recently wrote some code to consume SNS->SQS (for autoscale events) as well and ran into this problem and using RawMessage did fix the issues. However, what I found strange was that the errors weren't consistent. Before I set the message class to RawMessage, my boto SQS code would sometimes be able to read the message. I would have expected it to work or not, but I didn't spend much time investigating after setting RawMessage. Does anyone have thoughts as to how this could have happened?

--
Victor Trac  |  victortrac.com  |  twitter.com/victortrac  

On Tue, Sep 11, 2012 at 8:26 AM, Nika Jones <nikajo...@gmail.com> wrote:

Hello Csillag 

What's happening is the Message is expected to be encoded in base64 but when it comes from SNS Amazon doesn't do that.

So you need to pull back a RawMessage object and not the usual Message object (which does the automatic encoding)

Try to use the following code:

from boto.sqs.message import RawMessage

def que():

    sqs = SQSConnection(<access_key>, <secret_key>)

    queue = sqs.lookup(my_queue)

    queue.set_message_class(RawMessage) # This is the Magic line.

    for msg in queue.get_messages():

        print msg.get_body()

Nika

On Friday, August 10, 2012 2:49:14 AM UTC-7, Csillag wrote:

Hi,

I am new to boto (and AWS in general, as well), and I have encountered a strange bug.

If I create an SNS topic, and an SQS queue, and subscribe the queue to the topic (with boto's subscribe_sqs_queue() API), then messages published on the topic are often corrupted, when they come out of the queue.

If I call message.get_body() on them, I get garbled binary data.

Here is what I know:

 - If I inject messages to the queue directly (not via the topic), then there is no problem, only if I use the topic->queue data path

 - I can receive the messages sent to the topic if I am subscribed via email without any problem, only the queue delivery fails

 - Whether the problem manifests seems to depend on the content I am sending. For example "random text" does not trigger it, but "random text message" does.

 - Although the problem appears to be triggered by the content of the message published to the topic, simply base64-encoding it before transfer does not help.

 - If (instead of using boto) I use Amazon's Java SDK to read the messages from SQS queue, there is no corruption: the same message can be read properly.

What could be the reason of this?

Thank you for your help:

   Csillag

--
You received this message because you are subscribed to the Google Groups "boto-users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/boto-users/-/oWixIFe3ubsJ.