Rabbitmq receiving blank messages

[User123]

I am using Rabbitmq libraries for node.js and golang.

The Node client sends request and Go server process it. These requests response are send via Rabbitmq.

Everything works fine but recently I have encountered a serious issue. And this is happening randomly.

Here's the exact scenario.

-Node.js publishes 10 messages at a time.

-Golang process this 10 messages(Prefetch count is 10) and keeps on sending acknowledgment for completed messages(using another queue)

-After all the 10 messages are acknowledged node.js send another 10 messages.

But it sometimes happens that Golang receives blank messages(delivery.Body == nil) After that it(Golang server) just doesn't receive other messages. so because of that the remaining messages are left in the queue(Ready state). Nor does it receive any new messages that are sent in the queue. It receives the message only when the Golang server is restarted.

This generally happens when blank message is received.

I am not able to find why is it happening.

Can you please help as it is a critical issue for us.

Thank you.

[Michael Klishin]

On 27 May 2015 at 09:22:36, User123 (twist...@gmail.com) wrote:
> But it sometimes happens that Golang receives blank messages(delivery.Body
> == nil) After that it(Golang server) just doesn't receive other
> messages. so because of that the remaining messages are left
> in the queue(Ready state). Nor does it receive any new messages
> that are sent in the queue. It receives the message only when the
> Golang server is restarted.

Sounds like something causes a channel in the client to be closed. See RabbitMQ
log files for clues. I don't know if the Go client can be indicating channel
errors with blank deliveries (intentionally or not). 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

 On 27 May 2015 at 09:45:47, Michael Klishin (mkli...@pivotal.io) wrote:
> Sounds like something causes a channel in the client to be closed.
> See RabbitMQ
> log files for clues.

You can also use Wireshark or RabbitMQ Tracer [1] to see what's
coming up and down the wire. Firehose and rabbitmq-tracing can be used as
alternatives that only capture published messages [2].

1. http://rabbitmq.com/java-tools.html
2. http://www.rabbitmq.com/firehose.html

[User123]

Hi,

Thanks for the suggestions.

I performed the check again and then looked into the log file when a file delivery was received.

This is what it shows

=INFO REPORT==== 27-May-2015::12:28:36 ===

accepting AMQP connection <0.4443.0> (127.0.0.1:2598 -> 127.0.0.1:5672)

=INFO REPORT==== 27-May-2015::12:29:31 ===

accepting AMQP connection <0.4485.0> (127.0.0.1:2607 -> 127.0.0.1:5672)

=ERROR REPORT==== 27-May-2015::12:29:37 ===

Channel error on connection <0.4443.0> (127.0.0.1:2598 -> 127.0.0.1:5672, vhost: '/', user: 'guest'), channel 1:

{amqp_error,precondition_failed,"unknown delivery tag 1",'basic.ack'}

What does it indicate? Which place should I look into.

This doesn't happen every time though.

[Michael Klishin]

 On 27 May 2015 at 10:05:44, User123 (twist...@gmail.com) wrote:
> =ERROR REPORT==== 27-May-2015::12:29:37 ===
> Channel error on connection <0.4443.0> (127.0.0.1:2598 ->
> 127.0.0.1:5672, vhost: '/', user: 'guest'), channel 1:
> {amqp_error,precondition_failed,"unknown delivery tag
> 1",'basic.ack'}

Your application either acks on the wrong channel (not the same
the delivery was on) or double-acks deliveries.

[User123]

I don't think the deliveries are doubly acked.

But since the queue needs to be accessed from two different apps(node.js and Golang) for sending and receiving. I am creating the channels and queue on both the sides. 

So is that creating the problem?

[Michael Klishin]

 On 27 May 2015 at 10:27:07, User123 (twist...@gmail.com) wrote:
> I don't think the deliveries are doubly acked.

Then you can be consuming with automatic acknowledgements but trying to ack
a message.

> But since the queue needs to be accessed from two different apps(node.js
> and Golang) for sending and receiving. I am creating the channels
> and queue on both the sides.
> So is that creating the problem?

No. I'm talking about different channels on a single connection.

You haven't specified what node.js library you are using but if it is node-amqp,
keep in mind that it is abandonware that has no shortage of known bugs. Use amqp.lib.

[User123]

Hi,

Both the channels had auto_ack field set to false.

And I am using the node.js library https://github.com/squaremo/amqp.node

On further debugging I have found the exact scenario where this happens.

Here it is:

- Server (Golang) receives 5 messages with delivery 1,2,3,4,5.

-Now the Server sends acknowledgement in this order 1,2,4,5,3

-So now the error "unknown delivery tag 3" is thrown

So error arise when the deliveries are not acknowledged in the order they are received.

Is there a way to fix this?

[Jean-Sébastien Pédron]

On 28.05.2015 09:17, User123 wrote:
> Hi,

Hi!

> Here it is:
> - Server (Golang) receives 5 messages with delivery 1,2,3,4,5.
> -Now the Server sends acknowledgement in this order 1,2,4,5,3
> -So now the error "unknown delivery tag 3" is thrown
>

> Is there a way to fix this?

What do you use for the "allUpTo" argument to #ack()?

--
Jean-Sébastien Pédron
Pivotal / RabbitMQ

[User123]

Hi,

I am not suppling any allupto parametr when doing ack. Is thats the cause.

Where should it be provided for client or server?

On Server I do this:

delivery.Ack(true)

[Jean-Sébastien Pédron]

On 28.05.2015 10:07, User123 wrote:
> I am not suppling any allupto parametr when doing ack. Is thats the cause.
>
> Where should it be provided for client or server?
>
> On Server I do this:
> delivery.Ack(true)

I'm referring to this documentation:
http://www.squaremobius.net/amqp.node/doc/channel_api.html#toc_73

I don't know Javascript, so I don't if it is the correct function to
look at.

This function takes a optional argument, allUpTo, which indicates if
previous messages should be acknowledged (if allUpTo is true) or not
(allUpTo is false or omitted).

I guess "delivery.Ack(true)" means allUpTo is true (though I may read
this incorrectly). In this case, when you acknowledge message #4,
message #3 is acknowledged at the same time. When you explicitely
acknowledge it again at the end, this raises an error because the server
doesn't know this message anymore.

[Michael Klishin]

On 28 May 2015 at 11:23:14, Jean-Sébastien Pédron (jean-se...@rabbitmq.com) wrote:
> > On Server I do this:
> > delivery.Ack(true)
>
> I'm referring to this documentation:
> http://www.squaremobius.net/amqp.node/doc/channel_api.html#toc_73
>
> I don't know Javascript, so I don't if it is the correct function
> to
> look at.

amqp.lib defaults to false if the parameter is omitted, as far as I can tell:
https://github.com/squaremo/amqp.node/blob/master/lib/api_args.js#L277-282

and the capital A in Ack suggests the function is invoked on the Go side.

In the actual client, Ack takes 2 explicit argument, so there may be a wrapper
involved:
https://github.com/streadway/amqp/blob/master/channel.go#L1555-1560

I highly recommend to the OP to stop the guess work and see what's being actually
sent with Wireshark or Tracer. 

[User123]

Yes you are right. The allUpTo flag was set to true which was the real cause. Was totally unaware of this flag. My mistake.

Thank you both for all the help and suggestions.

[Alvaro Videla]

You can use that flag to improve performance in combination with the right basic.qos setting:

See: https://www.rabbitmq.com/blog/2012/05/11/some-queuing-theory-throughput-latency-and-bandwidth/

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.