Maximum length of routing key that binds with queue in AMQP

[Urmik Shah]

Hello,

I came to know that maximum length of routing key that binds with queue in AMQP is 255 ASCII characters.

I have tried with RabbitMQ .Net library and NODE AMQP library. From that I can't bind more than 255 characters routing key with queue.

But from RabbitMQ management plugin, I can bind more than 255 characters routing key with queue.

What should be reason? 

I want to extend the limit of routing key's length that binds with queue. How can I do this?

Regards,

Urmik 

[Michael Klishin]

Routing key is 255 characters long in the protocol (AMQP 0-9-1). You can't extend it.

The reason why you can do it via management is that management plugin runs in the same
VM as RabbitMQ and uses what's known as "direct client", a client that uses distributed
Erlang facilities instead of a TCP connection. So, protocol serialisation is not involved .
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alvaro Videla]

On Wed, Oct 8, 2014 at 4:05 PM, Michael Klishin <mkli...@pivotal.io> wrote:

The reason why you can do it via management is that management plugin runs in the same
VM as RabbitMQ and uses what's known as "direct client", a client that uses distributed
Erlang facilities instead of a TCP connection. So, protocol serialisation is not involved .

If this is the case, isn't this a bug?

-Alvaro

[Michael Klishin]

On 8 October 2014 at 17:26:39, Alvaro Videla (videl...@gmail.com) wrote:
> If this is the case, isn't this a bug?

Lack of validation, yes.

[Laing, Michael]

Actually I believe a routing key is a shortstr which is max 255 octets not ASCII. We use UTF8 encoding actually for some of ours.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Silva]

I read through the AMQP 0.9.1 standard, and it defines short strings ( len <= 255 ) and long strings ( len < 2**32-1 )
It doesn't seem to say anywhere that a 'topic' is a short string. 

How hard would it be to use 'long strings' for headers generally (including topic, routing keys, and queue names ?)
(yes, obviously I want longer keys... )

[Michael Klishin]

That would be a fair non-trivial spec violation and require updates to all client libraries. Don't hold your
breath on that happening any time soon, team RabbitMQ has much bigger fish to catch for the next few years. 

[Peter Silva]

I get that this is a significant change, but I'm trying to understand why you state that it is a 'spec violation' ...  Can you explain that with a specific reference?   I read the spec and it does not say that anywhere.   Earlir, I had innocently concluded that there was no such limit, and implemented something on that basis, only to get bitten after the fact.

The standard states that:
   -- AMQP sends frames, made of bits, integers, strings, and field tables  (2.3.1)
   -- an AMQP message is a method frame followed by zero or more content frames, here it refers to "string tables" instead of "Field tables" which are assumed to be the same. (2.3.5.1)
   -- there are short strings and long strings. (4.2.5.3)
   -- there is a field table, which is a long string packed with short strings.  (4.2.5.5)
  
The document does not state any field is a short string.  It does not state that that native fields are encoded into a field table.

As far as I can tell,  the spec does not say that these fields are short strings.   Many (all?) implementations obviously think they are, but I'm wondering if the implementations are actually violating the spec, and not the other way round.   or perhaps there is just a common interpretation by all the implementors that is not actually derived from the spec at all.

[Michael Klishin]

There are defined data types (domains) and fields in the spec. Changing a field type is

 * A deviation from the spec (which we have done in the past where the spec isn't clear or sensible but certainly try to limit)

 * A breaking change that affects all clients

 * Not really something we are asked about often

I'm not sure what leads you to believe that AMQP 0-9-1 protocol method fields do not have types defined in the spec.

They do. basic.publish's routing_key field is a short string:

http://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.publish.routing-key

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tom Molesworth]

Hi Peter,

Have a look at the exchange name and routing key definitions in https://www.rabbitmq.com/resources/specs/amqp0-9-1.xml, shortstr is used quite heavily there.

The PDF document amqp-xml-doc0-9-1.pdf might be more readable: see section 1.7.2.3. Method queue.bind for example:

routing­key shortstr - message routing key

Perhaps you're only looking at the amqp0-9-1.pdf documentation here?

cheers,

Tom

To post to this group, send email to rabbitm...@googlegroups.com.

[Peter Silva]

for:

http://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.publish.routing-key

That reference link in HTML is "a complete reference to RabbitMQ's implementaton"
I have no doubt about how rabbitmq is implemented.   That isn't a protocol specification.

As linked from the original post, I was referring to: https://www.rabbitmq.com/protocol.html
where one of the options is a PDF document entitled:  "AMQP Protocol Specification" that does not mention any addenda whatsoever.
yes that's the document I was looking at.  your reference to 1.7.2.2 was key...

It turns out there are two pdf documents,  One document named 'amqp0-9-1.pdf which skips from 1.4.x to 2, and appears to be the one on the web site.  In the link provided, there is amqp-xml-doc0-9-1.pdf are many more sections with the juicy details are.  

How is one supposed to know that one version is, ahem, abridged?

[Michael Klishin]

Peter, please. You seriously think that there can be frivolous spec-looking document on rabbitmq.com that a RabbitMQ team engineer

would link to just to prove his/her argument?

That "quick reference" document is a brief HTML version produced by the RabbitMQ team

from the PDF and XML documents that date back to the most recent revision of AMQP 0-9-1. We had to produce

it because linking to sections in PDF documents on the Web is not very portable at best.

There is also http://www.rabbitmq.com/amqp-0-9-1-errata.html, which is NOT in the PDF you think is the only

source of truth yet it provides critically important corrections that were discovered by a lot of head banging from

the people who implemented AMQP 0-9-1 and a bunch of other protocols.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tom Molesworth]

They are separate and complementary documents. One of the nice features of the AMQP spec is that the classes and methods are provided in XML format, which allows autogeneration of library code. That's one reason why there are two documents: one provides details for each class and method, autogenerated from that XML, and the other provides a high-level overview and explains terms and concepts.

There's no abridged version - the section numbers in one document have no relation to the section numbers in the other.

This is just the way that the AMQP group provide the documentation - it's quite common to have multiple documents for a spec, but if you can think of ways to improve how this is presented on the RabbitMQ site I'm sure they would be considered.

Note that both documents are already linked from https://www.rabbitmq.com/protocol.html - that's the "specification" and "generated doc" links respectively.

cheers,

Tom

[Richard Crawshaw]

I've recently been burned by the 255 character limit on the routing key too. We are routing messages based on the .NET Type being sent, so the obvious thing for us to do was to set each routing key to the assembly qualified name. In initial testing the type names were quite short, no nested types, short namespaces, minimal generics. However when I came to test in a production like environment it was a different matter.

I double checked the RabbitMQ documentation and it told a different story to the exception that was being thrown. The documentation shown here for QueueBind

https://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v3.6.10/rabbitmq-dotnet-client-3.6.10-client-htmldoc/html/type-RabbitMQ.Client.IModel.html#method-M:RabbitMQ.Client.IModel.QueueBind(System.String,System.String,System.String)

makes no reference to any hard limit on the length of the RoutingKey (or the queue name or the exchange name: I still haven't tested these with names longer than 255 characters). However the exception message says: "Short string too long; UTF-8 encoded length=269, max=255". This is the only clue that I have seen that there is an actual hard limit, until I came explicitly searching for it. So what else hasn't been included that isn't obvious? This makes me nervous.

It would be most helpful if the RabbitMQ documentation said "string max 255 characters" or even "string 255", instead of just "string".

I shouldn't find myself some time down the track with a product that breaks randomly because some things are just over the 255 limit and others just under simply because I wasn't alerted to this hard limit in the documentation. Nor should I have to pour over the details of the AMQP specification and tie each of its restrictions back to a RabbitMQ implementation detail. I expect RabbitMQ to do that with their documentation; after all they are the experts in that field, not me.

Regards

Richard

[Michael Klishin]

We will add a note to queue.bind, however,

improving documentation is rarely as trivial as Richard makes it sound.

Every client library has several functions that involve routing keys (usually at least 5). Various guides 

mention routing keys as well. Should every single instance mention the limit? Or should they link to a spec guide that covers domains?

The former is a complete overkill as there are over 20 libraries. The latter means 

that consulting the spec is necessary (as it is, at some point, with any protocol used directly, including HTTP).

What would help the most in my opinion are a guide on bindings (currently they are covered in many places) and field name included into the error message.

Richard, what doc guide "told you a different story"

if I may ask?

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

We started with https://github.com/rabbitmq/rabbitmq-dotnet-client/commit/6759a28fb6961f420e7b519aa8e7c4eabd743d16.

For those finding this thread in 2017 or later, keep in mind that .NET client is now decoupled from

server releases, so chances are there will only be 4.x and 5.x releases going forward. Therefore 3.6.x API reference

docs won't change.

On Thu, Jun 22, 2017 at 4:40 PM, Michael Klishin <mkli...@pivotal.io> wrote:

We will add a note to queue.bind, however,

improving documentation is rarely as trivial as Richard makes it sound.

Every client library has several functions that involve routing keys (usually at least 5). Various guides 

mention routing keys as well. Should every single instance mention the limit? Or should they link to a spec guide that covers domains?

The former is a complete overkill as there are over 20 libraries. The latter means 

that consulting the spec is necessary (as it is, at some point, with any protocol used directly, including HTTP).

What would help the most in my opinion are a guide on bindings (currently they are covered in many places) and field name included into the error message.

Richard, what doc guide "told you a different story"

if I may ask?

On Thu, 22 Jun 2017 at 07:33, Richard Crawshaw <richard.crawshaw@countrynet.net.au> wrote:

I've recently been burned by the 255 character limit on the routing key too. We are routing messages based on the .NET Type being sent, so the obvious thing for us to do was to set each routing key to the assembly qualified name. In initial testing the type names were quite short, no nested types, short namespaces, minimal generics. However when I came to test in a production like environment it was a different matter.

I double checked the RabbitMQ documentation and it told a different story to the exception that was being thrown. The documentation shown here for QueueBind

https://www.rabbitmq.com/releases/rabbitmq-dotnet-client/v3.6.10/rabbitmq-dotnet-client-3.6.10-client-htmldoc/html/type-RabbitMQ.Client.IModel.html#method-M:RabbitMQ.Client.IModel.QueueBind(System.String,System.String,System.String)

makes no reference to any hard limit on the length of the RoutingKey (or the queue name or the exchange name: I still haven't tested these with names longer than 255 characters). However the exception message says: "Short string too long; UTF-8 encoded length=269, max=255". This is the only clue that I have seen that there is an actual hard limit, until I came explicitly searching for it. So what else hasn't been included that isn't obvious? This makes me nervous.

It would be most helpful if the RabbitMQ documentation said "string max 255 characters" or even "string 255", instead of just "string".

I shouldn't find myself some time down the track with a product that breaks randomly because some things are just over the 255 limit and others just under simply because I wasn't alerted to this hard limit in the documentation. Nor should I have to pour over the details of the AMQP specification and tie each of its restrictions back to a RabbitMQ implementation detail. I expect RabbitMQ to do that with their documentation; after all they are the experts in that field, not me.

Regards

Richard

On Saturday, September 17, 2016 at 10:33:00 AM UTC+10, tom wrote:

They are separate and complementary documents. One of the nice features of the AMQP spec is that the classes and methods are provided in XML format, which allows autogeneration of library code. That's one reason why there are two documents: one provides details for each class and method, autogenerated from that XML, and the other provides a high-level overview and explains terms and concepts.

There's no abridged version - the section numbers in one document have no relation to the section numbers in the other.

This is just the way that the AMQP group provide the documentation - it's quite common to have multiple documents for a spec, but if you can think of ways to improve how this is presented on the RabbitMQ site I'm sure they would be considered.

Note that both documents are already linked from https://www.rabbitmq.com/protocol.html - that's the "specification" and "generated doc" links respectively.

cheers,

Tom

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Staff Software Engineer, Pivotal/RabbitMQ