Error when retrying message due to frame_too_large error from RabbitMQ

[Espen Wang Andreassen]

Hi,

NServiceBus (6.0.0) with NServiceBus.Rabbit (4.0.0) against RabbitMQ 3.6.1.

Would it be possible to enable configuration of the frame-size the rabbitmq-client requests when connecting to the server?

Sometimes nservicebus is unable to put messages back to rabbit for retry due to what I suspect is a too large header. This typically happens if we have a problem with sagadata-mapping, where the saga is unable to persist data, or an incoming event causes the nhibernate-request to fail. This is of course not a normal type of error, but it is still important to us that we get those messages back on the retry/error queue.

If I understand correctly, rabbit will split the body content into several frames, but this is not possible for the header/properties-portion of the message, and thus it might get to large for the default frame size.

The error is as follows:

Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=501, text="FRAME_ERROR - type 2, all octets = <<>>: {frame_too_large,146959,131064}", classId=0, methodId=0, cause=

Stack trace:

at RabbitMQ.Client.Framing.Impl.Connection.EnsureIsOpen() at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.CreateModel() at NServiceBus.Transport.RabbitMQ.ConfirmsAwareChannel..ctor(IConnection connection, IRoutingTopology routingTopology, Boolean usePublisherConfirms) in C:\Build\src\NServiceBus.RabbitMQ\Connection\ConfirmsAwareChannel.cs:line 14 at NServiceBus.Transport.RabbitMQ.ChannelProvider.GetPublishChannel() in C:\Build\src\NServiceBus.RabbitMQ\Connection\ChannelProvider.cs:line 27 at NServiceBus.Transport.RabbitMQ.MessageDispatcher.Dispatch(TransportOperations outgoingMessages, TransportTransaction transaction, ContextBag context) in C:\Build\src\NServiceBus.RabbitMQ\Sending\MessageDispatcher.cs:line 18 at NServiceBus.DelayedRetryExecutor.<Retry>d__1.MoveNext() in C:\Build\src\NServiceBus.Core\Recoverability\DelayedRetryExecutor.cs:line 51 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.RecoverabilityExecutor.<DeferMessage>d__4.MoveNext() in C:\Build\src\NServiceBus.Core\Recoverability\RecoverabilityExecutor.cs:line 97 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at NServiceBus.Transport.RabbitMQ.MessagePump.<Process>d__29.MoveNext() in C:\Build\src\NServiceBus.RabbitMQ\Receiving\MessagePump.cs:line 229 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NServiceBus.Transport.RabbitMQ.MessagePump.<Consumer_Received>d__28.MoveNext() in C:\Build\src\NServiceBus.RabbitMQ\Receiving\MessagePump.cs:line 168

Any ideas? I didn't find a way to set the frame_max the client requests during configuration of the transport.

Now, it might very well be that increasing the frame size is the wrong solution, and any other suggestion is of course welcome.

Best regards,

Espen Wang Andreassen

(Norwegian Broadcasting Corporation)

[Brandon Ording]

Hi Espen,

Your analysis sounds correct to me. Certain messages in your system are ending up with headers that exceed the default frame_max size of 131072 bytes. While RabbitMQ can split up content body frames that are too large, the other frame types cannot be split. BTW, in case you're wondering why the error message says 131064 and not 131072, it's subtracting the bytes required for the frame header and frame_end sequence that all AMQP frames require.

You're correct that we don't expose a way to set the client's frame_max for the transport. The reason for this is that the client can only ever negotiate a smaller frame_max size than what the server first offers. By keeping it at the default value, the client will always choose whatever size the server first offers.

The good news is that you can adjust frame_max on your RabbitMQ broker, and the client will start using that new size without any additional changes.

For example, if you wanted to double the frame size to 256 KB, you should add the following to your rabbitmq.config file:

{frame_max, 262144}

Once your broker picks up the new config changes, which requires it to be restarted, you should be able to handle larger frame sizes without an error.

Hope this helps,

Brandon