IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

[Subba Rao]

Hi All,

We have recently started using RabbitMQ in our project and as part of our testing, we are sometimes getting the following exception. Has anyone experienced this issue in the past? if so, can you please let me know what could be the root cause and potential solution?

IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

Thanks,

Bala

[Subba Rao]

I have observed one more thing, If I re-process the failed record, it is working fine without any framework size error. I am not sure if the frame size is accumlated across all messages that were bering processed at that time.

Can someone throw light on what this frame size is? will this value gets reset after processing every message from the queue?

Thanks,

Bala

[Gavin M. Roy]

 I think knowing your client library and platform are important here.

Here's a preview link for Chapter 2 of my book that goes into frames and frame size in detail:

https://livebook.manning.com/book/rabbitmq-in-depth/chapter-2/

Sorry for the link like that but it's easier for me than typing it up in an email.

Regards,

Gavin

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e0d66699-8966-45ce-9b05-285c80c3daeeo%40googlegroups.com.

[Tessa Plum]

Subba Rao wrote:
> I have observed one more thing, If I re-process the failed record, it is
> working fine without any framework size error. I am not sure if the
> frame size is accumlated across all messages that were bering processed
> at that time.

Can you tell what server and client library version you were using?

Regards.

--
Tessa Plum
https://plum.ovh/

[Arnaud Cogoluègnes]

An outbound message seems to have headers that exceed the maximum frame size the client and server negotiated. You can increase the maximum frame size on the server side [1] and with the ConnectionFactory#setRequestedFrameMax(int) method on the client side (the default is 131072).

It would suggest though to look into the messages that trigger this exception because headers of about 1 MB can be considered excessive.

Note the maximum frame size does not limit the maximum size of a message payload, as it can be transmitted in several frames.

[1] https://github.com/rabbitmq/rabbitmq-server/blob/2b542351573a007740c9863f6a6359ca4dc344d8/docs/rabbitmq.conf.example#L271

________________________________________
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Subba Rao <subbu...@gmail.com>
Sent: Friday, July 24, 2020 6:56 PM
To: rabbitmq-users
Subject: [rabbitmq-users] IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com<mailto:rabbitmq-user...@googlegroups.com>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f5c940aa-38c5-4d82-ab36-4a6a00263a5bo%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Ff5c940aa-38c5-4d82-ab36-4a6a00263a5bo%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=02%7C01%7Cacogoluegnes%40vmware.com%7Cecbfa92adc75459609ab08d82ff281cd%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637312065905112189&sdata=DZ8JzbhLKrZASpdMKnMqz6AZP5I1g8TVm6pty72FiMM%3D&reserved=0>.

[Arnaud Cogoluègnes]

> I have observed one more thing, If I re-process the failed record, it is working fine without any framework size error. I am not sure if the frame size is accumlated across all messages that were bering processed at that time.

Are you sure there's no "accumulation" of properties/headers in your code? That is a AMQP.BasicProperties.Builder instance that is re-used across different outbound messages and could accumulate data that would make the properties of a message too big?

The frame size is not accumulated across all messages.

> Can someone throw light on what this frame size is? will this value gets reset after processing every message from the queue?

An outbound message is sent in several frames (figure 2.4 from Gavin's book shows it very well). The content header frame is the problem in your case, it is too big because of the properties, this is why I'm suggesting some "leak" in the properties building, but this is just guessing, some code would be helpful.

[Subba Rao]

Hi Arnaud,

Thanks for the response. I have a followup question.

this header limit is per message in the rabbitmq queue and sum of all the message headers in the queue at a given point of time.

For instance, I have a message1 with header size 100kb and message2 with header size 100kb in the queue at the same time. Now, in this case can we expect that both messages will be processed without any issue as each header size is less than 128kb? or can we expect that rabbitmq will sum up both the header sizes (100 + 100 kb) = 200 KB which is greater than 128 kb limit and both messages will be logged as an exception.

The reason why I am asking is, we have observed that when I re-process the same record (which is failed with frame size error in the first place) it is getting processed successfully without any error and no difference in the headers. why it failed in the first place and worked fine next time? that's why I am asking above question?

Thanks,

Bala

On Monday, July 27, 2020 at 12:57:56 AM UTC-6, Arnaud Cogoluègnes wrote:

An outbound message seems to have headers that exceed the maximum frame size the client and server negotiated. You can increase the maximum frame size on the server side [1] and with the ConnectionFactory#setRequestedFrameMax(int) method on the client side (the default is 131072).

It would suggest though to look into the messages that trigger this exception because headers of about 1 MB can be considered excessive.

Note the maximum frame size does not limit the maximum size of a message payload, as it can be transmitted in several frames.

[1] https://github.com/rabbitmq/rabbitmq-server/blob/2b542351573a007740c9863f6a6359ca4dc344d8/docs/rabbitmq.conf.example#L271

________________________________________
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Subba Rao <subbu...@gmail.com>
Sent: Friday, July 24, 2020 6:56 PM
To: rabbitmq-users
Subject: [rabbitmq-users] IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

Hi All,

We have recently started using RabbitMQ in our project and as part of our testing, we are sometimes getting the following exception. Has anyone experienced this issue in the past? if so, can you please let me know what could be the root cause and potential solution?

IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

Thanks,
Bala

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com<mailto:rabbitmq-users+unsubscribe@googlegroups.com>.

[Arnaud Cogoluègnes]

There's no header limit per se, there is a limit on the size of frames, and this is not related to the way messages are enqueued on the server side. The client library throws the exception you are seeing, because it is enforcing this frame limit.

About your question, yes, if the headers of 2 messages are both under 128 kb, there should be no problem.

I understand a failed message goes fine after a retry, so this may be something wrong in your code or in the client library. Sample code to reproduce the error would be helpful, as well as the version of the broker, client version, Erlang, and Java. Keep in mind also that a channel should not be used by multiple threads at the same time.

________________________________________
From: rabbitm...@googlegroups.com <rabbitm...@googlegroups.com> on behalf of Subba Rao <subbu...@gmail.com>

Sent: Monday, July 27, 2020 7:35 PM
To: rabbitmq-users
Subject: Re: [rabbitmq-users] IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

Hi Arnaud,

Thanks for the response. I have a followup question.

this header limit is per message in the rabbitmq queue and sum of all the message headers in the queue at a given point of time.

For instance, I have a message1 with header size 100kb and message2 with header size 100kb in the queue at the same time. Now, in this case can we expect that both messages will be processed without any issue as each header size is less than 128kb? or can we expect that rabbitmq will sum up both the header sizes (100 + 100 kb) = 200 KB which is greater than 128 kb limit and both messages will be logged as an exception.

The reason why I am asking is, we have observed that when I re-process the same record (which is failed with frame size error in the first place) it is getting processed successfully without any error and no difference in the headers. why it failed in the first place and worked fine next time? that's why I am asking above question?

Thanks,
Bala

On Monday, July 27, 2020 at 12:57:56 AM UTC-6, Arnaud Cogoluègnes wrote:
An outbound message seems to have headers that exceed the maximum frame size the client and server negotiated. You can increase the maximum frame size on the server side [1] and with the ConnectionFactory#setRequestedFrameMax(int) method on the client side (the default is 131072).

It would suggest though to look into the messages that trigger this exception because headers of about 1 MB can be considered excessive.

Note the maximum frame size does not limit the maximum size of a message payload, as it can be transmitted in several frames.

[1] https://github.com/rabbitmq/rabbitmq-server/blob/2b542351573a007740c9863f6a6359ca4dc344d8/docs/rabbitmq.conf.example#L271<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frabbitmq%2Frabbitmq-server%2Fblob%2F2b542351573a007740c9863f6a6359ca4dc344d8%2Fdocs%2Frabbitmq.conf.example%23L271&data=02%7C01%7Cacogoluegnes%40vmware.com%7C08adb26f5a404e54528708d83253671c%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637314681078591656&sdata=zAxxXjg4Q5kykZ6LyWA2eTmbJFquPIodoAmod%2Bc38m0%3D&reserved=0>

________________________________________
From: rabbitm...@googlegroups.com<javascript:> <rabbitm...@googlegroups.com<javascript:>> on behalf of Subba Rao <subbu...@gmail.com<javascript:>>

Sent: Friday, July 24, 2020 6:56 PM
To: rabbitmq-users
Subject: [rabbitmq-users] IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

Hi All,

We have recently started using RabbitMQ in our project and as part of our testing, we are sometimes getting the following exception. Has anyone experienced this issue in the past? if so, can you please let me know what could be the root cause and potential solution?

IllegalArgumentException: Content headers exceeded max frame size: 1057821 > 131072

Thanks,
Bala

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com<javascript:><mailto:rabbitmq-user...@googlegroups.com<javascript:>>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f5c940aa-38c5-4d82-ab36-4a6a00263a5bo%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Ff5c940aa-38c5-4d82-ab36-4a6a00263a5bo%2540googlegroups.com&data=02%7C01%7Cacogoluegnes%40vmware.com%7C08adb26f5a404e54528708d83253671c%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637314681078591656&sdata=uLc1IY%2BLbJueeLdDb9g8f0RemZysL62SV3818gcbWEI%3D&reserved=0><https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Ff5c940aa-38c5-4d82-ab36-4a6a00263a5bo%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=02%7C01%7Cacogoluegnes%40vmware.com%7Cecbfa92adc75459609ab08d82ff281cd%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637312065905112189&sdata=DZ8JzbhLKrZASpdMKnMqz6AZP5I1g8TVm6pty72FiMM%3D&reserved=0<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2Ff5c940aa-38c5-4d82-ab36-4a6a00263a5bo%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=02%7C01%7Cacogoluegnes%40vmware.com%7C08adb26f5a404e54528708d83253671c%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637314681078591656&sdata=Jrjl%2BvtIK8hb6Ai5AJCUz22hen6klfN%2FcTV%2BEypxQeg%3D&reserved=0>>.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com<mailto:rabbitmq-user...@googlegroups.com>.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/20313a58-15a4-4cc4-a935-9c32cf507444o%40googlegroups.com<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Frabbitmq-users%2F20313a58-15a4-4cc4-a935-9c32cf507444o%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=02%7C01%7Cacogoluegnes%40vmware.com%7C08adb26f5a404e54528708d83253671c%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637314681078601654&sdata=%2F%2FDfOhuhZnbfwCCcAJmBtP1fnd4vP2nhmjUW%2FcvgSDY%3D&reserved=0>.