Max message size limit && message removal notification
25 views
Skip to first unread message
Viktor Klyestov
Sep 5, 2022, 9:55:37 AM9/5/22
to rabbitmq-users
Hello,
We'd like to decouple our system, so that it is easy to connect new data consumers. For that reason we want to integrate some sort of message broker like RabbitMQ. The problem is that we have fairly huge data chunks (up to several GB) with low ingestion rate (one chunk per 5-10 seconds). We can not split them. To overcome message size limitation we are considering to store message body as a file on hard drive and publish metadata (file_guid, client_id, date etc). In order to reduce required disk space we need broker to notify when message was removed / consumed by all consumers and as far as I know there is no such feature. Moreover, it is not ideal to read the same body from hard drive multiple times. Can somebody suggest anything? I've never used RabbitMq, maybe it is not the way to go here...
We'd like to decouple our system, so that it is easy to connect new data consumers. For that reason we want to integrate some sort of message broker like RabbitMQ. The problem is that we have fairly huge data chunks (up to several GB) with low ingestion rate (one chunk per 5-10 seconds). We can not split them. To overcome message size limitation we are considering to store message body as a file on hard drive and publish metadata (file_guid, client_id, date etc). In order to reduce required disk space we need broker to notify when message was removed / consumed by all consumers and as far as I know there is no such feature. Moreover, it is not ideal to read the same body from hard drive multiple times. Can somebody suggest anything? I've never used RabbitMq, maybe it is not the way to go here...
Reply all
Reply to author
Forward
0 new messages