RabbitMQ bridge using Flutter

[Mark Jenkins]

Hi All,

May be of general interest to the group generally and happy to share the code as another 'bridge' example (with apologies in advance for my coding style!)

I've been recently working on my own message bridge between STOMP and RabbitMQ but written in Flutter/Dart.  Why Flutter? My logic here is it allows you to build to a native desktop app that will run on Windows, Linux and MacOS - so in theory multiplatform with native performance. Also runs on Web clients and iOS/Android but who wants a bridge server running on their mobile 😆👀.

Design

I've written my own STOMP connector class and the solution uses MQTT to connect Flutter to RabbitMQ as that library is more mature than AMQP in the Dart ecosystem.  Its intended to run as single publisher, single consumer - a pipe basically.

  

The Rabbit queue is set up as 'classic' and durable.  RabbitMQ and MQTT Plugin configurations are all 'out of the box', the only change being that Rabbit has been opened up from 127.0.0.1 to allow connections from my development Mac and that I've added a queue bound to the default exchange for MQTT which is 'amq.topic'.  A routing key is used to direct traffic to the queue (and this is the 'topic' given to MQTT in Flutter).

I plan that a RabbitMQ consumer will eventually persist the messages to a MySQL db to sit with the snapshot, timetable and ref data I've already collected via Apache/PHP/MySQL.

Progress

I have the server running on MacOS successfully consuming /topic/darwin.pushport-v16 and delivering to a RabbitMQ queue hosted on a 2nd MacOS machine.   With some caveats however!

Caveats

• Doesn't process ERROR frames at the moment
  
• Doesn't handle heartbeats
• UI controls (buttons) rather than full automation at this point
• While I'm getting a high data rate and I've got no gaps in message numbers when consuming from the STOMP side, I'm getting around 10-15% of messages being dropped on the RabbitMQ side ☹️ I've not yet pinned down the MQTT library or the Rabbit configuration.  

It'd be really appreciated if anyone has similar experience with dropouts on RabbitMQ could assist and if there are any common settings/gotchas with RabbitMQ that I should look out for...

Many thanks in advance!

Mark

  

[Mark Jenkins]

Edit:  Fixed the issue 😃 which was down to my inability to count rather than the messaging code...  so we have a working Flutter bridge 👍

[petermount]

I've been using Rabbit since day one & not had any issues with rabbit dropping frames other than when I had /var filling up & hadn't noticed so Rabbit throttled itself until space was freed.

That said I've been using AMQP & not MQTT - the latter is designed for IoT applications so I don't know how it handles error conditions or performs with heavy loads.

[Mark Jenkins]

Thanks Peter.  

Turns out Rabbit was behaving, it was me not recc'ing sent and received message counts properly, schoolboy error.  I also tweaked MQTT's publishing function from 'at least once' to 'exactly once'.  Managed to send 30k messages through without a drop out or slow down, so it's working well now.  Throughput end-end is around 6 to 9 messages per second (non ack-ed) which equates to 10k messages in about 20 mins.  Without a baseline I don't know if that's 'good'.  I've seen it peak at 33/s which suggests the bottleneck is messages arriving at the the STOMP listener rather than MQTT to RabbitMQ, guessing it matters the time of day that its run.

Generally MQTT appears reliable and logging is good so its easy to track activity, but early doors yet in terms of testing.  If of interest this link gives a quick summary of the split of responsibilities between the protocol and the app in terms of handling resilience: 

https://www.hivemq.com/blog/are-your-mqtt-applications-resilient-enough/

Eg: One of the features mentioned is MQTT not supporting a very high message rate and I've used an internal stream within my Flutter app do decouple receive/send so I can throttle the MQTT send rates if I start observing issues (not an issue so far).

Cheers

Mark