Re: [mqtt] Re: Remote bridge/gateway for intermittent connections

[Nicholas O'Leary]

Hi Frank,

"WebSphere MQ Telemetry daemon for devices" is the productised version of RSMB - the small broker implementation that is suitable for running on the type of arm/linux board you talk about.

Mosquitto is the open-source equivalent to RSMB.

The sort of scenario you describe is quite typical for MQTT; using a broker at the 'edge' for passing messages around locally, with a subset of them being passed back to a central point.

With both RSMB and Mosquitto (their config files are almost interchangable), you can define a bridge to a remote broker. If set up correctly, they will queue up messages to be sent over the bridge whilst it is not connected.

So, you are certainly on the right track and many of the core pieces you want already exist.

Regards,

Nick

On 30 May 2013 14:29, Frank P <fpagl...@mindspring.com> wrote:

Hello All,

Sorry, my original post was pretty vague and didn't ask any specific questions. To reiterate, I'm interested in a class of remote data loggers that are sophisticated enough to use an MQTT broker internally to pass raw data between the sensors and actuators, but that a subset of messages (processed data and alerts) would want to make it back to central server on the Internet. The device would need to queue messages, until some point at which it would make a phone call (cellular, satellite, or even POTS), connect via PPP, transfer all the accumulated messages en masse, and then hang up and repeat. On the opposite side, the server may also be accumulating commands to the device which it sends as soon as the device calls in and connects.

After reading up on MQTT, it seems that I am looking for something described in the IBM literature as the "WebSphere MQ Telemetry daemon for devices".

So allow me to ask: does something like the "daemon for devices" exist in the open-source world? Is anyone working on something similar?

If not, perhaps it's time to start typing. Can can anyone point me to some specifications for this type of application? (Alert me to pitfalls or useful features).

Would this be of general interest?

Thanks,
Frank

--
--
To learn more about MQTT please visit http://mqtt.org
 
To post to this group, send email to mq...@googlegroups.com
To unsubscribe from this group, send email to
mqtt+uns...@googlegroups.com
 
For more options, visit this group at
http://groups.google.com/group/mqtt
 
---
You received this message because you are subscribed to the Google Groups "MQ Telemetry Transport" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Hiram Chirino]

You might want to try Apache ActiveMQ [1] along with Apache Camel [2].
ActiveMQ can handle receiving your topic based MQTT traffic. You'd
create your traffic filtering/aggregation logic /w Apache Camel and
forward the results to an ActiveMQ queue which can be periodically
drained to a central location.


[1]: http://activemq.apache.org/
[2]: http://camel.apache.org/

--
Hiram Chirino

Engineering | Red Hat, Inc.

hchi...@redhat.com | fusesource.com | redhat.com

skype: hiramchirino | twitter: @hiramchirino

blog: Hiram Chirino's Bit Mojo

[Karl Palsson]

Or, you know, don't. [1]

bog standard mqtt bridges do this with qos 1. We're doing _exactly_ what you're talking
about right now. Multiple remote constrained devices run a local MQTT broker (mosquitto)
for local applications, and there's a bridge configured to a "central" MQTT broker. Local
message producers publish with qos1, so they are durably queued properly when the bridge
connection is down. (mosquitto also supports queueing qos0 messages, but that's not MQTT
standard) All you need for this is to make sure your bridge connection id (aka client id)
is stable. You'll likely also want to tweak things like, "how many messages are allowed to
be queued"

With that, you're working out of the box. Pull the network cord? No problems. Power off one
end? No problems. ifdown the expensive gsm link and ifup later? No problems. The local
broker just bridges it's queued messages out whenever it can get a connection the configured
central server. No connection? Messages are queued.

Cheers,
Karl P


[1] Don't get me wrong, camel's cool, I've used it to great effect, and it changed the way I
worked with big blobs of java, permanently and for the better. I just don't think it's at
all what's being asked for here.

[Hiram Chirino]

Camel came into the picture since it seems like he wanted some
filtering and aggregation. The brokers don't usually do that out of
the box.
Then again if the device is too constrained, Java might not be in the picture.

[Frank P]

Thanks for the help.

Currently there is no Java on the embedded devices, though even mid-level boards these days could probably handle it. But an enterprise solution seems rather heavyweight, and you always have to be careful putting server software on an embedded device in terms of additional resource usage.  An app can degrade a Flash chip or SD Card rapidly if it does a lot of disk I/O.

So Mosquitto definitely seems the way to go. But Camel seems an interesting possibility on the server side.

But, with Mosquitto, when the connection is down, does the broker just poll to see if it's reestablished? Can it be given some guidance, such as a trigger from a PPP Up action/script ("quick, go now")?

And when the Internet connection is re-established, do the brokers have to perform an MQTT connect conversation all over again, or do they keep a "logical" connection established above the network anomalies?

[Stefano Costa]

On 05/31/2013 03:15 PM, Frank P wrote:
> Currently there is no Java on the embedded devices, though even
> mid-level boards these days could probably handle it.

...apart from products based on GPRS modules from Cinterion (EGS5) that
can run Java firmware, on a standard JVM. We're currently porting Paho
to our EGS5-based tracking devices. I agree that (even here) Java on
embedded devices is not for realtime or I/O intensive applications!

S

--
Stefano Costa, R&D Manager
M +39 335 6565749
Skype stefanocosta.bluewind
Twitter @stefanobluewind
http://www.bluewind.it

[Roger Light]

Hi Frank,

> But, with Mosquitto, when the connection is down, does the broker just poll
> to see if it's reestablished? Can it be given some guidance, such as a
> trigger from a PPP Up action/script ("quick, go now")?

You can configure bridges to restart automatically, which would poll
until re-established, or to be "lazy". A lazy bridge will drop its
connection after a certain amount of time with no communication and
then only attempt to reconnect once a certain number of messages are
queued. There are bridges that will connect once and never again, but
these are of limited use at the moment.

It isn't currently possible to control bridges externally, or to
reconfigure them without restarting the broker. This is one of the few
remaining areas where rsmb provides more functionality than mosquitto.

> And when the Internet connection is re-established, do the brokers have to
> perform an MQTT connect conversation all over again

This is correct.

Cheers,

Roger

[Alexandros Kritikos]

Frank / all,

just a reminder though, bridging is not part of the MQTT specification and as a result its not something that will be standardised through OASIS. This means that it might not work exactly the same in all brokers and some compliant brokers might not offer it at all. I do believe its a useful feature that should be standardised but this will likely happening after the core spec is approved.

Thanks,

Alex

[Roger Light]

Hi Frank,

You could also create your own client to do bridging. That way it can
behave exactly as you want and isn't subject to broker specific
implementation of bridging as Alex suggests.

Cheers,

Roger

[Frank P]

Hey Roger,

You could also create your own client to do bridging. That way it can
behave exactly as you want and isn't subject to broker specific
implementation of bridging as Alex suggests.

Yes, that was my original assumption, though I am happily surprised that Mosquitto does a lot more than I had thought on first look. So now I'm trying to figure out if my requirement is very specific, or if there's a more general need for something at the mid-level of embedded devices. I can foresee a growing class of devices, like the ones I often deploy, that are relatively sophisticated (Linux/Android based) but low (battery) powered. In that world, a relatively sophisticated broker like Mosquitto is warranted locally, but the connection to the world is expensive in terms of battery power and connection costs. So in this case the bridge would not be trying to overcome occasional, accidental disconnects, but rather make optimal use of short, periodic connections.

My exploration continues.

[Andy Piper]

On Fri, May 31, 2013 at 2:47 PM, Alexandros Kritikos
<alex.k...@my-channels.com> wrote:
> just a reminder though, bridging is not part of the MQTT specification and as a result its not something that will be standardised through OASIS. This means that it might not work exactly the same in all brokers and some compliant brokers might not offer it at all. I do believe its a useful feature that should be standardised but this will likely happening after the core spec is approved.

Agreed, and this is noted on the mqtt.org wiki - it definitely would
be useful for more of the broker implementations to support bridging
though!


--
Andy Piper | Farnborough, Hampshire (UK)
blog: http://andypiper.co.uk | skype: andypiperuk
twitter: @andypiper | images: http://www.flickr.com/photos/andypiper

[Andy Stanford-Clark]

One of the cool features of the RSMB broker is the ability to manually stop and start bridge connections. One of the use cases this feature was designed for, was exactly what you're talking about... at some point in time you decide to bring up a comms connection, then the script that does that triggers RSMB to start the required bridge connection to a remote broker. So the time interval of polling for the auto-reconnect does not reduce the utilisation of the link's uptime.

"someone should" add that to Mosquitto ;)

AndySC

[Andy Stanford-Clark]

Also worth mentioning that by careful design of topic, QoS and bridge connection configurations, you can do some cool stuff with delivery of prioritised messages over different links.

So for example you could arrange that if you only have a GSM connection, you only start a certain connection which releases (say) the most urgent messages. But if you have the luxury of a wifi connection, you can start the "batch" queue connection and bulk upload a mass of messages that were being stored up.

For extra points (with an additional subscriber app) you could even arrange for a publication to an "emergency" topic to kick the startup script for a satellite phone modem, bring up the emergency topic's bridge connection, and deliver just that message then disconnect.

From this you can infer some of the cool projects we've worked on over the past few years with MQTT :)

AndySC

[Frank P]

This is very interesting. Exactly the type of scenarios that I tend to target. After looking into this a bit further, I started wondering what might be a better solution... adding this type of functionality to an existing broker, or developing a flexible, filtering, persistent bridge (client) that could connect any two brokers over a variety of channels?  Incorporating this functionality into a broker would, no doubt, have better run-time efficiency, but a separate client might be more flexible.

Any thoughts?

[Raphael Cohn]

For me, I'd plump for a separate client. That's far easier. The actual efficiency trade offs aren't usually that bad for this sort of dedicated router. Most broker designs are predicated on lots of incoming connections - this client is going to be predicated on lots of outgoing connections.

Raphael Cohn
Chief Architect, stormmq

Co-Chair, OASIS MQTT Standard

Secretary, OASIS AMQP Standard
raphae...@stormmq.com

+44 7590 675 756

UK Office:
Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom
Telephone: +44 845 3712 567

Registered office:
16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom
StormMQ Limited is Registered in England and Wales under Company Number 07175657
StormMQ.com

On 5 June 2013 14:05, Frank P <fpagl...@mindspring.com> wrote:

This is very interesting. Exactly the type of scenarios that I tend to target. After looking into this a bit further, I started wondering what might be a better solution... adding this type of functionality to an existing broker, or developing a flexible, filtering, persistent bridge (client) that could connect any two brokers over a variety of channels?  Incorporating this functionality into a broker would, no doubt, have better run-time efficiency, but a separate client might be more flexible.

Any thoughts?