Multiple hosts

[Bob Frankston]

One reason I've been hesitant to go all-in on MQTT is that having a single broker is a point of failure. Is there any standard for client libraries to support multiple hosts for sending and receiving? Perhaps using GUIDs to avoid duplication?

[Simon Walters]

AFAIK - No

Simon

[ran...@bevywise.com]

Hello Bob, 

We have been running Broker on the cloud servers on  AWS , Google as well as Digital ocean for a long time for now. We are not seeing much of issues with respect to failures on the application side as well as the server side. 

In the similar fashion we do load balancing on Apache / Nginx for HTTP, we  will be able to do balancing for MQTT using ngnix and Brokers today can be set on Active - Passive as well as Active - Active modes.

I believe your objective is not to send data to two different region to mitigate catastrophic failure. 

Building a client to connect / send to a second server should be much simple. But could not get convinced with the need for today. 

Hope this helps. 

Ranjith 
Bevywise Networks

[Jon Levell]

Hi Bob.

If there is no persistent state for a client (i.e. they connect with CleanSession=True in MQTTv3 or SessionExpiry=0 in v5) then you can just have multiple brokers and either have a loadbalancer in front of the brokers or use a client that can connect to one out of a list of IPs (the high availability column in this list of clients hosted by Eclipse: https://www.eclipse.org/paho/index.php?page=downloads.php )

If there is persistent state, then you need servers that can understand and sync that state between them. (In the Eclipse Amlen broker - we call that "Highly Available Pairs") - once you have servers syncing the state for a client then (again) you can either use a loadbalancer or use a client that can connect to multiple different IPs.

(disclaimer: I'm the project lead for Eclipse Amlen)

Jon

Jon Levell

Project Lead for Eclipse Amlen
https://www.eclipse.org/amlen/

On Sat, Aug 27, 2022 at 2:45 PM Bob Frankston <bob.fr...@gmail.com> wrote:

One reason I've been hesitant to go all-in on MQTT is that having a single broker is a point of failure. Is there any standard for client libraries to support multiple hosts for sending and receiving? Perhaps using GUIDs to avoid duplication?

--
To learn more about MQTT see https://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/e58e5955-51f0-4159-aab6-51796aae3e12n%40googlegroups.com.

[Joseph Ottinger]

Wow, I'd never heard of Amlen - very cool. What are the runtime requirements for Amlen? What about distributing as a docker image, as mosquitto does? .. I should join the slack and bug the entire WG, shouldn't I? :)

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/CAE%3DyfU92p5LLks7nm%2Bj1WvGq66cHOFw898ufotEtr%3D9hxGncfw%40mail.gmail.com.

--

Joseph B. Ottinger
http://www.enigmastation.com

To the beautiful and the wise,

  The mirror always lies.

[Bob Frankston]

Jon, I guess it comes down to how I view MQTT and the concept of a persistent state. I approach this from the perspective of an Internet based on a strict separation of the relationships between the endpoints and the stuff between them (AKA, the networking facilities and services). This means that there can be a convention for clients and listeners (publishers and subscribers) to be broker agnostic and resilient. But for that to work, it needs to be a standard used in devices I don't own or control.

I understand the value of services in the path, like caching services on the Internet, but those are quite distinct from the core networking capabilities. That's clear because they were added later. MQTT seems to have both from the get-go so it. For MQTT it will take some effort to be explicit about the two views but it's worth thinking about since MQTT has become the de facto publish/subscribe intermediary beyond the SCADA design point.

[Bob Frankston]

Ranjith -- your example of AWS is interesting since it creates a strong point of failure in the connection between the local system and AWS and provides a strong reason for why I want multiple brokers. To use a simple example, if I use MQTT to turn on a light in my house, it shouldn't depend on an AWS connection. Hmm, load balancing is a good reason to be wary of any persistent state in a broker. 

[Jon Levell]

Hi Joseph,

You mention Mosquitto, compared to Mosquitto - the runtime requires for Amlen are *huge*: 4GiB memory and 5x that disk.

I have a branch that reduces them quite a bit but *nowhere* near Mosquitto, Amlen is usually used in deployments where there are thousands or millions of devices.

We current ship a dockerfile but no prebuilt image though we are working on a kubernetes operator and that will require an image so watch this space.

Jon

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/CAEhvUywONgoLiZVJQ3AOv5Ebe85N5%3DuYHTCgrD-xG6qB%2BLn8mQ%40mail.gmail.com.

[Jon Levell]

Hi Bob,

I understand your desire for clients to be broker agnostic. MQTT is a standard and is does define what the persistent state is:
https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html#_Toc3901230

We need persistent state for the qualities of service that avoid duplicate messages, the broker has to remember which messages are in flight so if the connection breaks and is re-establish and the messages resent the broker needs to know which messages it has already seen - we've standardised that so clients don't need to know what broker they are connecting to.

If you are just streaming messages and not checking whether they were sent correctly (QoS 0 in the MQTT terminology) then you don't need to think about how the brokers share the state if you have multiple brokers.

Jon.

P.S. are you this Bob? : https://rmf.vc/bob_frankston_bio

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/e9ae73b1-744f-42f2-b978-7056c0abfdd3n%40googlegroups.com.

[Bob Frankston]

We should probably split the persistent state discussion into a separate conversation.

This is a flavor of the intelligent network conversation. Services that manage state for me are useful, but even if use them, they have an impact in giving me a single point of dependency rather than multipath resilience. And such promises are illusory at best. If a broker dies, all those smarts are worse than nothing -- they are a lie. Having a separate protocol that might use MQTT as a transport could then ask if the packet was really delivered or not.

I understand in the days of relay logic one couldn't have a smart client but, today, we could splurge $2 to stick an ESP8266 on the side of the client.

[jonq...@gmail.com]

Hi Bob,

In this case the protocol managing state doesn't create a single point of dependency - the state is standardised and there are lots of brokers that understand the standard and can  sync the client state between multiple instances of the server (with the instances well separated in network terms).

Without the state as part of the standard, applications which wanted reliable message would have to invent the state and multi-stage acknowledgements themselves in non-standard ways.

Jon.

[Bob Frankston]

I don't want to get too deep into the protocol discussion here though, obviously, I have strong opinions. I do hope systems using MQTT can survive catastrophic failures.

For my use my concern is that when I use MQTT to connect to various devices, I can only specify a single broker rather than a family of them. Perhaps I can have multiple DNS instances with the same name, but that has other limits.HomeAssistant for one has trouble with using a DNS name rather than an IP address.

[ran...@bevywise.com]

Dear Bob, 

The connectivity issues is beyond the MQTT and it can happen to any protocol.   

If we are going to have two hosts with different domain for the server application, then the possibility of having a split brain i.e, subscriber connecting to one Broker  and publisher going to connect to a different broker will happen and if the connectivity between the two brokers is lost due to the same network issues, then it is going to be a problem together to identity / resolve quickly. 

I understand we cannot afford to loose connectivity for commercial  as well as for  business cases.   If our customers are too skeptical about their connectivity issues, we recommend to our Business customers having a local server and connect to the server via a public IP NATed to the server or via VPN , so that the server will be available for their local usage 100% 

Cheers 

Ranjith 

[Bob Frankston]

There are no guarantees. A local server can still go down. This is why protocols like TCP have been so resilient. They aren't dependent upon any particular intermediary. We can have a protocol to assure relationships between endpoints. It's also vital that we have real security, not just hop-by-hop encryption. If you care, then 100% of the packets must be encrypted with certificates. But for casual use I can use it locally with clear text. These are well-understood concepts - resilience is better than hardening.

For my own system, I have multiple intermediaries and dual home system. I want to go much further and enable resilience everywhere but that's a longer discussion about economics and system architecture which takes us well beyond MQTT.

[Philip Couling]

Bob 

I think you have oversimplified the existing technology outside of MQTT. In an earlier email you suggested the ideal was to have everything decomposed into clients, endpoints, and transparent transport.  That model is functionally incomplete and not correct for any scalable system.  

This is important because The MQTT broker fulfils multiple roles that also exist in real world REST (HTTP) systems.  These roles cannot (should never) be fulfilled by the client or the endpoint.  They are not transparent.

The missing piece could be named "orchestration".  In your lightbulb example

- you would never put your user authentication and  authorisation into the lightbulbs themselves.

- new light bulbs (fresh out the box) need to know if they should be on or off to be in sync with others in the same room.  The state of the room needs to be held somewhere.

- your client needs to be able to send instructions to all bulbs in a room, but knowledge about which bulbs exist  (which "endpoints") cannot live on the client. Otherwise every client would need to be updated every time a lightbulb was changed or added.

The message broker adopts aspects of the "orchestration" role which logically MUST exist in any scalable solution. It can only not exist on trivial "one client one device" systems.

No aspect of MQTT's design dictates the broker must be in the cloud. A perfectly legitimate solution would be to have one broker physically on site.

No aspect of MQTT dictates there should be only one broker. if a cloud solution is also planned, you could still have one on site and create a bridge from site to cloud (daisy chain).  This would let your client make calls directly to site if it had the right credentials.

Multiple clustered message brokers exist. If uptime is a concern then, by all means, set up a cluster and throw a chaos monkey at it to test whether it really behaves as well as the vendors say it does.

But in short the design of MQTT does not introduce anything particularly new, it just rearranges the grouping of how those roles are performed.

It certainly doesn't introduce a new point of failure.  MQTT is the only protocol I know with good reconnection semantics making reliability a major selling point of the protocol.

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/36fa93a5-50a7-435d-b5a2-e405997c5400n%40googlegroups.com.

[Bob Frankston]

We're revisiting old ground. My main point is that I don't see clients that support multiple brokers for resilience. I am talking about the first broker one is associated with. That's a problem for me -- not whether it's in a distant cloud or not because even locally, I want resilience. I know that I could implement it myself, but the devices reporting to me need to support it. If there is a standard for secondary points of entry (first broker), then please tell me and show me how to do it in standard clients other others' devices.

When I see "you would never put your user authentication and  authorisation into the lightbulbs themselves..." I realize we think very differently about system design, and I know from experience that we're not going to make much progress here, especially with so givens and musts. I will note that the issue of defining relationships among devices and devices being part of multiple meta-devices is a deep topic in its own right.

[ran...@bevywise.com]

Bob, 

The overall resilience should be built by putting everything together. 

How about solving the connectivity issue by  keeping two GSM or Two network ( ETH0 , WLAN0 ) on the device.  So that one of the network will help reach the server.   I believe the TCP also will have the same issues if the communication fails. 

I still believe that the server should be made resilient with load balancing with proper fail over system, irrespective of whether it is hosted in one region or multiple region or how the data is stored  by the service provider.  The service provider should keep the DNS handy to auto swap based on the need or failure.  This helps better than running two brokers. 

For your case, the instant message is mandate and need connection always.  However for the other cases, we can have the caching algorithm to avoid data loss due to communication issues. 

As you said, building such a system should not be a big effort, but may not be needed for most ( 99%) of the use cases. 

My 2 cents !! 

Ranjith  

[Philip Couling]

I don't generally see this supported.  The preferred approach is to deal with this serverside with either a mesh of TCP load balancers or load balanced DNS records [with health check].  The latter deals with geographically diverse brokers at the expense of responsiveness to downed. Both gracefully handles changes to the number of message broker instances without a need to reconfigure every client and every device.

The only time I've seen multi hostname clients was where there was some concern about DNS stability; the clients were configured with different domains in different TLDs from different registrars.  These were custom coded clients though.

Forgive my directness, it's not my intent to tell you how to design anything.  I only wish to convey industry norms.  I presume these to be informative because where you chose to diverge from the norm you often find "holes" in the feature sets you require. Where you chose to follow the norm, you usually find a lot of legwork has been done for you. Ultimately it's your choice though.

I'd be curious to hear of any real world IOT systems which do not have some from of orchestration separate from client and device. To date I know of none.  

Kind regards

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/41a80b5b-506f-47aa-8183-723b684cd6fcn%40googlegroups.com.

[Bob Frankston]

MQTT is useful because it's commonly implemented and enables massaging among independently designed endpoints. Support for multiple brokers by clients is a simple suggestion and would make the system more resilient and more widely useful, especially in consumer applications without a maintenance staff.

I understand that any system works very well for the current applications because they co-evolve, and it's often hard to imagine other use cases. But leveraging what exists is the way forward https://rmf.vc/IEEEBetaLife.

[Philip Couling]

I think the problems you perceive are already solved problems, just solved with a different solution to what you imagine.  For now I'm assuming that crossing administrative boundaries is off topic and you are exploring designs for closed networks of devices.  Otherwise that's another very deep rabbit hole.

Reading through your blog and taking hints from your "home" description I think you'd get much more value through exploring "zeroconf" solutions to this problem and similar (https://en.wikipedia.org/wiki/Zero-configuration_networking).  These are focussed very much on solutions that do not require an administrator.

• First consider MQTT-SN
• Not generally considered part of zero-conf but has the same autoconfiguration philosophy.
• Clients can automatically discover brokers
• Multiple brokers can work together no the same network providing failover and load balancing
• Next consider if mDNS would help if MQTT-SN doesn't
• It's a well established protocol with a lot of support (even my old printer supports it)
• It co-exists with DNS so you can also setup fallback DNS server(s) if devices really cannot use mDNS.  A fair number of home routers make this easy.
• Try it out, just set the client to use the .local tld.  Eg if the broker hostname is named my-broker then try my-broker.local
• AFAIK multiple hosts can have the same domain name in mDNS.
• DNS-SD might be supported but I suspect it's out-of-the-box support is much less than mDNS.

However

> Support for multiple brokers by clients is a simple suggestion and would make the system more resilient and more widely useful

It looks great as a single sentence.  In practice I don't believe it works anywhere nearly as well as you might immagine.  I've worked on a system that did this and it was the worst point of failure of the whole system.  With only 4,000 devices we were forever having to re-tune the design.  Controlling the problem at the DNS or TCP load balancer was far more effective and very little work indeed.  Problems:

• Multiple hostnames allow the client to decide which node to use for itself.  
• When one broker fails, its clients all simultaneously detect the problem and try the next one... resulting in a thundering hurd flooding it to destruction.  
• Then the [growning] hurd moves onto the next... and the next...  Now all of your brokers go down like dominoes.
• Preventing random network events triggering these harmonics is really not trivial.
• Disconnected clients cannot be reconfigured remotely.  
• When there are serious and unpredictable problems there's no way to remotely redirect things.  You've accidentally created your own DDOS bot swarm with no way to turn it off.  
• In the home you could go around switching things off, but even a college capus would have real trouble.
• This horribly compounds the thundering hurd problem. 
  
• The logic of how a client should handle multiple domain names is not obvious
• Should the hosts be ordered, prioritized, or randomized?
• If prioritised then should clients continue to attempt to reconnect to the higher priority hosts when they have been forced to connect to the lower one? MQTT uses long-lived TCP connections.  Unlike HTTP or SMTP, the choice of when to revert to a higher priority hostname is unclear
• If randomized a lot of embedded devices struggle with generating random numbers.  The only thing worse than a thundering hurd following a strict order is a thundering hurd following the same psudo random order.
• Is every manufacturer likely to implement this detail correctly even if it is standardized?
• Simple administration is problematic. 
•  At any one time 1-5% of devices may be offline even when customers are contracted to keep them on permanently.  If decommissioning one server node means removing one hostname from 4,000 devices, then how do you deal with the stragglers that didn't get the memo?
• In your blog you mention over 100 devices.  Does a home user really want to manually reconfigure all 100?

Conversely DNS is already multi-redundant from having multiple IP addresses per domain name.  Home DNS with the .home, .corp, .local TLDs is already pretty easy to administer at home. It also solves all of the problems above at the corporate level.

I think I'll make this my last post on the topic.

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/a1a1686c-5c7b-4af6-a1bd-3d1a59566f1en%40googlegroups.com.

[Bob Frankston]

I agree, I don't want to keep going around on this. I should note that I realize I'm repurposing a protocol designed for telemetry. But that's how systems evolve - repurpose what exists.

I think the key point is that I have limited control over what other devices, so I need something adopted as a standard that others can use. mDNS is interesting but limited to the physical topology of a local network. I designed a more general protocol in 1997 at Microsoft but we got UPnP instead. I'm also thinking consumer so managing a separate DNS isin't viable.

Overall the issues being raised are similar to those faced by any distributed packet network.

[Bob Frankston]

OK, one more note -- there is a workaround in having a fallback broker adopt the IP address of the failed broker. It's not ideal but it can work in some cases. It would still be useful to have a fallback protocol.

[Gambit Communications]

We have documented 400,000 connections with the old IBM MessageSight, eg. see

https://gambitcomm.blogspot.com/2019/01/200000-mqtt-connections-to-ibm.html

Now we will shoot for more with Eclipse Amlen.

[Andy Stanford-Clark]

We look forward to seeing what you find with Amlen :)

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/69953f02-f4f0-4651-a514-c2b7c60f6b82n%40googlegroups.com.

[Praveenkumar Vijayakumar]

Hi all,

Does anybody have any idea by using Locust to create multiple MQTT connections?

To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/39246D17-74D0-4C42-BD47-E2B672677470%40stanford-Clark.com.

--

Thanks & Regards

Praveen Kumar V Associate Quality Engineer Rently Software Development Pvt Ltd. India Ph: (+91) 8637478137 Self-Tours. Smart Doors. Smart Communities.

Disclaimer: The content of this email is confidential and intended for the recipient specified in message only. If you have received this email by mistake, please delete it from your mailbox, and do not forward it or any part of it to anyone else.
Warning: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.

[Gambit Communications]

https://twitter.com/MIMICSim/status/1585714711726673920