device configurations MQTT topics design

[Harut Martirosyan]

Hi

I'm creating an IoT system where I should be able to remotely re-configure thousands (or millions) of devices. 

We can simplify the term configuration down to a simple map with keys/values.

Each device has a thing called a template, that hosts all possible configuration keys, as well as default values for that type of device.

The default values still can be changed, and the change should (somehow) be propagated to child devices.

Each individual device can override it's template values to specific ones, so the final configuration of the device should be the template configuration with specific configurations overriding template configurations.

So the questions is, how to organize all that to be efficient using MQTT.

1. Device subscribes to individual topic devices/1 and server-side software publishes the final config to each individual topic

• This is good because each device receives the final config and there's nothing it should be worried about
• This is bad because server needs to send thousands of messages, this approach doesn't really scale.

2. Client subscribes to an individual devices/1 and template templates/1 topics and performs overrides when receives separate configs.

• This is good because you publish just 1 message to corresponding topic when changing device config or template config.
• This is bad because device gets all the configuration that it shouldn't really get
• This is bad because devices should handle merging itself.

3. Any better way of organizing this?

Thanks.

[Ranjith Kumar Diraviyam]

Following is my understanding of your Question. 

1. The devices will be configured in batches. 

2. The configuration sent to all devices are same. 

3. You don't want to send message to already configured devices 

4. And at the same time reduce the load on the server. 

5. I believe it is a one time configuration. 

If my inference is right, One of the option I would suggest is, 

You can allow all the devices to subscribe to a common topic. However, you can send message to a specific device for a particular topic. You will be able to select the list of devices and send the message. 

Hope this helps. 

Ranjith 

www.bevywise.com 

[Phil Scanlon]

You could go for a combination of the 2.  If some of the messages that are to be delivered are unique to a device, then you should look at using the topic hierarchy to separate the device specific configuration from the common pieces.  Scale and device addressing is one of the concerns that the message broker is designed to address.

The key decision you are making is:  are my devices completely dumb, or do i want to perform some compute at the edge.  I currently see a combination of the 2.  If devices are just transmitting data, then they can be relatively dumb.  If they are doing 2 way command and control, then they are more intelligent and you need to process more logic on the device.

For Common messages - Fan out is the role of the broker.  a publisher (of the configuration) should use a wildcard (device/*) to publish -  1 send, and the broker will take care of managing the delivery to each registered device.  Additionally, you can use the device ID as part of your topic hierarchy - adds security and ensures subscribers can only subscribe to their one device topics.  In theory the common template should only be sent as part of the device registration process (so could still only be sent to a single device, or if there is a change to the core template (all devices).  This is also very common in use cases where you are managing device firmware updates.

Some thoughts on your topics: 

Publish to                 Message is delivered to

device/*                    All devices

device/$deviceid/     single device

template/*                all devices

template/$deviceid/ single device

template/*/new       devices registered to receive a new template

You could also have some other topic level to create subsets of devices (based on their version, brand, capabilities, etc).  You could even use a date/time topic to be able to address any devices that have registered after a specific date time.  e.g. when a device receives the template, it subscribes to a topic device/$deviceid/yyyy/mm/dd  your server side logic can then send messages to devices registered on a specific date, or ranges.  Alternatively you could have a topic for new devices: device/$deviceid/new and your server side can periodically send a message to device/*/new with the template.

you may also need to take into consideration a device requesting it's configuration (reboot, initial config etc).

feel free to PM me if you want to discuss in more detail.

[Harut Martirosyan]

The configurations I was trying to describe are not fixed, i.e. users can change both device and template configurations at any time. Each device theoretically can have a unique configuration (that overrides the template configuration). Device's template is fixed, i.e. device is always of the same type. My questions is about how to organize all that configuration changes so that if a template parameters are changed, broker doesn't end up propagating the change to each device individually.

[Harut Martirosyan]

Thanks for your reply.

The combination approach was my way to go at the moment.

Templates are actually sealed to device at production time, the device can never receive a new template or remove existing one. Only the configuration of the template can be changed.

[Harut Martirosyan]

Thanks for your reply.

The combination approach was my way to go at the moment.

Templates are actually sealed to device at production time, the device can never receive a new template or remove existing one. Only the configuration of the template can be changed.

On Thursday, January 25, 2018 at 5:44:23 AM UTC+4, Phil Scanlon wrote:

[Harut Martirosyan]

There's also one more major issue:

The device may end up on inconsistent configuration state, i.e. if both specific and template configurations are changed, and device receives only one of them (due to splitting configurations to separate topics).

On Thursday, January 25, 2018 at 5:44:23 AM UTC+4, Phil Scanlon wrote: