Some outsider feedback

[Nick Kolpin]

Hi all,

I'm interested in setting up distributed sensors for various uses and recently came across your radio protocol. Since you'll be getting together to work on the protocol and implementations this weekend I thought I'd provide some feedback from someone who is interested in using it and has just read through the docs. I hope the feedback and suggestions are useful and that you all have a fun and productive time over on the island. My plans are to try and get some nodes and a gateway running at Oxford Hackspace in the near future. 

Cheers,

Nick

Layer 1 & 2 protocols

-------------------------------

Radio isn't really an area I have much experience in. I'm assuming, having looked at the HopeRF boards that a decent amount of this comes from the capabilities of the selected chip. Most of the details make sense to me, but I don't really understand the description of how the checksum is calculated. I understand the HopeRF chip handles this for me, but when I first read the documentation I found this section very confusing.

Packet format

---------------------

This mostly makes sense to me. Having a specific delimiter between each of the data formats may make parsing the string easier in some languages. The sequence count only being 'a' on boot makes me think there's some sort of specific message on boot to be sent that I don't understand and didn't find documented. On the packet format page it has some details on node IDs, but I assume there may be other constraints elsewhere? Here it says the node ID is 16 bytes, without being explicit of how it's encoded, in the server API the node ID appears to be an integer (of unspecified precision). 

Packet data types

---------------------------

I think it's a great idea to have physically meaningful data with standardised units that can be understood by a human who happens to read the data string. I don't understand why there's a Sx data format because it isn't well defined. 

A generic dimensionless numeric format (Nx,y,z?) would be good for transmitting a value that is not covered by one of the other data types but is a list of numbers rather than a generic string that will not be parsed. 

I think an optional single timestamp (in unix nanoseconds or just float unix seconds?) data type per packet would be of use. This could override the timestamp stored on servers of when the packet was actually transmitted to the server, allowing low power nodes to batch packet transmission but later appear from the server as if they were sent at the correct time.

Server API

----------------

I assume the returned data are JSON encoded (or similar?), it's not clear from the API documentation page.

I don't really understand the difference between node ID and node callsigns. 

In the logtail, etc, shouldn't the ID of the originator of the message be a field, as well as the callsign of the gateway?

In the nodeData and lastData endpoints the data format is encoded in an integer. I think it makes more sense to use the unique character from the packet instead of various software implementations having to map those to integers.

Also in the nodeDat and lastData it seems all data types are really handled as floats (of unspecified precision). This isn't made clear in the data type documentation. 

Suggested improvements

-------------------------------------

Here are some general improvements that might be useful to consider:

* Tidying up and centralising the documentation: When developing and debugging things I find it very useful to have a single document (such as a PDF) to refer to, even if the documentation is also on some web pages. Along with the basic definitions of the protocol, expected and/or suggested behaviour for parts of the system (eg node, repeater, gateway) are often useful. Having good quality, clear documentation really is very helpful in developing around the protocol. As a good example of protocol documentation (that I happen to use regularly), see https://svnweb.cern.ch/trac/cactus/browser/trunk/doc/ipbus_protocol_v2_0.pdf

* Defining versions of the protocol: The documentation should preferably give a version number for the protocol along with a statement of what is fixed with a major version (low level protocol, etc.) and what may change in minor versions (eg adding but not removing data packet formats). 

* The protocol is currently designed to be fire and forget packets from a node's point of view. However I wonder if a simple acknowledge mechanism could also be implemented. The scenario I'm thinking of is a sensor deployed in the field that mostly just sends back measurements which are not a problem if lost, but also may flag some values as out of range, requiring some sort of action. For the alarm level messages a request for acknowledgement could be added to the packet, with the gateway sending back (via any repeaters) an acknowledge packet when the original message has been uploaded to a server. The sensor could then listen for this acknowledge and if it isn't received attempt an alternative communication method.

* Allow nodes to define a manifest describing the data types used: Currently each value is just as a data type and number (eg temperature 1 and 2). When registering nodes could users also upload a more meaningful description per expected transmission value? As an example "inside" and "outside" temperatures.

[James Coxon]

Hey Nick,

Thanks for the feedback, I try and answer some of the questions (though some might be better picked up by others especially about the ins and outs of the server side and some of the questions we managed to answer on IRC!) Apologies for rough inline replies.

Layer 1 & 2 protocols - if you use the RFM69 for as your radio then it internally does the CRC, there is more info there as people have used other chipsets and so just its just documentation. My suggestion to anyone starting is to stick with the RFM69.

Packet format - the 'a' packet is for boot, lots of nodes will send a hard coded Location to update the map, also if you see loads of 'a' packets it means that your node is resetting which might therefore need investigation. In regards to node ID - not sure what happens server side.

Packet data types - yes agreed about SI units, that said I'm certainly at fault for often sending my voltage readings as mV as it avoids working with floats on a micro. The S unit is for a photosensor, no one is ever going to truly calibrate this and can't use Lux as a measurement as we've already used up the L data type for location. The 'N' data format is a good idea, I don't see why that couldn't be added to the server side, from the UKHASnet hardware side of things it doesn't care what data is being passed around.

We've discussed the idea of timestamped packets - this is a tough one - they would allow you to backfill data in but it adds complexity to nodes as you'd have to have a method of keeping track of time (lots of nodes powerdown due to their power sources), you'd then need to sync the nodes times etc. I'm not sure server side is setup to accommodate backfilling data either. That said you could easily send the timestamp in the X field and then parse it yourself if you felt that you needed it - UKHASnet was actually conceived to not need a server (human readable etc) so you could have your gateway pass to the UKHASnet servers and perhaps your own dashboard if the data was critical (though it breaks the first rule of UKHASnet as all packets are unreliable!).

Server API - I'm going to skip this as its not my domain at all.

Suggested improvements - I'd agree that our documentation is lacking, even after a weekend of hacking stuff I haven't yet added anything to the wiki. Perhaps we need to have a documentation drive at some point.

In regards to an acknowledgement for packets - as I said before it wouldn't be difficult to customise local nodes but I'm not sure that there is much to be gained, either the packets are going to get through or not, one way to try and improve the chance of getting that data through would be to increase the freq of packets if the value had hit an alarm (i.e. if it was intermittent interference getting in the way). One nice thing about the system is that with a few distributed nodes if one link goes down then the packet will just go the other way back to the gateway/server. I think that if you are monitoring something that was really important than having an alternate comms might be a good idea, perhaps have an infield GSM gateway, it could parse local packets and normally it just acts as a simple UKHASnet repeater (relying on packets being passed to the main gateway) but if it sees something bad then it dials home

I'd agree a manifest would be something that would be helpful, for situations outside the use of SI units or to highlight the different sources of temperature. Again something to be added to the list of things to do...

Hope that helps a little bit - really grateful for you spending sometime thinking about the network as a whole, the key thing now though is that you get some nodes running!

James

--
You received this message because you are subscribed to the Google Groups "UKHASnet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ukhasnet+u...@googlegroups.com.
To post to this group, send email to ukha...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/ukhasnet/CADAK5N7sJ_1mJ1wkiFzrsMi-z7f75uOPjoYdYxOT9NVCrcJ-eA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Axford]

Comments inline (and some cut&paste to keep things together)

On 20 March 2016 at 08:29, James Coxon <jac...@gmail.com> wrote:

On 16 March 2016 at 22:28, Nick Kolpin <nick....@gmail.com> wrote:

 

 

Packet format

---------------------

This mostly makes sense to me. Having a specific delimiter between each of the data formats may make parsing the string easier in some languages.

One of the challenges here is that the rfm69 packet buffer is only 64 bytes so this imposes a limit on packet length this has meant trying to have the bare minimum of stuff in the packets, adding a delimiter to make the data section easier to read has little benefit to the protocol by can add to the bytes used. Currently all bar one data type is numeric and the :string type should be last (might need to confirm the documentation states that. This means it's not to difficult to parse the current data strings.

 

The sequence count only being 'a' on boot makes me think there's some sort of specific message on boot to be sent that I don't understand and didn't find documented. On the packet format page it has some details on node IDs, but I assume there may be other constraints elsewhere? Here it says the node ID is 16 bytes, without being explicit of how it's encoded, in the server API the node ID appears to be an integer (of unspecified precision). 

 

Packet format - the 'a' packet is for boot, lots of nodes will send a hard coded Location to update the map, also if you see loads of 'a' packets it means that your node is resetting which might therefore need investigation. In regards to node ID - not sure what happens server side.

Currently I think the only actual restrictions on Node ID is that they are limited to a maximum of 16 characters and the parser limits the characters used to being in the set [A-Za-z0-9]. There's been a bit of debate on irc this evening as to whether these should upper case only (as all the original nodes were). If we went for upper case only one sensible solution might be to accept lower case but then convert them to uppercase internally (this would be least impact for existing nodes). As noted the documentation on this seems to be fairly lacking and various bits of code might need some tightening up.

The integer representation of Node IDs you see on the website are just an internal ID generated by the Database. The same is also the case for the Upload ID seen on the logtail and IRC Bot.

I don't really understand the difference between node ID and node callsigns. 

In the logtail, etc, shouldn't the ID of the originator of the message be a field, as well as the callsign of the gateway?

I think node ID might be a bit of a confused term. In some places it's the same as a node callsign (I think we originally aimed to stay away from calling things callsigns to distinguish it from amateur radio), however internally within the database and things interacting directly with it node ID can also refer to the integer id used within the DB. This might be something we need to clarify better in the documentation.

Having links for the originator (and intermediate nodes) might be more challenging as at the point of display the packet hasn't been processed. In the case of new nodes they might not even exist in the DB at that point.

Also in the nodeDat and lastData it seems all data types are really handled as floats (of unspecified precision). This isn't made clear in the data type documentation. 

A couple of the earlier data types are being stored as floats in the DB even though they are described as integers in the documentation. This is really a historical thing and could probably be fixed although that might be interesting to do. I'll see if I can at least update the documentation for those cases. I'm not sure if the api is doing anything more with the data after it leaves the DB.

If you haven't found it yet, most of the server side code is also on github at https://github.com/UKHASnet-hub this may help answer some questions (and probably raise more). Anyone is welcome to improve the wiki, you'll need to create an account then ask on IRC for it to be approved to edit. Pull requests are welcome for improvements/fixes to the code.

Mike