Arduino MRT1000 as GPS SignalK node

[Brian P]

Hi. 

I'm new to SignalK, but understand the issues and approve of the approach. I'm researching building inexpensive but reliable (and redundant) boat instruments to take with me on small rental boats, and was looking for a suitable communications protocol between measurement devices and a display. Right now, I have built combined I2C sensor and display units using basic Arduino's and LED displays, and the second project had me connecting a GPS via Arduino to multiple LED display units. 

Now I'd like to separate the sensors from the displays. Looking at robust packages incorporating as many features as possible, I came across the Arduino MKR1000 https://www.arduino.cc/en/Main/ArduinoMKR1000 which incorporates a power supply unit, processor and wifi module. I ordered one of these to play around with, as this should be sufficient hardware to read out a GPS unit and construct SignalK JSON messages. This is probably within my capabilities. 

What is not in my capabilities is writing the script to establish communication between the Arduino and the SignalK MQTT server. Has anyone tried writing such a thing for Arduino before? 

Cheers, 

Brian. 

[rob...@42.co.nz]

Hi Brian,

This project has a Signalk implementation that needs updating, but has the functionality required to send/receive messages https://github.com/rob42/FreeboardMega/tree/signal_k_dev/lib/SignalK. The critical part is the streaming json support, so you dont blow the RAM with a big message.

Also this would be a good start https://github.com/rob42/FreeboardPLC_v1_2 and the related Due code https://github.com/rob42/freeboardDue, which should be compatible with the SAM chip in the MRT.

The FreeboardPLC_v1_2 project is quite a way ahead of the signalk and Due project. All need some love, it would be much appreciated if you could contribute, along with support for the MRT.

Rob

[Brian P]

Hi Rob,

I'll do my best, but it will take a few months before I expect to be able to make a useful contribution. I had a look at the code, which to me looks good (though I was a bit surprised you didn't use an existing GPS library (such as TinyGPS++)). 

I will have to set up a test system first which can receive and interpret the SignalK communications, I'll probably install OpenPlotter for this purpose. 

The hope is that in the end I'll be able to put together independent wireless sensors using the MKR1000 for the GPS, compass and environmental monitoring systems, that require only power and an on-board wifi network. 

Cheers, 

Brian. 

[rob...@42.co.nz]

Ok no rush.

There was a reason for not using TinyGPS, I forget what just now.

Cheers
Rob

[Brian P]

Hi Rob, 

It might be because you wanted the autobaud functionality in your GPS code... TinyGPS++ is quite nice, but it also doesn't extract what I wanted it to, so I had to use the custom TinyGPS field extractor (to get the coordinates in DM.M, and the "void" character). (Now there's also a new, even smaller GPS library, NeoGPS, though I haven't looked into that yet.)

I took a look at your code, and it's somewhat beyond my programming skill. I can see where the signal K model is located and how it's set up, but I cannot find where the GPS messages are translated into SignalK. Now I'm a bit torn between using ArduinoJson (https://github.com/bblanchon/ArduinoJson) to construct a simplified message, or to lift the models from Freeboard instead. Alternatively, I can try just installing the Due component of Freeboard onto the MKR1000, and see what happens :). I understand the niceties incorporated in Freeboard, of not having to reprogram anything for a GPS signal K nodule (node module) or an environmental monitoring nodule, and that's what the Freeboard version seems to be able to do. 

The MKR1000 seems to be quite capable so far, it contains six hardware serial communications modules (SPI / I2C / UART) under the hood, which can be multiplexed to the digital pins. So I now just use a spare set of pins for connecting the GPS to.

I did see one potential minor improvement in your code. In https://github.com/rob42/FreeboardMega/blob/signal_k_dev/FreeboardMega.cpp, you maintain an "interval" value in the "calculate" method. Later on you use this to periodically perform particular functions. However, you could bypass intervals altogether and compare to the builtin millis() instead. 

After constructing the GPS sentences, I'll have to try and see if (and how) I can establish a network connection, preferably automatically, from the Arduino to my Raspberry Pi 3 running OpenPlotter. Then the trick will be to get a UUID from the RPi Signal K server and have it subscribe to the incoming GPS message stream. Experiments continue...

[Brian P]

One more quick update, I set up OpenPlotter on the RPi3, and I can connect to it from the MKR1000 via Wifi. Next up, subscribing to the SignalK server via MQTT, probably using the following library: 

https://github.com/knolleary/pubsubclient

Cheers, 

B.

[rob...@42.co.nz]

Hi Brian,

The arduino receives the GPS data from the GPS (and sorts out baud etc with autobaud) but just forwards the String to the RPi for further processing. The RPi sends LAT and LON back to the arduino (along with other data) to update the arduino internal data model. Hence you cant find the code cos its not there :-)

I tried ArduinoJson but its based on a fixed buffer size, and its too small for signalk many messages. If you increase your buffer size you blow the tiny RAM on the arduino, hence I went to the streaming model, which has no json size limit.

The streaming json code and the data model is useful as it allows easy production of signalk messages on the fly, when the incoming data tends to arrive in various messages. So the architecture two main processes (incoming messages)>(translate to internal model)>(push into model). (Trigger)>(model to signalk)>(out).

I found the direct pass-through of incoming messages (like NMEA) to outgoing signalk with no state held internally was problematic. There are many cases where you want signalk data that requires data from several NMEA messages, which are gone by then, eg calculate anchor watch distance.

[Brian P]

Hi Rob, 

Thanks for the explanation. We have one benefit now: 250 k of memory on the MKR1000. I have most of the proof-of-concept code in place now, and I'm only using 16% of the memory, which means there's more than 200k of space left for constructing a JSON message. I will now take a look at whether that can be done using your streaming library (i.e. whether I understand enough of the library to apply it). 

I've managed to make an MQTT connection from the Arduino with the broker running on the Raspberry Pi. It looks like the broker has to be manually started on the Pi from the command line for that to work. It *looks* like the broker is running from the OpenPlotter settings panel (as it lists an IP address and port), but apparently that's just for show. I'll double-check on the OpenPlotter forums whether this is really not running.

Cheers, 

Brian. 

[Brian P]

Hi Rob,

No luck sending a message over that MQTT connection, and I have no idea why the publishing of messages fails, problem could lie on either side. ATM, my MKR1000 is starting up with the following debug messages (I've modified the GPS coordinates, so the checksum is now incorrect...):

-->

Scanning available networks...
** Scan Networks **
number of available networks:14
[snip]
3) SignalK Signal: -57 dBm Encryption: WPA
[snip]
Attempting to connect to WPA SSID: SignalK
Waiting 1 seconds for network to settle
Attempting to connect to WPA SSID: SignalK
Waiting 1 seconds for network to settle
You're connected to the networkSSID: SignalK
BSSID: [snip]
signal strength (RSSI):-54
Encryption Type:2
IP Address: 10.10.10.204
MAC address: [snip]
Attempting MQTT connection...connected
Subscribing to ... signalk.put
{"self":"D344B1D0","version":"1","vessels":{"D344B1D0":{"uuid":"D344B1D0","environment":{"depth":{"belowTransducer":{"value":10.83,"timestamp":"2016-04-15T17:56:50.000Z","source":{"type":"NMEA0183","sentence":"DBT","label":"signalk/signalk-parser-nmea0183"}}}}}}}$PMTK001,220,3*30
$PMTK001,300,3*33
$PMTK001,314,3*36
$GPGGA,104712.000,5023.2238,N,01545.2436,E,1,07,0.98,84.9,M,44.8,M,,*58
$GPRMC,104712.000,A,5023.2238,N,01545.2436,E,0.21,205.96,250916,,,A*63
Successful GPS fix. Coordinates in D M.M format are:
  Latitude: 50 degrees, and 23.22 minutes North.
  Longitude: 15 degrees, and 45.24 minutes East.
$GPGGA,104715.000,5023.2238,N,01545.2436,E,1,07,0.98,84.9,M,44.8,M,,*59
$GPRMC,104715.000,A,5023.2238,N,01545.2436,E,0.14,149.20,250916,,,A*62
Sending SignalK Demo...
Failed to publish SignalK Demo

<--

It has no problem connecting to the Openplotter signalk MQTT server, but I don't think I quite understood the whole subscription thing. I've put my proof-of-concept Arduino code in a repository here:

https://bitbucket.org/toQDuj/signalk-gps-nodule

If you're interested, and know what could be wrong with this, I'd be very happy to hear! 

Cheers, 

Brian. 

On Sunday, September 11, 2016 at 11:45:13 PM UTC+2, rob...@42.co.nz wrote:

[rob...@42.co.nz]

Hi Brian,

In the signalk java server you can turn on voluminous debug :-)
in the log configuration screen add a row for:

nz.co.fortytwo.signalk.processor
nz.co.fortytwo.signalk.handler

Set them to debug, no restart required. The logs will print on the console and into the log files too.

Might give a clue. Maybe turn them on, try a MQTT message and off, then study the output.
Put anything interesting here

Rob

[Brian P]

Will do. For this I'd probably need to switch from openplotter to freeboard on the raspberry pi. Will take a moment. 

--
You received this message because you are subscribed to a topic in the Google Groups "Signal K" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/signalk/oGtUONASVOE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to signalk+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian P]

Hi Rob, 

Back from a journey. 

Could you tell me which log configuration screen are you talking about? In a fresh freeboard install (onto a full raspbian jessie) I can only see a "freeboard-server/conf/freeboard.cfg" and a "freeboard-server/conf/log4j.properties" pair of configuration files. A grep -r "signalk" in the freeboard-server directory also reveals nothing on where or how signalk is handled. The configuration menu in the web interface also shows no information on logging settings. 

Just in case you were talking about a logging menu in OpenPlotter, I checked it out briefly but could not find anything there either. 

Cheers, 

Brian. 

[Brian P]

Also, quick update. I'm able to store data in a SignalkModel, although I'm using a simpler approach than used in Freeboard, e.g.:

      skmodel.setValue(NAVIGATION_POSITION_LATITUDE, (float)atof(NMEALat.value()));

and to get the value:

      skmodel.getValueFloat(NAVIGATION_POSITION_LATITUDE)

Now the trick will be to build a constructor for building a JSON SignalK message out of that model. 

I'd be happier, though, if I could set and read the values by using something like:

      skmodel.navigation.position.latitude

but that doesn't seem to return a usable value. Setting a value that way misses the model-integrated data type check, which is nice to have. 

All in all, this is quite a bit different from the way we do things in Python, and I'm learning a lot. 

Cheers, 

B.

[Brian P]

Another quick update. I modified the Freeboard SignalK model and helper classes to output to a PString instead of HardwareSerial. The required buffer size for this message has to be slightly shy of 3 kByte. Reserving 4 kByte for this message does not give any problems. It should, therefore, be no problem to send the message over MQTT to the server. 

I'm still getting nowhere with publishing that message, however, or finding out what needs to be done in that respect. 

[rob...@42.co.nz]

The log management screen is in the signalk-java (https://github.com/SignalK/signalk-java) project. The freeboard-server project (https://github.com/rob42/freeboard-server) doesnt have direct control of logging, you have to edit the conf/log4j.properties

You need the signalk-java project up and running for MQTT. Its got a broken build just at the moment, so it wont install the apps, but otherwise ok.

Yes java is a statically typed language, as opposed to python/javascript which are dynamic. The big bonus of static typing is really powerful IDE's (aka eclipse) so most problems are found at design time rather than runtime. But the object.method.method.value thing is not there as a consequence. You can do that in derived java projects like the groovy language.

The SignalkModel has a number of methods for getting deep values and subtrees.

[rob...@42.co.nz]

The methods on signalkModel do quite a lot under the hood. The underlying map is a ConcurrentSkipList, which allows any number of concurrent reads and writes without locks and collisions. That proves quite important when you have lots of processes sharing data. It also manages the source, sourceRef, and timestamp data for a value, and has facilities for securlty access control.

During the evolution of signalk a number of things surfaced (like subscriptions and security) which required some lateral thinking. On the surface there seem to be lots of simple ways to get a result, and there are for limited use-cases, but when you go further they get REALLY difficult. A case in point is the node server implementation, which I believe doesnt use the single concurrent model approach, but they have found subscriptions challenging.

The java server uses a two stage approach. Incoming data is translated into clean signalk and is placed into the model. Events or subscriptions then trigger outgoing data to be read from the model and sent. Thats hugely simplified problems that require arbitrary collections of data like low frequency periodic subscriptions.

[Sailoog]

Hi Brian,
sorry for the delay, this is how MQTT works in openplotter: https://github.com/sailoog/openplotter/issues/113

[Brian P]

Hi Sailoog,

Thanks for the response. So, just to make sure I'm not misunderstanding, this is a way to get messages from a wireless sensor into openplotter, right? All other examples I've seen so far using SignalK and MQTT are for sending status messages from openplotter to an external visualisation application. 

If this works, it would open up the gates for a whole bunch of wifi-connected sensors, so I'm quite keen on getting this off the ground.

B.

On Mon, Oct 10, 2016 at 7:17 PM, Sailoog <in...@sailoog.com> wrote:

Hi Brian,
sorry for the delay, this is how MQTT works in openplotter: https://github.com/sailoog/openplotter/issues/113

--

[Brian P]

Hi Rob. Cross-post of a status update on the openplotter github issue tracker:

Alright, message is coming through after adjustment of the buffer sizes on the Arduino MQTT software side. In the terminal window from which openplotter was started, I'm getting the following error now, everytime the status message is sent. This indicates that it's being received by SignalK, but there may be a problem with SignalK/Openplotter's JSON parser?

`

Could not parse JSON:{"context": "vessels.urn:mrn:imo:mmsi:00000000","updates":[{"source":{"type": "notifications","src":"mqtt"},"timestamp":"2016-10-11T19:29:25.429Z","values":[{"path":"notifications.mqtt.gpsput","value":{"message": "{"vessels":{"self":{"navigation":{"courseOverGroundTrue": {"value":0.0000000000},"courseOverGroundMagnetic": {"value":93.0000000000},"headingMagnetic": {"value":0.0000000000},"magneticVariation": {"value":0.0000000000},"headingTrue": {"value":0.0000000000},"pitch": {"value":0.0000000000},"rateOfTurn": {"value":0.0000000000},"roll": {"value":0.0000000000},"speedOverGround": {"value":0.0000000000},"speedThroughWater": {"value":0.0000000000},"state": {"value":"Not defined (example)"},"anchor":{"alarmRadius": {"value":0.0000000000},"maxRadius": {"value":0.0000000000},"position":{"latitude": {"value":0.0000000000},"longitude": {"value":0.0000000000},"altitude": {"value":0.0000000000}}},"position":{"latitude": {"value":0.0000000000},"longitude": {"value":0.0000000000},"altitude": {"value":0.0000000000}}},"alarm":{"anchorAlarmMethod": {"value":"sound"},"anchorAlarmState": {"value":"disabled"},"autopilotAlarmMethod": {"value":"sound"},"autopilotAlarmState": {"value":"disabled"},"engineAlarmMethod": {"value":"sound"},"engineAlarmState": {"value":"disabled"},"fireAlarmMethod": {"value":"sound"},"fireAlarmState": {"value":"disabled"},"gasAlarmMethod": {"value":"sound"},"gasAlarmState": {"value":"disabled"},"gpsAlarmMethod": {"value":"sound"},"gpsAlarmState": {"value":"disabled"},"maydayAlarmMethod": {"value":"sound"},"maydayAlarmState": {"value":"disabled"},"panpanAlarmMethod": {"value":"sound"},"panpanAlarmState": {"value":"disabled"},"powerAlarmMethod": {"value":"sound"},"powerAlarmState": {"value":"disabled"},"silentInterval": {"value":300},"windAlarmMethod": {"value":"sound"},"windAlarmState": {"value":"disabled"},"genericAlarmMethod": {"value":"sound"},"genericAlarmState": {"value":"disabled"},"radarAlarmMethod": {"value":"sound"},"radarAlarmState": {"value":"disabled"},"mobAlarmMethod": {"value":"sound"},"mobAlarmState": {"value":"disabled"}},"steering":{"rudderAngle": {"value":0.0000000000},"rudderAngleTarget": {"value":0.0000000000},"autopilot":{"state": {"value":"off"},"mode": {"value":"powersave"},"targetHeadingNorth": {"value":0.0000000000},"targetHeadingMagnetic": {"value":0.0000000000},"alarmHeadingXte": {"value":0.0000000000},"headingSource": {"value":"compass"},"deadZone": {"value":0.0000000000},"backlash": {"value":0.0000000000},"gain": {"value":0},"maxDriveAmps": {"value":0.0000000000},"maxDriveRate": {"value":0.0000000000},"portLock": {"value":0.0000000000},"starboardLock": {"value":0.0000000000}}},"environment":{"airPressureChangeRateAlarm": {"value":0.0000000000},"airPressure": {"value":1024.0000000000},"waterTemp": {"value":0.0000000000},"wind":{"speedAlarm": {"value":0.0000000000},"directionChangeAlarm": {"value":0.0000000000},"directionApparent": {"value":0.0000000000},"directionTrue": {"value":0.0000000000},"speedApparent": {"value":0.0000000000},"speedTrue": {"value":0.0000000000}}}}}}

Could not parse JSON:"}}]}]}

`

[Teppo Kurki]

Your JSON is not valid, there is an extra quotation mark after vessels.

Before cross posting to multiple places a suspected fault in JSON parser you should validate your input first. It is very much more likely that the input is faulty.

[Brian P]

I know, I just got a bit overexcited since this was the first real progress I've had in weeks on this project. 

On Wed, Oct 12, 2016 at 6:48 AM, Teppo Kurki <t...@iki.fi> wrote:

Your JSON is not valid, there is an extra quotation mark after vessels.

Before cross posting to multiple places a suspected fault in JSON parser you should validate your input first. It is very much more likely that the input is faulty.

[Sailoog]

I think it is not your fault, the extra quotation is added by openplotter. There are several solutions Let's go to treat this in github: https://github.com/sailoog/openplotter/issues/113

El dimecres, 12 octubre de 2016 7:45:05 UTC+2, Brian P va escriure:

I know, I just got a bit overexcited since this was the first real progress I've had in weeks on this project. 

On Wed, Oct 12, 2016 at 6:48 AM, Teppo Kurki <t...@iki.fi> wrote:

Your JSON is not valid, there is an extra quotation mark after vessels.

Before cross posting to multiple places a suspected fault in JSON parser you should validate your input first. It is very much more likely that the input is faulty.

--
You received this message because you are subscribed to a topic in the Google Groups "Signal K" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/signalk/oGtUONASVOE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to signalk+u...@googlegroups.com.

[Brian P]

Hi Robert,

The Freeboard Signalkmodel which I've now modified a little to print to PString always sends the complete message (by its nature). Do you think it would be worthwhile to have an "isUpdated" boolean for every value, so that more sparse deltas can be sent? It might add significantly to the complexity of the model, so I don't know if it's worthwhile... 

Cheers, 

B.

On Mon, Oct 10, 2016 at 12:17 AM, rob...@42.co.nz <rob...@42.co.nz> wrote:

[Brian P]

Another update, and a question about the signalk message content:

I am now able to construct and send true signalk deltas, albeit with only a single value per message, based on the example from here: http://signalk.org/specification/master/data_model.html

My signalk message now contains, for example:

{

"context": "vessels.123456789",

"updates": [{

"source": {

"device": "/dev/gps001",

"pgn": "",

"timestamp": "2016-10-16T16:08:30Z",

"src": "signalkGPSNodule"

},

"values": [{

"path": "navigation.courseOverGroundTrue",

"value": 6.1126666

}]

}]

}

However, I cannot find the specifications on what the source fields are supposed to contain (apart from timestamp). Can anyone inform me about what "src", "device", and "pgn" are supposed to be in this case, or if they are supposed to be filled in at all in this case? 

Latest code for the project can be found here:

https://bitbucket.org/toQDuj/signalk-gps-nodule

[Teppo Kurki]

--
You received this message because you are subscribed to the Google Groups "Signal K" group.
To unsubscribe from this group and stop receiving emails from it, send an email to signalk+unsubscribe@googlegroups.com.

[Teppo Kurki]

Sorry about the empty and meaningless message - pressed something and off it went.

On Sun, Oct 16, 2016 at 7:23 PM, Brian P <bria...@gmail.com> wrote:

Another update, and a question about the signalk message content:

I am now able to construct and send true signalk deltas, albeit with only a single value per message, based on the example from here: http://signalk.org/specification/master/data_model.html

My signalk message now contains, for example:

{

"context": "vessels.123456789",

"updates": [{

"source": {

"device": "/dev/gps001",

"pgn": "",

"timestamp": "2016-10-16T16:08:30Z",

"src": "signalkGPSNodule"

},

"values": [{

"path": "navigation.courseOverGroundTrue",

"value": 6.1126666

}]

}]

}

However, I cannot find the specifications on what the source fields are supposed to contain (apart from timestamp). Can anyone inform me about what "src", "device", and "pgn" are supposed to be in this case, or if they are supposed to be filled in at all in this case? 

For N2K data: src + pgn + label

For NMEA0183 data: sentence + talker + label

Source information is for debugging and making sense out of server output. In custom setups like yours you would want to figure out what you want there and use $source string value, which is "A dot spearated path to the data. eg [type].[bus].[device]".

My (forthcoming) engine room temperatures sensor node is going to make the data available over tcp with custom mdns name, like engineroomtemps.local and sources like $source: "tcp.engineroomtemps.X", X being index of the temperature reading, multiple 1W sensors connected to the box.

Source information is not entirely mandatory, the idea being that the server can assign it, since it should know something meaningful about where the data is coming from, whereas a simple sensor will just output whatever values it has. In this case the server glue code can fill in the source data.

Sources should be definitely documented. It would be really great if somebody could

- review the examples with sources in the current docs, as the documentation was born over a length of time and source handling has evolved

- add a bit more documentation about source usage per #238 (see below), for example add to  https://github.com/SignalK/specification/blob/master/gitbook-docs/data_model.md

For now take a look at

https://github.com/SignalK/specification/blob/master/test/sources.js#L12-L64

and see discussion (a bit sprawling) at https://github.com/SignalK/specification/pull/238

Furthermore the relevant schema parts are https://github.com/SignalK/specification/blob/master/schemas/definitions.json#L14-L63

[Ric Morris]

Definitely see value in sensor nodes providing the source them selves.

The issue I see in the delta format from this perspective are the context -- which it is inconvenient for the node to learn about and comes from the network context rather than being something inherently about the node it's self -- and the timestamp -- which requires an RTC/battery or ntp call. Along with mDNS (not to much of an incumbrancer), websockets (definitely a pain the in hole) and http it all adds weight.

On Thursday, September 1, 2016 at 4:49:40 PM UTC+1, Brian P wrote:

Hi. 

I'm new to SignalK, but understand the issues and approve of the approach. I'm researching building inexpensive but reliable (and redundant) boat instruments to take with me on small rental boats, and was looking for a suitable communications protocol between measurement devices and a display. Right now, I have built combined I2C sensor and display units using basic Arduino's and LED displays, and the second project had me connecting a GPS via Arduino to multiple LED display units. 

Now I'd like to separate the sensors from the displays. Looking at robust packages incorporating as many features as possible, I came across the Arduino MKR1000 https://www.arduino.cc/en/Main/ArduinoMKR1000 which incorporates a power supply unit, processor and wifi module. I ordered one of these to play around with, as this should be sufficient hardware to read out a GPS unit and construct SignalK JSON messages. This is probably within my capabilities. 

What is not in my capabilities is writing the script to establish communication between the Arduino and the SignalK MQTT server. Has anyone tried writing such a thing for Arduino before? 

Cheers, 

Brian. 

[Brian P]

The sensor could send a bit of information on what it is, which would allow the user to identify it on the ship. For the GPS Wifi node, I'll probably use $source:"wifi.deck.gps001". 

The other problem you indicated is interesting as well. The GPS unit has UTC time by definition, but a compass or environmental sensor bank wouldn't. The available options are 

  1) For the signalk server to include an NTP server, providing timecodes to connecting modules, or

  2) For the signalk deltas to potentially omit the "timestamp"-flag, to be added upon receipt by the server. 

B.

--
You received this message because you are subscribed to a topic in the Google Groups "Signal K" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/signalk/oGtUONASVOE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to signalk+unsubscribe@googlegroups.com.

[rob...@42.co.nz]

Assuming a dumb device that just pings data, the java server partially identifies it by its source (serial port or network ip, etc) and potentially by mac, or misc transport param. Not foolproof but good enough for most things etc. A bevy of thermometers on a shared serial line would presumably need to send a unique identifier to be useful, but if they support that kind of setup they probably do, eg all TCP devices have mac and IP address.

If no timestamp is received with a message the server adds the current server timestamp. The assumption is if the device doesnt care about time and its local, then transmission delays are minimal, time is not critical, and adding timestamp within a few millisecs of transmission is near enough. eg I want current engine temp, but its fine if its accurate within 5 sec, cos it just doesnt change that fast.

Again if you have a device where time is critical you would expect that device to know the UTC time, get the UTC time,  or send a relative timestamp of some sort.

[Brian P]

I am very happy to see that the SK concept already adds more than I thought it did: no need for me to add the information I don't have to the delta. For example timestamp information (that I don't have) doesn't need to accompany a message from an environmental sensor bank. 

The sensor can be partially aware of what and where it is (i.e. deck-mounted GPS). It would be useful if such info can be provided along with the remainder of the SK delta. As I understand, for my example this can be done by setting a "$source:deck.gps" field, and the information that it came in on Wifi via MQTT will then be added by SK (?).

Cheers, 

B.

[rob...@42.co.nz]

Basically yes, although I cant remember off hand if the java server will pre-pend `ip_address.mqtt` to an incoming source. Probably a good idea if it doesnt. This kind of practical issue is why we need to build real devices to use the protocol we define, and refine the protocol as we go to make it properly functional..

[Brian P]

Right, well, I'll clean up the code best I can, and wait for OpenPlotter to implement their incoming SK parser. In the mean time, I'll also make either 1) a second SK-capable wifi sensor with humidity, light, pressure and temperature sensors, or 2) a wifi-connected display unit that can parse and display incoming SK messages. 

On Sun, Oct 23, 2016 at 4:55 AM, rob...@42.co.nz <rob...@42.co.nz> wrote:

Basically yes, although I cant remember off hand if the java server will pre-pend `ip_address.mqtt` to an incoming source. Probably a good idea if it doesnt. This kind of practical issue is why we need to build real devices to use the protocol we define, and refine the protocol as we go to make it properly functional..

--

[Ric Morris]

Brian, How about a wifi sensor that takes the pulse signal from a hall sensor paddle wheel and generates a signal-k delta feed from it?

[Brian P]

You're thinking mast-mounted wind-speed sensor, I see. Do you have a model you would recommend? 

[Ric Morris]

The Airmar ST300 would be the obvious choices of paddlewheel transducer. The speed in 10ths of a KTS is just a matter of reporting the number of pulses received in the previous 2.5 seconds.

In terms of a MHU a replacement PCB for the B&G 508 would be a *very* interesting project. Failing that there's always the analogue signal from the more expensive but industry benchmark 213.

[Brian P]

Recording the pulses at the slow rates indicated by the Airmar specs is absolutely no trouble at all. To back up this claim, I will cite for this my project of quite some years ago: a photon counter and rate meter unit, which measured at up to 50kHz rates. 

** discussion veering off course from here: **

I have little experience with sailing boat instruments (I'm building instrumentation for dinghy sailing and charter boat supplements), but I do have a bit of experience in instrument building and engineering. Therefore, I was a little surprised to see the recommendations for the moving-part-based ST300 and B&G 508. In particular the STW sensor is preciously close to the hull and should give you inaccurate readings due to the (probably) laminar flow along it. 

Curiosity piqued, imagining Pitot-based speed meters, I dug a little further. I found much more robust-looking alternatives without moving parts in the ultrasonic sensors, for example the Airmar CS4500 for STW, and the Maretron WSO100 for wind, pressure, humidity and temperature. These can speak NMEAx out of the box, so would just connect to an iKommunicate for translation purposes. There are alternative sources for ultrasonic anemometers, but I haven't found an inexpensive one worth adapting just yet.

Building such ultrasonics yourself is not straightforward nor cheap, but it can be done: https://soldernerd.com/2016/09/07/ultrasonic-anemometer-part-28-new-hardware-tested/

Anyway, 0.02.

--

[Mike Bremford]

I have the CS4500. It puts out the same "paddlewheel" format - so you could use the same board to interface with an ultrasonic sensor or an old-school paddlewheel one.

I'd love to give some numbers but unfortunately I'm up on the hard while we try to fix a ****ing leak from the ****ing centerboard case. I may be there some time. So the only data I have from the CS4500 so far is that on a stationary boat with zero to minimal current underneath, the sensor is putting out pulses at a frequency equivalent to about half a knot. And there's considerable variation around that, from 0.1kt to 1kt. I'm hoping it will settle down when the boat is actually moving, but it's not quite the result I hoped for - say what you like about a paddlewheel, but it doesn't spin of it's own accord.

I'm not sure I'd have a crack at a DIY ultrasonic sensor - if you know of the CS4500 you may remember the fabled DST800, the ultrasonic speed/depth sensor in a combined unit. This was announced and then nothing, for years. People would hear whispers of it at boat shows then report them breathlessly to Panbo, like a unicorn sighting. I presume Airmar couldn't make it work, and they have a lot of experience here. Ultrasonics are not easy.

Also google "LCJ CV7", a nice masthead ultrasonic anemometer. Cheaper than the Maretron and still no moving parts (hooray). Outputs NMEA0183 at 1Hz or 4Hz.

A few other things have come up in this very interesting discussion which I feel the urge to comment on :-)

1. Re. the pros and cons of wifi over USB, any takers on how long before racing crews take out a dedicated hacker with a laptop to jam the other boats wifi signals or - much more fun - break the encryption and add a consistant 5° to their wind sensor output!

2. All this talk of clock signals, timing and the need to synchronise sensors to output at 10Hz. I understand the concept - you use compass heading, roll/pitch, wind speed/angle, mast rotation, and the delta between course-over-ground and course-through-water to derive true wind and current, for example. This is exactly what I'm trying to do too. But I'm not sure it can be done at the sort of frequency that requires clock synchronisation.

The sensor data is already being oversampled. For instance you can't derive "roll" or "pitch" instantaneously on a moving boat - you have to use a shifting calculation based on accelerometer/gyro/magnetometer over time (Kalman filter or Madwick algortithm). A magenetometer also leaps around like crazy, you have to use a running average to derive a heading. I presume this is what Ric meant when he talked of sampling at 100Hz. Same for wind speed - even with a 4Hz ultrasonic sensor you are getting a value which is not instantaneous, it's up to 250ms old, and more likely even that is a running average to smooth out the samples anyway.

My point is that I can't think of a single situation where we care about the time from the sensor or could rely upon it to do something useful. The readings come in from MHU, depth, compass, whatever, and when they arrive at the SignalK server and that's their timestamp. Depending on the unit the values may be instantanous, a running average over the last second, or last 5 seconds. Either way you make the calculations based on the time the data arrived. The only sensor that gives an absolute time is GPS, but so what? What are you going to do with it if it disagrees with the local clock? A man with one clock knows the time, a man with two is never sure. You can't "delta" two GPS values to try and gain accuracy, GPS doesn't work like that. 

Any latency from the wifi or computer scheduling algorithm will be several orders of magnitude less than the latency inherent in the design of most of the sensors I could imagine finding on a boat. The absolute numbers we see reported for windspeed, SOG, heading etc. give us a false sense of certainty - they are heavily filtered, and unless you built the sensor you won't know how heavily. You can't squeeze more accuracy out of them by analyzing timestamps.

I'm also unsure of the value of updates faster than 10Hz - a human can't usefully respond at that sort of speed unless the data is presented very carefully. I can see the value for things like the AC lifting foils, adjusting the angle of attack and so on, but I would expect that sort of update rate means it's all done internally to the control unit. For human consumption, I think even 10Hz is pushing it for most. However I'm prepared to be schooled on this, as I suspect the racegeek guys at least have forgotten more about racing than I will ever know.

Cheers... Mike

--

You received this message because you are subscribed to the Google Groups "Signal K" group.

To unsubscribe from this group and stop receiving emails from it, send an email to signalk+unsubscribe@googlegroups.com.

[rob...@42.co.nz]

In Freeboard I used the Peet Bros Anemometer http://www.peetbros.com/shop/category.aspx?catid=28

Yes moving parts :-)
The arduino code to decode the pulses to what seems accurate wind speed is in the freeboard project https://github.com/rob42/FreeboardPLC_v1_2/blob/master/Wind.cpp

Its a marine version, very easy to work with, seems robust, and Peet Bros is well regarded. Ive toyed with making a standalone wind sensor that puts out signalk, possibly based on a teensy lite or something. That could provide a decent masthead unit with sk output <USD150.

If some-one wants to run with it I will help, I just dont have the time to do it myself. 

Rob

[Ric Morris]

One of those "what's the question again?" things.

For me the question isn't "what would be the best/low scale available solution" it's:

What would be useful to the most people?

What's already installed?

What's readily available at a cost effective price?

What's the accepted norm?

...

What can be hacked to give a 10Hz sample rate?

And for the MHU what would make it wireless because it makes the install so much simpler/cheaper.

For transducers that means the ST300/DST650 triducer (by the way the CS4500 also has a pulse signal output) by a very very long way.

By all means plug a (wired) LJ into the Komunicate (or D10 ;) ) but...

If you're looking for a project that can make an impact in terms of signal-k adoption then a wireless MHU is still a genuine challenge in the market.

You only have 2 options at present. Raymarine Tactic and Garmin. The wired analogue signal from the B&G213 is still the industry benchmark. The 508 just happens to have exactly the same physical parts (the PCB is different and puts out n2k only at 1-2Hz).

The reason I suggest the 508 is that it can be picked up with out the cable for GBP300. That gives you all the hardware required for a custom PCB that's powered by the anemometer and sends raw AWA/AWS over, say, Zigbee 900 RF.

That's an achievable project for the community with out investment or through a kickstarter, something that would be useful and encompasses all the challenges that would give focus to any discussion of "signal-k low power".

...

The fastest comfortable refresh rate for a display is around 3Hz. So you can get away with a 10Hz top level calculation. That's only ever going to be as good as your slowest signal and there's a lot of filtering and calibration to be done in between (IMU kalmans etc. as mentioned). Hence why the raw signal frequency needed is closer 100Hz.

It's the raw signal systems want because it gives the choice of where to put the calculations, filtering and calibration.

[Brian P]

Sorry to hear about the troubles with the leak in your boat, the trimaran itself looks very nice! As for the CS4500, I know it is not supposed to be very accurate at low speeds, but should improve once the speeds go up. I didn't know the reading could be that much off, though. Perhaps it's measuring fish? If only we could trust the claims / specs of manufacturers at face value...

As for your point 1.: It is very illegal to run jamming broadcasting equipment, not to mention the hot water you'll run into with the racing committee if they find you're disabling other boat's sensors (apropos, you don't need anything else than a magnetron pointed in the right direction for this, works magic for all kinds of electronics). As for the "tampering" bit... There is an option to sign a SK message using an asymmetric key pair. That way you'll know for sure if the message you're receiving isn't authentic. Place for a signalk "signature"-field?

As for the timestamps, the only way I can see to easily embed accurate timestamps in sensor output that really need it, is to include a GPS module in them. Otherwise they have to be hardwired. 

[Brian P]

Hi Ric, 

I'm sorry it took so long, but I finally understand why you insisted on the high sampling rates: you want to do combinatorial processing and filtering of the signals afterward. I'll be curious to know what incoming sampling rates the current set-up can handle, so once OpenPlotter interprets incoming MQTT deltas, I'll try flooding the queue. 

I think it's a good project to make an SK MHU. Looking at the preferred models, however, I don't think I can justify buying a 600 - 1000 Euro MHU or a paddle wheel without having a boat, so such a project might have to be left to others. Rob's suggestion for the Peet Bros variant might just do, but I won't have any way of testing it under real-life conditions. Realistically, I'll be bareboat chartering for the coming ten to twelve years (unless I happen to luck into a bunch of money or a boat), and they usually don't appreciate it all that much if I start messing around with their MHUs. 

As for the Zigbee, unless there's an absolute need to move away from Wifi, I would recommend keeping the supported hardware spectrum limited. Limitations there means that you don't run into strange compatibility and support issues, and allow you to optimise that what you have. I live and breathe by the KISS principle. 

--

[Ric Morris]

Moving from wifi:
Power consumption for the transmitter in what needs to be a wireless solution
Bandwidth congestion in marinas
Broadcast over distance in the presence of lots of metal and carbon fiber

We have a 508 on order to take apart and cnd will share findings.

[Mike Bremford]

Thanks Brian. I'm very relieved to hear the CS4500 is not so good at low speeds, I hadn't read that but it means I haven't necessarily wasted my money. I had two of hours of logged data from all sensors, including the CS4500, but once I noticed the leak this was all forgotten about for a couple of days - when I got back to it, the log files had rolled an hour earlier. Grr. So no data to analyze. And cheers for your blog comments, I am just putting two and two together.

Cheers... Mike

[Brian P]

Ok, let's talk power. 

I finally managed to find time to rig up a small usb current meter. The MKR1000 + GPS module draws about 130 mA at 5V, so 0.65W. The GPS module itself draws about 25 mA of that total. 

Using the module's power management functions, the power draw can be reduced, according to examples here: http://forum.arduino.cc/index.php?topic=410640.0 

There, it's indicated that the wifi unit itself draws around 60 mA.

Note that the switch from Wifi to Zigbee will only affect that 60mA of the total, reducing it to about 25% of that value (source): 15 mA. That should bring the total power requirements of this module down from 130mA to 85 mA (0.43W), saving 0.22W. This is for the bare module, however, and will only get less impressive when combining a module with a sensor:

When put in perspective to the power requirements of the sensors themselves: 

An ST300 paddle wheel draws 100 mA at 12V (source): 1.2W, whereas an ST4500 uses almost 2W. Some ultrasonic MHU's like the CV7, however, use as little as 0.3 W (source). That means that even in this last best case scenario, the total system (sensor + module) would use 0.95W for a Wifi version, or 0.73W for a Zigbee version.

In conclusion, you're not getting a whole lot of power savings in return for the headaches involved with implementing Zigbee. But I'd be happy to see someone try this in practice and prove me wrong :). 

Cheers, 

B.

[Ric Morris]

130mAh a lot of power draw. Most likely some bad choices for DC/DC converters on the MKR1000 may be?

For a GPS comparison the Garmin 19X HVS draws max 60mAh at 12v. The uBlox N7 would be the starting point as a benchmark GPS chip.

The ST300 is a terrible piece of kit. The only thing is the ST300 has been around for a long time and is rebadged by just about everyone and installed as standard by just about every big boat builder. For a better understanding a Signet Marine paddle wheel draws 20mAh at 12v and will carry on working down to 3-5v.

For speed transducer and GPS a wireless connection is pointless. Access is straight forward enough to run a cable.

But you've crossed wires there Brian. The suggestions regarding power consumption is based on a requirement not to run a cable up to a mast head unit. Once you run a cable for power, you might as well run a cable for data. And if you're going to run a cable for data then you might as well forget WiFi and go for a straight forward, soon to be NMEA standard and much more reliable Ethernet connection.

Here's a comparison of BLE, ANT and ZigBee (WiFi doesn't even get a look in):

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/IWS20201320wireless20power20consumption.pdf

(Note that there are significant difference in wake up for ZigBee 2.4GHz and 900MHz, 900 being almost instantaneous)

To unsubscribe from this group and all its topics, send an email to signalk+u...@googlegroups.com.

[Brian P]

On Monday, October 31, 2016 at 4:52:14 PM UTC+1, Ric Morris wrote:

130mAh a lot of power draw. Most likely some bad choices for DC/DC converters on the MKR1000 may be?

 

Yes, I was a bit surprised about that too, as I could get an Arduino Mini to run using only a few mA with no trouble whatsoever. However, by switching on (just a single) low power mode on the MKR1000+GPS, it now draws an average of 60 mA at 5V (0.3W). I think I can improve that a bit more, but this already helps. It's now actually a bit better than that Garmin, which (according to the specs) draws 40 mA at 12V (0.48W). (http://www.fondriest.com/pdf/garmin_gps17xhvs_spec.pdf). I need to understand the sleep modes of the microcontroller and WiFi chip a bit better to get a bit lower power use in the coming weeks. 

But you've crossed wires there Brian. The suggestions regarding power consumption is based on a requirement not to run a cable up to a mast head unit. Once you run a cable for power, you might as well run a cable for data. And if you're going to run a cable for data then you might as well forget WiFi and go for a straight forward, soon to be NMEA standard and much more reliable Ethernet connection.

Once we get an open NMEA ethernet standard, it might be worth taking a look at. For my current sailing purposes I'll be running off rechargeable batteries like you suggest, and have a minimal set of easily-placed sensors connected to a low-mounted raspberry Pi. For future wiring in my imaginary boat, however, I consider it to be much easier to wire only a power bus through a boat or up a mast as opposed to power + various flavours of data. I like the power+data bus in NMEA2k, but I don't like its closed specification and interoperability issues when mixing vendors. I will use that neat NMEA2k connector for power, though. 

Here's a comparison of BLE, ANT and ZigBee (WiFi doesn't even get a look in):

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/IWS20201320wireless20power20consumption.pdf

(Note that there are significant difference in wake up for ZigBee 2.4GHz and 900MHz, 900 being almost instantaneous)

I like the comparison, but note that they send a positively minuscule packet, and only compare the power requirements of the radio module. They ignore the power drawn by the sensor and the Cortex M0 microcontroller, both of which can be in the tens of mA. My worry is that even if you save close to 100% of power on the radio module, you'll still have quite a bit of power requirement from the other components in your complete sensor. Therefore, just switching radio might not save you as much power as you'd imagine. I think it is worth checking alternatives to WiFi eventually, but it's not one of my current worries. Added to that is the fear that by going to one such radio alternatives, you end up in another bunch of issues specific to that type of radio and comms, i.e., as Mike said: you swap one set of problems for another. 

[Ric Morris]

A CORTEX M0 is serious over kill for a MHU unless you're putting an IMU up there.

Given ZigBee 900Mhz is essentially an analogue signal the objective would be to send the analogue value for the AWA and may be even the ticks them selves from the hall sensor for AWS to a station in a more convenient location.

The challenge is to send the signal from the power generated by the anemometer. Tacktic (MICRONET RF signal) and Garmin (ANT) both manage it some how (caveat that the Garmin may have solar back up as well, which may be because there is calibration going on in the mast head unit)

[Brian P]

That would be interesting. I can imagine something like the self-powered MHU, and a "base station" for receiving the ZigBee messages, translating it to SK and forwarding it to the server. 

The MHU would require a supercapacitor, probably an Arduino Mini and some way to harvest energy from an anemometer. That last bit is something I'm not sure about if it can be done with the B&G anemometers you suggested, if they are reed relay-based, there's no energy coming from them. I'm not sure hall-effect transistors output anything either. Otherwise, a tiny solar cell would be required, requiring a redesign of the anemometer casing, etc. etc. Not a small project.

--

You received this message because you are subscribed to a topic in the Google Groups "Signal K" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/signalk/oGtUONASVOE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to signalk+unsubscribe@googlegroups.com.