Sample JSON messages for Engine Diagnostic data

[Robert Ryan]

EngineMonitor is an open source project for older marine engines that lack decent electronic monitoring of vitals such as EGT, raw water temps, RPMs, and etc.  It currently has 8 sensors and publishes the data over WiFi and a hard-wired 16 X 2 LCD display that lives in a small enclosure mounted on the dashboard.  It uses Arduino to collect the data and drive the LCD monitor.  A Raspberry PI is serially connected to the Arduino to receive sensor values and publish them via a web server to clients such as iPads or any tablet with a browser.  The project lives in BitBucket at https://bitbucket.org/R_P_Ryan/enginemonitor/wiki/Home.  The project includes all the sensors and there is no integration with other marine electronics such as depth sounders or chartplotters.

SignalK seems like an ideal format to publish the data in.  I would like to take the reference implementation and expand it to include engine information and publish everything over the RPI's web server.  However I can't figure out how to structure the JSON messages and would rather avoid websockets for the first iteration.  I've looked for sample JSON messages for sensor values such as "Raw Water Temperature" and "Exhaust Gas Temperature"

I've installed the reference implementations: instrumentpanel-master, signalk-js-client, and simpleguages to better understand SignalK.  However it doesn't seem obvious where the JSON strings are stored or what format the JSON needs to be in when returned from the RPI.

Currently the web server returns an HTML formatted table with the diagnostic data and a JSON string with the same data:

{ 

"Raw Water In" : 58.1, 

"Raw water Out" : 82.2, 

"Coolant Entering Heat Exchanger" : 173.3, 

"Coolant Exiting Heat Exchanger" : 132.4, 

"Cylinder Head" : 182.5, 

"Engine Room" : 88.6, 

"Oil " : 178.7, 

"Exhaust Gas" : 473.8, 

"RPM" : 2400

}

It seems with some simple changes to the structure of the JSON it could be SignalK compliant and readable by SignalK-compliant displays or web pages.

Questions:

1. Is this application appropriate for SignalK?

2. Where can I find the data dictionary for these types of values?

3. Are there any SignalK apps for engine diagnostic data?

Thanks,

Robert

[Teppo Kurki]

On Sun, Jan 24, 2016 at 4:57 AM, Robert Ryan <rober...@gmail.com> wrote:

EngineMonitor is an open source project for older marine engines

Now yours is one interesting and well documented project - thanks for sharing! And a great application for using SK. 

SignalK seems like an ideal format to publish the data in.  I would like to take the reference implementation and expand it to include engine information and publish everything over the RPI's web server.

The schema was just recently expanded for more engine parameter coverage. Please see

https://github.com/SignalK/specification/blob/master/schemas/groups/propulsion.json

https://github.com/SignalK/specification/blob/master/test/data/propulsion.json

The latter is sample data in full format, but what you are looking for is delta like this:

https://gist.github.com/tkurki/97a175c09f690158226a

 

 However I can't figure out how to structure the JSON messages and would rather avoid websockets for the first iteration.  I've looked for sample JSON messages for sensor values such as "Raw Water Temperature" and "Exhaust Gas Temperature" 

I've installed the reference implementations: instrumentpanel-master, signalk-js-client, and simpleguages to better understand SignalK.  However it doesn't seem obvious where the JSON strings are stored or what format the JSON needs to be in when returned from the RPI.

Currently the web server returns an HTML formatted table with the diagnostic data and a JSON string with the same data:

{ 

"Raw Water In" : 58.1, 

"Raw water Out" : 82.2, 

"Coolant Entering Heat Exchanger" : 173.3, 

"Coolant Exiting Heat Exchanger" : 132.4, 

"Cylinder Head" : 182.5, 

"Engine Room" : 88.6, 

"Oil " : 178.7, 

"Exhaust Gas" : 473.8, 

"RPM" : 2400

}

It seems with some simple changes to the structure of the JSON it could be SignalK compliant and readable by SignalK-compliant displays or web pages.

Questions:

1. Is this application appropriate for SignalK?

Certainly!

 

2. Where can I find the data dictionary for these types of values?

The schema is the definite reference. Some notes related to your data

- there is only one entry for coolantTemperature

- raw water and cylinder head figures are missing

3. Are there any SignalK apps for engine diagnostic data?

InstrumentPanel should be able to provide generic gauges, but I'm not aware of any others yet. Using stuff from SimpleGauges to suit your data is pretty straightforward.

Reading between the lines it appears that you already have the data on the RPi. 

With me being familiar with the node server if I were you I would just plug the data directly into the node server: if the Arduino would send line-oriented SK delta json over serial line the server could just read from the serial port and that's it. This way the system would be "push": whenever the Arduino pushes new data the server would get it and it would push it over WebSocket to any connected clients.

https://github.com/SignalK/signalk-server-node/blob/serialport-provider/providers/serialport.js

Another approach would be to use the command line / executor provider, which spawns whatever command you want that produces SK delta. This is for the case where you want to use Python or some other language to talk to the Arduino. 

https://github.com/SignalK/signalk-server-node/blob/master/settings/commandline-provider-settings.json

You could also run a script (as in the settings file above)  periodically to pull the data from your current http server, convert it to SK delta and output to stdout for the server.

By using either node or java server you would get the SK basic protocol with discovery, rest api and WebSocket api out of the box and only need to implement shuffling your own data into the server.

 Let me know if you can proceed with this info - I'd be more than happy to help, as your application is really interesting and I'm interested in getting this on my boat. 

You might also want to join our Slack discussion forum at http://slack-invite.signalk.org/. Ongoing discussion is easier there, but lacks public searchability unfortunately.

[Robert Ryan]

Thank you for the very well documented reply.  This gives me a lot to work with.  I've worked with websockets in the past and found some issues with them dropping the connection, in which case it drops down to long polling.  Pushing data would be better from an architectural perspective because the Arduino takes about a second to retrieve all the temperatures from the DS18B20 sensors.  During this time it won't respond to requests for data.  With the push approach it would not get requests, instead it would send to values over the serial wires to the RPI.

This gives me a lot to work with.  I'll let the group know how it goes when I have something.

Thanks,

Robert

[Teppo Kurki]

On Sun, Jan 24, 2016 at 8:36 PM, Robert Ryan <rober...@gmail.com> wrote:

Thank you for the very well documented reply.  This gives me a lot to work with.  I've worked with websockets in the past and found some issues with them dropping the connection, in which case it drops down to long polling.  

That was probably related to some library you were using - plain ws is just that, plain ws, and if you use the signalk js client lib with the node server you get automatic reconnect with exponential backoff. Useful for example when your phone wakes up.

Looking forward to the results - let me know if you have questions.

[Adam]

Hi Robert,

Very nice project, I can see this being very valuable for many boats without electronically controlled engines.  My old boat needed exactly this although it was gas (petrol), would this project work with this type of engine?   My present boat is electronic so data is present via Canbus and NMEA 2000.  In any case I'll post it on my blog's resource page as a signal K project.  Please keep us updated as you progress.

thx, Adam

----

www.signalkool.com

[Robert Ryan]

Thanks for the compliment!  The system should theoretically work on petrol, but the high RF noise could cause problems.  Older diesels are all mechanical and can run without a battery or alternator once started.

--
You received this message because you are subscribed to a topic in the Google Groups "Signal K" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/signalk/7DCpiWnpDNM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to signalk+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rusty Coan]

Robert,

Your project is VERY similar to what I'm working on.  Since being introduced to Signal K my project goals have shifted to simply reading engine data and pushing the data to signal K.  I'm in the very early stages, still waiting on boards and sensors to arrive.  It seems like your project is geared toward a single engine application, whereas my project would be geared toward a sportfish with twin diesels and a diesel generator.  My project will also include fuel flow meters and a few extra engine parameters.

I'm considering taking a fork of your project and re-working it to suit my needs.  It will most likely be near the end of Feb before I get all my parts and pieces in and that is when I'll begin.

Are you ok with me taking a fork and possibly collaborating with you on this?

[Robert Ryan]

Absolutely take a fork. I'm going to start committing version 2, which has the RPI integrated.  For twins or gensets I'm thinking there would be an Arduino per engine, then each Arduino would talk serial to a single RPI. On the wiki there's a page for version 2 that shows the html gauges.

I have several temp sensors based on the DS18B20 chip. Basically I've drilled out s.s. bolts and embedded the chips inside with epoxy, then run a lead to the Arduino. I ran some comparisons of just epoxying a sensor chip onto the outside of a metal pipe to measure the water temp inside and it was like 40f lower than the actual fluid temp. So I'm convinced that the chip must be housed in a metal probe that is insulated from the surrounding metal to get an accurate reading. In other words the fluid can be 40f hotter inside the pipe than the surface temp of the pipe. So if the sensor is inside a metal plug screwed into a bung on the pipe is probably reading too low. Ideally the sensor would be immersed in the fluid and insulated from the metal pipes by something non metal.

I'm also going to add boost and the fuel flow meters. My setup requires two fuel meters per engine since there's a fuel return line to the tank from the pump.

With the RPI there are so many possibilities. I'm thinking about adding a music player, and reference data like tide tables, rules of the road, etc.  Also add in a database with some web forms for oil change, fuel fill, and other maintenance.

Would be nice to have a data base of fishes in season on RPI as well.

I also want to be able to put the RPI in WiFi AP mode so a WiFi router is not necessary, but then also connect a gateway to any local hotspots like at the yacht club.

I'd like to add a hard drive with movies for the kids and stream via chrome cast to the flat screen in the main cabin.

So many possibilities. Wishing I was retired and wealthy to spend more time on this.

What year make and model engines do you have?

BTW, we can change the bit bucket name so it's not my name, that way we can share credit. I got the domain name zencaptain.com and was going to use it to publicise the project. So we could change the bit bucket url to anything, or use zencaptain.

What other sensors where you thinking? I was thinking about adding the liquid level sensor strips from spark fun to show levels in the diesel, water, and water tanks. Would be nice to put strain gauges under the tanks since they're so cheap and incredibly accurate, you could show flow rate with then but the retrofit is too intense.

--

[Rusty Coan]

My engines are Twin CAT 3208 355hp Turbo's.  The genset is a Westerbeke 7.6 BTD.  The boat is a 38 Ocean Super Sport.

I'm liking your ideas a LOT!!!  I'm going to be looking over your code and feeling my way around.

One thing I have under way is to implement Satellite Text communications.  We do multi-day offshore trips where we are 140mi+ offshore at times and far away from cell service and I always use satellite text message services to stay in touch with the wife and family.  In the past I've used the inReach from DeLorme.  To be honest I've not been happy with it and it is NOT cheap.  I think our service is somewhere in the $60/m range.   In the future I hope to use the RockBlock satellite text messenger bare connected to an arduino or maybe to the RPI.  I ordered one of these guys today, have not located a marine antenna yet that I like, but there are many iridium marine antennas available. https://www.rock7.com/shop-product-detail?productId=46

Basically I want the sat texting to be fully integrated into my system so you can send/receive sat texts from all displays.  Also, the sat module could text location information to a web based server so that family and friends could track your course (a nice feature of the inReach).  When you are at home and the boat is in the slip it could also text daily updates, although that might be cost prohibitive and a 3g cell card may be a better 'daily update' solution.

I've already started mocking up what I'm considering for a gauge cluster on paper.  I don't think I'm going to be using any traditional 'gauge' designs b/c they just flat out consume too much space.  But rather design the UI so that it is clean, modern, intuitive and using colors (flashing red, red, yellow, green) so you can, at a glance, know if anything needs attention.

I also want to have one gauge display layout that we can throw up on the cabin TV 32" when offshore.  That way, when I'm in the cabin and someone else is driving I can monitor.

Instead of doing one arduino per engine I'm thinking of separating them by type of sensor, but I could easily be convinced otherwise lol.  So one Mega for the flow sensors.  One for one set of sensors and maybe a third board for other sensors (if necessary).  This way if someone just wanted the 'boat sensor' package they could just get it, then later add fuel flow, or sat text...etc.

For fluid levels I was just going to use the existing sending units in the tanks, but if there's a better way I'm definitely game.

[Robert Ryan]

Did you see the arduino-rpi layout at https://bitbucket.org/R_P_Ryan/enginemonitor/wiki/Version_2_0?

What year are the CATs?  Do they have an electronic diagnostic socket?

Have you seen sigFox?  Apparently you can send very small 12 bit messages for hundreds of miles for very very cheap.  there are shields for Arduino.

Would be nice to have a way to merge your SignalK work into the RPi side of things.

[Rusty Coan]

Here are a few comps of the gauge design I'm thinking of implementing.  As I said before, not really any gauges going on, just clean data.  Would probably do a daytime and nightime view, one being dark background the other being a lighter background.

This is a single iPad sized device as the display.  The waterproof eReaders are my target at the moment I think.

Here is another version of single display in portrait.  Just a slightly different layout.

This is probably what I'll be shooting for.  Two devices, portrait, side by side mimicing a glass helm.

These are all just photoshop mockups I just threw together.  I'm sure some tweaks need to be made and I'm certain I have left off possible data....  Open to ideas good/bad/ugly.

What data format is your arduino passing to the Pi? Or I guess, how is the Pi getting the data?  I understand it's a serial connection

[Rusty Coan]

My Cats are circa 1984.  They have been rebuilt recently.  I'm not aware of any digital port on them anywhere. Pretty sure everything about them is analog.

[Rusty Coan]

Added a few data fields after I posted last night.  Added to this version are Air Filter and Fuel Filter, and a couple of UI improvments.  Air filter will be measured with a vacuum sensor between the air filter and the turbo.  Fuel filter would be measured with a fluid vacuum sensor between the filters and the engine. Also added the yellow colored dot to signify you are in a warning area

Anyone know of a sensor that would measure fluid vacuum for the fuel filters?

[Rusty Coan]

Pic didn't attach to last post.

[Teppo Kurki]

Nice gauge design. Do you have any plans as to what tools you're going to use for the gauge interface?

[Rusty Coan]

Not exactly sure what you are referring to by tools.  Could easily throw together an iOS app for this, it may even work on Android.  The first version will most likely make an attempt at a dynamic html page layout that adjusts to the viewport size of the device.  This way the design could be functional in handhelds (smart phone) or on tablets (iPad, android, eReader) or on 32" cabin Tv's!

[Teppo Kurki]

On Wed, Feb 3, 2016 at 6:25 PM, Rusty Coan <rwc...@gmail.com> wrote:

Not exactly sure what you are referring to by tools.  Could easily throw together an iOS app for this, it may even work on Android.  The first version will most likely make an attempt at a dynamic html page layout that adjusts to the viewport size of the device.  This way the design could be functional in handhelds (smart phone) or on tablets (iPad, android, eReader) or on 32" cabin Tv's!

Thanks, you interpreted my question correctly.

I was trying to see in my head the cell outlines and what it would take to implement something like this in InstrumentPanel (https://github.com/SignalK/instrumentpanel). You could define a css palette that hides the cell outlines when not in edit mode and there isn't anything particularly complex in your layout, but getting the details polished (right proportions etc) will take time. 

This is an area that could use a joint effort I think. IP is not something I feel particularly proud of, function over form eh....

[Rusty Coan]

I have some guys that are absolutely fantastic with dynamic html layout.  I have my hands in a little bit of everything, but smart enough to know when someone is smarter than me in a certain area!  ha!  Once I get the layout where we are comfortable with it I will have one of my html guys take a look and advise.  Then I'll hand it off to an actual designer for any final cleanup/tweaking, then hand it back to the html/css/javscript magician to make everything look good in all displays.

My goal is to ONLY use design elements that are functional and add to the layout.  Unnecessary lines/borders/buttons/graphics/gauges...etc are going to be left out.  Simple, clean, easy to use, easy to read.

I think the initial inclination for everyone is to re-create the classic 'gauge' design with a nice decorative bezel.  But do we really need to do that?  All that stuff (the look, size and shape) was a form of function in it's own right.  Gauges needed to be round b/c they had needles to point to the values.  The bezels were in place b/c they had to somehow mount them to a dash.  The round shape of them alone absolutely eats space.  The average boat dash has a TON of unusable space.  On top of that, unless you are the normal captain of the boat, finding the right gauge to check something can be an absolute nightmare.

With the digital format we are working with we are not constrained by needles, and bezels... we do not HAVE to have them, like the traditional gauge does.  So why do we need to use them in our designs?

I'm not 100% against graphical representations, in my design I used bars for tank levels.  I do think that graphical representations of the data ONLY need to be done when it is beneficial to the user and makes the data easier to use/absorb.  RPM's may need a similar graphical element as well, I'm not sure.  For my boat, I have engine synchronizers, so I'm not manually matching rpm's, I just flip a switch. 

I was kinda envisioning a UI editor/configurator for these pages, basically a list of checkboxes.  For instance, if your boat doesn't have a freezer, then no reason for that spec to show up...  So you could essentially edit and select what values show up on your version of the display.  The Garmin 7616xsv on my boat now has some pretty neat configuration options where you can customize your display layout.  I'd like to mimic that within reason.

[Teppo Kurki]

I was kinda envisioning a UI editor/configurator for these pages, basically a list of checkboxes.  For instance, if your boat doesn't have a freezer, then no reason for that spec to show up...  So you could essentially edit and select what values show up on your version of the display.  The Garmin 7616xsv on my boat now has some pretty neat configuration options where you can customize your display layout.  I'd like to mimic that within reason.

With InstrumentPanel I've chosen the logic that there is no prefined set of gauges / items, but as IP discovers new items in the data stream it automatically adds a new gauge to the layout. Once it displays all your available data you can then drag & resize the gauges and there's the option to uncheck any data items you don't want to see. 

Some data items have a special gauge design, like wind gauge and compass. I hope to have also alternative designs that you can pick from. Day/night palettes and alternative palettes in general are somewhere on the backlog as well.

If anybody wants to have a particular gauge design in IP I'd be more than happy to help out in getting it working.

I'm anxiously looking forward to what people come up with now that data is starting to be a lot more accessible from a browser (and other applications).

[Robert Ryan]

This is pretty awesome.  I'll definitely have to put some more time back into the EngineMonitor.  Version 1.0 is stable but not networked.  Version 2.0 adds the RPi and puts it onto the network.  

Node has been around so long it's about time I learned it.  20 years ago if you told me javascript would rule the server side I'd laugh my ass off at you.

[Teppo Kurki]

On Sun, Jan 24, 2016 at 8:36 PM, Robert Ryan <rober...@gmail.com> wrote:

Thank you for the very well documented reply.  This gives me a lot to work with.  I've worked with websockets in the past and found some issues with them dropping the connection, in which case it drops down to long polling.  Pushing data would be better from an architectural perspective because the Arduino takes about a second to retrieve all the temperatures from the DS18B20 sensors.  During this time it won't respond to requests for data.  With the push approach it would not get requests, instead it would send to values over the serial wires to the RPI.

This gives me a lot to work with.  I'll let the group know how it goes when I have something.

Just checking up on this - were you able to proceed with your Signal K application?