NMEA (GPS) log utility

[Fabian Tollenaar]

I've written a very small CLI utility that might be useful (I'm sure everybody already has a solution for this, but I thought I'd share it anyway). It logs NMEA sentences to a file "gps.nmea", which can later be used to playback the NMEA data on a virtual serial port. It's been tested with a GPS, but since it simply logs the raw input I'm sure it would work with any NMEA0183-emitting device. The utility that plays back the nmea data on the virtual serial port is forthcoming. 

https://github.com/fabdrol/nmea-gps-logger

I've written this first and foremost in order to get some more insight in working with serial ports on unix, which is why I probably reinvented a wheel. 

[Teppo Kurki]

Gpsd provides playback utility, can't recall the exact mechanism.

Gpsd provides also json data. I chose not to use it,  because I wanted to proxy the raw data out to udp as well. Gpsd can provide bith raw and parsed data, so I may change around and use gpsd json. 

Gpsd json to Signal K data model would be wortwhile and gpsd AIS json a wheel we could use instead of inventing a new one. 

Worth reading: http://gpsd.berlios.de/protocol-evolution.html

--
You received this message because you are subscribed to the Google Groups "Signal K" group.
To unsubscribe from this group and stop receiving emails from it, send an email to signalk+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tim Mathews]

GPSD support would be nice. Right now I'm getting live GPS data from a Garmin 17x, but it's in my basement so reception is spotty and I don't have a solution for getting a clear view of the sky that doesn't involve a hammer drill and me on a ladder.

[Kees Verruijt]

We should definitely re-use as much as possible of the GPSD format, like field names for AIS and GPS data.

[Fabian Tollenaar]

Do we have a sample of the GPSd format? I tried using GPSd with my GPS, but it didn't really work. I found it much easier to just connect to the serial port directly. 

Attached a screenshot of a basic Consumer I've written. It consumes Signal K data and shows that on a map. It tracks the route as well - as long as it's open. However, since I'm in the office all day that track is not very exciting ;)

Cheers, Fabian

Ps, as I said, the logger was mainly an exercise. I'm reinventing the wheel on the simulator for that exact same reason. Atm I'm using the utility socat to create a virtual serial port, so I can send the file with NMEA data back into a serial port and use the same Provider as I'd use with a real GPS to supply data to the server. 

Op donderdag 3 april 2014 15:47:40 UTC+2 schreef kees:

[rob...@42.co.nz]

Not sure if you want this format (http://www.catb.org/gpsd/gpsd_json.html), its backwards to a JSON based 3char style dislocated message concept. I think we are better to create a mapper/convertor aka NMEA stuff. Better yet, lets invite gpsd to support signal k.

[Kees Verruijt]

Sent from my iPad

> On 3 apr. 2014, at 23:10, "rob...@42.co.nz" <rob...@42.co.nz> wrote:
>
> Not sure if you want this format (http://www.catb.org/gpsd/gpsd_json.html), its backwards to a JSON based 3char style dislocated message concept. I think we are better to create a mapper/convertor aka NMEA stuff.

I looked at the actual format now and I have to say I agree, it is obtuse.

> Better yet, lets invite gpsd to support signal k.

Maybe, once we get something that is good.

That reminds me, how are we going to do lookup tables, eg enumerate values? We could just transfer the numerical value and come up with a URL path that can be used to retrieve the translated value. We ought to take I18N into account from the start.

So if the message contains "x"=y then the client can ask ....?lookup=x to get all descriptions in en-US or ...?lookup=x&value=y&lang=se to get the Swedish description for just value y. Response in json, of course.

[Fabian Tollenaar | Starting Point]

A parser for the GPSd JSON output wouldn’t be too hard to build. jamesp/node-mea module outputs a custom JSON format, I have a small utility that translates it to Signal K (although it still needs a lot of work). It could easily be adapted to translate GPSd JSON to Signal K. Although it’s written in node, it would not be that hard to make it work like a cli util that one can pipe after GPSd. 

 

[Fabian Tollenaar | Starting Point]

If somebody has a dump of GPSd data, I’ll transform it and put a version on Github, so we can work from there. Might be useful for other devs who want to play around with Signal K.

On 3 Apr 2014, at 23:31, Kees Verruijt <ke...@verruijt.net> wrote:

[Teppo Kurki]

On Fri, Apr 4, 2014 at 10:27 AM, Fabian Tollenaar | Starting Point <fab...@starting-point.nl> wrote:

A parser for the GPSd JSON output wouldn’t be too hard to build. jamesp/node-mea module outputs a custom JSON format, I have a small utility that translates it to Signal K (although it still needs a lot of work). It could easily be adapted to translate GPSd JSON to Signal K. Although it’s written in node, it would not be that hard to make it work like a cli util that one can pipe after GPSd. 

I don't think GPSd supports non-gps NMEA sentences, so it is not really a universal solution for NMEA.

Navgauge has no AIS support, but if I get around to doing something about it my plan is to use GPSd JSON output for that.  It already does the more complex AIS message parsing.

Fabian: You've maybe noticed that Navgauge uses node-nmea and I've added minimal support for a bunch of sentences to it. I also added nmea encoding to support N2K => Navgauge => NMEA over udp => iNavX.

I was just thinking the other night about what would be the best way to add SignalK support to node-nmea. Are you doing it as a separate module or within node-nmea?

 

[Fabian Tollenaar]

Teppo: 

I'm using a separate "module" (more like, a single file) that does some basic translation. GPS is simple, so I merely check what kind of message node-nmea has parsed (i.e., nav-info, track-info) and then taking out the necessary information, transforming it as necessary. At the moment this proces is still very incomplete (for instance, dates are not parsed properly) - I wanted to get my Provider-Server-Consumer logic up and running before perfecting. 

That said, come to think about it, I don't think that's the best way to go. In my opinion, it would be best to build a 'standard' parser for NMEA 0183 to Signal K (directly) without relying on any other translations (like GPSd JSON or node-nmea's output) - as they could change at any time and they just decrease performance (I mean, Parsing to JSON, and then translating to different JSON.. that already sounds slow). As the NMEA0182 protocol is huge and badly documented, the parser should be build in modular fashion.

A parser could work like this: 

1. NMEA 0183 sentence is fed into the parser - either programmatically or via StdIn (e.g. getnmeafromgps --raw /dev/ttyUSB010101 | signalk-nmea0183 | signalk-utp-server).
   
2. For each line of incoming NMEA data, the parser checks if there is a plugin available that can handle that particular line of NMEA data.
1. If there is not, the line is discarded (throwing an error would not be useful, as there is a lot of data coming in and the parser would still work perfectly for other lines)
3. The plugin is called to handle that particular line of data
1. The plugin parses the data into sub-object of a full signalK object (e.g. "navigation": {} or even "location": {} if the line is coming from a GPS that doesn't give COG and SOG). 
2. The parser receives the sub-object and puts it in the right place in the SignalK object. If the data is already present (e.g., GPS data from two different sources) the data with the latest timestamp is used - in favour of the other data. 
4. The parser outputs a full Signal K object including all lines that could be parsed. Unparsed lines are simply ignored. 

When the logic that checks the plugins and folds their results back into a signal k object is build, adding parsers for whatever line of NMEA data we can find could become a true community effort. 

I haven't really explored the source of node-nmea yet, so I have no idea how modular that is and if we could simply fork it to get started. If not, and if everyone agrees that we need a direct NMEA0183 to Signal K parser, I'm willing to start work on this and get the ball rolling. Teppo, what do you think?

Op vrijdag 4 april 2014 15:24:45 UTC+2 schreef Teppo Kurki:

[Kees Verruijt]

On 04 Apr 2014, at 16:43, Fabian Tollenaar <fab...@starting-point.nl> wrote:

Teppo: 

I'm using a separate "module" (more like, a single file) that does some basic translation. GPS is simple, so I merely check what kind of message node-nmea has parsed (i.e., nav-info, track-info) and then taking out the necessary information, transforming it as necessary. At the moment this proces is still very incomplete (for instance, dates are not parsed properly) - I wanted to get my Provider-Server-Consumer logic up and running before perfecting. 

That said, come to think about it, I don't think that's the best way to go. In my opinion, it would be best to build a 'standard' parser for NMEA 0183 to Signal K (directly) without relying on any other translations (like GPSd JSON or node-nmea's output) - as they could change at any time and they just decrease performance (I mean, Parsing to JSON, and then translating to different JSON.. that already sounds slow). As the NMEA0182 protocol is huge and badly documented, the parser should be build in modular fashion.

A parser could work like this: 

1. NMEA 0183 sentence is fed into the parser - either programmatically or via StdIn (e.g. getnmeafromgps --raw /dev/ttyUSB010101 | signalk-nmea0183 | signalk-utp-server).
   
2. For each line of incoming NMEA data, the parser checks if there is a plugin available that can handle that particular line of NMEA data.
1. If there is not, the line is discarded (throwing an error would not be useful, as there is a lot of data coming in and the parser would still work perfectly for other lines)
3. The plugin is called to handle that particular line of data
1. The plugin parses the data into sub-object of a full signalK object (e.g. "navigation": {} or even "location": {} if the line is coming from a GPS that doesn't give COG and SOG). 
2. The parser receives the sub-object and puts it in the right place in the SignalK object. If the data is already present (e.g., GPS data from two different sources) the data with the latest timestamp is used - in favour of the other data. 
4. The parser outputs a full Signal K object including all lines that could be parsed. Unparsed lines are simply ignored. 

When the logic that checks the plugins and folds their results back into a signal k object is build, adding parsers for whatever line of NMEA data we can find could become a true community effort. 

I haven't really explored the source of node-nmea yet, so I have no idea how modular that is and if we could simply fork it to get started. If not, and if everyone agrees that we need a direct NMEA0183 to Signal K parser, I'm willing to start work on this and get the ball rolling. Teppo, what do you think?

Definitely looks really worthwhile. I also like your thinking of making this a UNIX style filter. This allows extremely simple reuse.

What language do you intend to use?

[Fabian Tollenaar | Starting Point]

I’d write it in Node (javascript), preferably. Node.js makes it extremely easy build the parser in such a way that it can work stand alone (as a CLI utility), executed from another process (regardless of the language) and (added bones) in the browser.. 

Something else I was thinking of, but I’m not sure if it adds value or makes anything any easier. Would it be an idea to write a little utility (NPM module for parsers written in Node) that parser plugins can require in order to automatically be up-to-date with the specification of Signal K? I’m thinking of something like this: 

- Plugins asks the K utility for the Schema of the K sub-object “environmental”. 

- An object defining all properties and their types, defaults etc is returned (not dissimilar to what most ORM/ODB libraries do - e.g. http://mongoosejs.com/docs/guide.html). 

That way, we ensure (to a point) that a parser plugin’s output is always up to date with the spec, regardless of wether the author updates it regularly. Of course, this would only work for incremental (backwards compatible) spec updates, bigger, feature-breaking updates would require some work by the dev. 

Btw, something completely and utterly unrelated to this: I noticed in the current Signal K spec that we’re using speedOverGround for SOG but cogTrue and cogMagnetic for COG. Isn’t it nicer to either choose sog, cogTrue & cogMagnetic OR speedOverGround, courseOverGroundTrue & courseOverGroundMagnetic?

[Kees Verruijt]

On 04 Apr 2014, at 17:25, Fabian Tollenaar | Starting Point <fab...@starting-point.nl> wrote:

I’d write it in Node (javascript), preferably. Node.js makes it extremely easy build the parser in such a way that it can work stand alone (as a CLI utility), executed from another process (regardless of the language) and (added bones) in the browser..

Cool.

Something else I was thinking of, but I’m not sure if it adds value or makes anything any easier. Would it be an idea to write a little utility (NPM module for parsers written in Node) that parser plugins can require in order to automatically be up-to-date with the specification of Signal K? I’m thinking of something like this: 

- Plugins asks the K utility for the Schema of the K sub-object “environmental”. 

- An object defining all properties and their types, defaults etc is returned (not dissimilar to what most ORM/ODB libraries do - e.g. http://mongoosejs.com/docs/guide.html). 

That way, we ensure (to a point) that a parser plugin’s output is always up to date with the spec, regardless of wether the author updates it regularly. Of course, this would only work for incremental (backwards compatible) spec updates, bigger, feature-breaking updates would require some work by the dev. 

Not sure whether this would work. If an attribute changes a name (which we will have a lot of in the beginning, see your own suggestion of COG/courseOverGround) then the author’s code that sets an attribute would break, or not set the correct attribute anymore.

What I do see making sense is where we build our utilities such that they do not include Signal K specifications in their own source, but always build from the Signal K git.

Then a typical workflow could be:

cd signalk; git pull; cd myownproject; git pull; make

With a compiled language the make would break if I used any attributes of the model that have been reworked, forcing me to update my code. Not sure how this would work with node.js or other interpreted languages though.

[Fabian Tollenaar | Starting Point]

I guess you’re right.. you'd end up doing a lot of useless double work. E.g.: you create a property (X) to describe a property (x), allowing the property x to change whilst X always stays the same. It works, but it’s unnecessary.

Example (stupid, and useless):

var Schema = {

navigation: {

groupDescriptor: {

name: “navigation”, 

type: “object”,

contains: [“gps”, “compass”, “ais” /*, “whatever”*/]

},

cogTrue: {

name: “cogTrue”,

type: “float”,

value: 0.0,

unit: “knots"

}

}

// ...

};

function schemaFor(group, measure) {

if(typeof Schema[group] === “undefined”) return;

if(typeof Schema[group][measure] === “undefined”) return;

return Schema[group][measure];

}

It was just a thought. Sorry for spamming your mailboxes with garbage.

[Teppo Kurki]

1. 
   NMEA 0183 sentence is fed into the parser - either programmatically or via StdIn (e.g. getnmeafromgps --raw /dev/ttyUSB010101 | signalk-nmea0183 | signalk-utp-server).
   
2. For each line of incoming NMEA data, the parser checks if there is a plugin available that can handle that particular line of NMEA data.
1. If there is not, the line is discarded (throwing an error would not be useful, as there is a lot of data coming in and the parser would still work perfectly for other lines)
3. The plugin is called to handle that particular line of data
1. The plugin parses the data into sub-object of a full signalK object (e.g. "navigation": {} or even "location": {} if the line is coming from a GPS that doesn't give COG and SOG). 
2. The parser receives the sub-object and puts it in the right place in the SignalK object. If the data is already present (e.g., GPS data from two different sources) the data with the latest timestamp is used - in favour of the other data. 
4. The parser outputs a full Signal K object including all lines that could be parsed. Unparsed lines are simply ignored. 

When the logic that checks the plugins and folds their results back into a signal k object is build, adding parsers for whatever line of NMEA data we can find could become a true community effort. 

I haven't really explored the source of node-nmea yet, so I have no idea how modular that is and if we could simply fork it to get started. If not, and if everyone agrees that we need a direct NMEA0183 to Signal K parser, I'm willing to start work on this and get the ball rolling. Teppo, what do you think?

You pretty much described how node-nmea works, expect that the comma delimited fields are parsed universally:

https://github.com/jamesp/node-nmea/blob/43e955794d60174ac049dc5400536839e74e071b/nmea.js#L47

My next todo item there would have been to make the plugins dynamic, eg. require all the files in a directory, assume they all export ID, put those into parsers like so:parsers[plugin.ID]=plugin 

and you have a lookup table for the parsers.

See for example https://github.com/jamesp/node-nmea/blob/master/codecs/GLL.js#L25

Anyway, I'm all for it, go ahead! https://github.com/signalk/signalk-nmea would be my suggestion for a home. I'm not sure if aggregation of the tree belongs in this module, but we can extract it later if needed. Also ideas like reading from a socket, playback at constant rate are easy to add later. 

Now I would just like to get the ball rolling.

Just add a few sentences, I can fill in more.

[Fabian Tollenaar | Starting Point]

That is very nice, we might be able to fork it and save us some work. I'll take a look. 

My next todo item there would have been to make the plugins dynamic, eg. require all the files in a directory, assume they all export ID, put those into parsers like so:parsers[plugin.ID]=plugin 

and you have a lookup table for the parsers.

That sounds good, is an easy one to add. I'll set them up like this:

1. Check if Parser is ran as a fork (e.g if(process.send) {}),

2. If not check if Parser is receiving piped data on StdIn

3. If not Parser is running stand-alone

I don't think we should add playback and reading from a socket (unless by piping something like socat into the parser), in order to keep things decoupled. 

I am in favour of aggregating the child objects provided by the plugins into a fully featured Signal K tree. My reasoning is that all Signal K formatted data should be self-contained and a spec-valid Signal K object. 

Thus, even if output from the NMEA0183 parser is going to be merged with a Signal K object from a N2K Parser on a RPi server somewhere below deck, both objects should be valid on their own (in my opinion).

Sent from my iPhone

[rob...@42.co.nz]

Freeboard-server is a stand-alone java web server, starts from the command-line. It has a 'demo' ability - pretty crude but its reads a text NMEA0183 file as source, at a predetermined rate, eg lines/ms.

Where it may be useful here is that it outputs an NMEA datastream on port 5555, a websockets datastream (currently modifying to SignalK), and logs. Plus it reads whatever is plugged into USB/serial.

The NMEA parsing is all via @ktuukkan's excellent NMEA library, and quite extensive and extensible. Internally I use Apache Camel for content routing, that allows very easy changes.

There is some NMEA samples on my site https://github.com/rob42/freeboard-server/blob/master/src/test/resources https://github.com/rob42/freeboard-server/tree/master/tracks

A bit rough, I'll upload more today (I hope)

Rob

On Thursday, April 3, 2014 10:46:00 PM UTC+13, Fabian Tollenaar wrote: