replay183

[Keith Young]

Every so often someone asks "How can I replay boat data I've logged?".  Usually this is for testing something at home with data logged on board.  Curiously not an issue which troubles those of us who live on boats but no matter...

In order to "play back" a logged data stream we have to record not only the data we see, but when we see them so that the timing between data is the same on replay as when it was recorded.

You can log sentences to a log file by specifying a "file" interface.  If adding it on the command line it would look like this:

file:direction=out,filename=/tmp/logfile,timestamp=ms

That last bit tells kplex to add a millisecond timestamp in NMEA-0183v4 TAG format. 

Unfortunately we can't just feed this log file into another instance of kplex.  Those of you who've tried this will know that kplex just reads its inputs as fast as it can (discarding TAG blocks).  This is generally faster than it can send over a network or serial lines, so inevitably some data will be lost if there is too much to buffer.

I've hacked together a program to replay timestamped data.  Source is on github:

https://github.com/stripydog/replay183

To replay data, first log it with TAG timestamps.  Millisecond timestamps are much better for this than second resolution timestamps.  The syntax for replay183 is:

replay183 [-r] <filename>

where <filename> is your log file.  The first timestamped sentence is sent to standard out immediately, with subsequent timestamped sentences sent at the same times relative to the first as they were when recorded.  Without "-r", sentences are terminated only by a newline (<LF>) which is convenient for piping into the standard input of kplex.  With "-r", sentence are "correctly" terminated with "<CR><LF>". Note that this isn't replaying exactly as they were sent as we're taking liberties with sentence termination according to the replay needs.

It's all a bit rough and ready but it's there to play with if anyone needs it

[kmpm]

I would love to have a sample nmea log to replay.
I don't have any available and need some data to test some basic stuff with before going "live"

[Andrew Sinclair]

I have tried piping this to kplex but kplex responds with "no inputs". I must be missing something.

[Keith Young]

I have tried piping this to kplex but kplex responds with "no inputs". I must be missing something.

I wasn't explicit before.  You do have to tell kplex to read from standard input with something like "file:direction=in" on the command line (stdin/stdout are default if no filename is given) when you pipe the output from replay183 to it

[Andrew Sinclair]

That fixed it.  The only problem now is that the replay doesn't seem to replay with any delays despite the timestamp from kplex.  The log is over a day and flies through in seconds.  Here is a section of log I am trying:

\c:1468645546406*60\$GPBWC,134702,4515.9060,N,08117.2652,W,310.2,T,,M,22.10,N,Cursor,A*7C

\c:1468645546430*65\$YXMTW,19.1,C

\c:1468645546548*6B\$GPRMC,134702,A,4501.6829,N,08053.2795,W,4.4,272.1,160716,9.8,W*77

\c:1468645546643*63\$YXVHW,0,M,5.3,N

\c:1468645546677*64\$GPGGA,134702,4501.6829,N,08053.2795,W,1,6,1.70,181,M,,,,*14

\c:1468645546741*60\$WIMWV,44.0,R,11.0,N,A

\c:1468645546784*69\$GPGSA,A,3,27,8,9,16,7,,,30,,,,,4.00,1.70,3.60*32

\c:1468645546921*68\$GPGSV,3,1,10,27,63,95,35,8,53,170,37,9,52,239,39,16,47,55,33*74

\c:1468645547054*62\$GPGSV,3,2,10,7,43,304,41,23,40,200,,26,19,66,,30,16,299,35*70

\c:1468645547134*65\$GPGSV,3,3,10,21,7,38,,120,8,107,*7A

\c:1468645547279*6F\$WIMWV,44.0,R,11.0,N,ASeatalk = 53

\c:1468645547455*67\$YXMTW,19.1,C

\c:1468645547668*6B\$YXVHW,0,M,5.3,N

\c:1468645547767*65\$WIMWV,40.0,R,11.0,N,A

\c:1468645547776*65\$WIMWV,40.0,R,11.0,N,A

\c:1468645548029*67\$GPAPB,A,A,7.05,R,N,V,V,291.6,T,Cursor,310.2,T,246.6,T,A*73

\c:1468645548123*6C\$GPGLL,4501.6830,N,08053.2829,W,134704,A*3F

[Keith Young]

The log you have here is just under 4 second's worth of data.  I just copied that to a file on my mac and it seemed to play out correctly.  Let me just check you're using it as intended....

If the file is called /tmp/logfile if you just run:

replay183 /tmp/logfile

you *should* see sentences being output from the log file at the rate they were generated.

Piping it into kplex which has an existing configuration file, you'd do:

replay183 /tmp/logfile | kplex file:direction=in

your data is coming in at about 5 sentences per second.  running replay183 (without piping it into kplex) just results intrepidly scrolling text then something is clearly wrong...

[Andrew Sinclair]

So I just tried it again on my raspberry pi, I run "./replay183 ../logfile.txt and the full file takes about 2 mins to scroll by but it is a days worth of data.  The 4 seconds is just an extract from the file to illustrate the format.  Must have something to do with the raspberry Pi if it works ok on your machine.

[Keith Young]

So I just tried it again on my raspberry pi, I run "./replay183 ../logfile.txt and the full file takes about 2 mins to scroll by but it is a days worth of data.  The 4 seconds is just an extract from the file to illustrate the format.  Must have something to do with the raspberry Pi if it works ok on your machine.

I'll have access to a pi this evening (european time) and will try it then.  Could you upload a slightly larger file to test with (e.g. about 10 mins worth)?  Thanks.

[Andrew Sinclair]

I PMed you the full file and here is a longer sample.

[Keith Young]

A combination of unaccustomed warm weather and sauvignon blanc has hampered my research this evening but could it be that the date on your pi is set to something before the log file was recorded?

Setting the date to now might be a workaround (which (apologies) I haven't verified).

That's not to say your computer is wrong not the program: If that is the case I should very definitely account for the clock time being in the past and I'll update the code to do that...but possible not tonight

[Keith Young]

OK scrub that: workaround doesn't work (but it was a good plan).  In any case I've confirmed the problem exists on the pi but not on x84_64 or OS X.  Will look further tomorrow.

[Keith Young]

Found it and fixed it on my pi this morning but didn't have time to check it and push it to github.  Use of a "long" where a "long long" was needed on 32-bit ARM...I'll push the revised version this evening

[Keith Young]

Now updated on github.  Can you pull it and test?  Many thanks

[Andrew Sinclair]

I pulled and compiled.  That fixed it.  Now the sentences are coming out at the right timing on the raspberry Pi.  Thanks for that.

[Lorenz Nielsen]

Hello!

Apologies if I am necroing an old thread, but I have been reading through the documentation and the forum and I have a question which I believe fits with the 'replay183'.

We want to use NMEA data from a normal .txt file from the raspberry pi and then TX it to radios among other equipment. Normally we would just plug GPS etc into the nmea network, but we will be placed in buildings where it isn't possible to use this without a fair amount of hassle.
We have used a laptop for it previously, but would like to move to the smaller formfactor that the raspberry pi offers.

Is it correct that with kplex and replay183 we will be able to feed the nmea data to external equipment?

We're going to be using this hat for the Pi:
https://www.hwhardsoft.de/english/projects/rs485-shield/

Kind regards,
Lorenz Nielsen

[Keith Young]

Is it correct that with kplex and replay183 we will be able to feed the nmea data to external equipment?

Yes!  Simulations like this are what it was for.  Make sure the data are first recorded with NMEA-0183v4 timestamps per the README then use replay183 to read the file and output either into kplex's standard input or into a named pipe.  Give it a go and let me know if you have problems