Kplex and freeboard

[joachim bakke]

Hi and thank you for your help on email. Better to have the discussions here, so you don't have to repeat...

I am planning to use kplex on a raspberry pi to multiplex between an off-the-shelf multiplexer and a nexus server (previously silva, now garmin). The output of this goes to a tcp port and to freeboard. Freeboard takes in the nmea data and sends them on a java display together with maps and graphics so my ipad, computer or phone can become a display.

I'm fairly new to linux, but keep trying tngs until they work.

After getting the wifi ap thing to work, i need to send the consolidated nmea data to a named pipe (fifo) from which freeboard can read them. If i make a fifo (/dev/kplexout) and reboot the rpi, will the fifo still be active, or do I need to run a script on startup?
And in the config file for kplex, will that fifo be listed as [serial]?

Thanks again.

[Kim Bøndergaard]

Hi Joachim

Funny how we all get the same idea's.

I'm not really answering your question (Keith can do that much better), just telling you that I'm now able to create a complete reproduceable raspberry pi image (due to oe-lite (http://oe-lite.org)) making my raspberry pi a selfcontained accesspoint directing all http requests (https requests to come) to my local webserver. Besides being webserver it also provides my nmea (from my Nexus NX2 server) to my android app via kplex's tcp-server mode.
I'm planning to go live testing on my boat tomorrow.



My plans for the future is:
* webpage presenting some nmea data (freeboard is a good candidate I think)
* extend my app to have "instrument look and feel". Right now it just gives me the metrics like SOG, COG etc.

[strip...@gmail.com]

After getting the wifi ap thing to work, i need to send the consolidated nmea data to a named pipe (fifo) from which freeboard can read them. If i make a fifo (/dev/kplexout) and reboot the rpi, will the fifo still be active, or do I need to run a script on startup? 

It depends on the filesystem you create it on.  If it's a "real" filesystem then yes, it will persist.  If it's a pseudo filesystem like udev (which /dev is on ubuntu 12.04) then no, you'll have to recreate it each time.  So either create the pipe on  a "proper" filesystem (somewhere in /var for example?) or stick something into the startup script for freeboard or kplex (whichever starts first) to say something like

MYPIPE=/dev/kplexout

[ -p $MYPIPE ] || mkfifo $MYPIPE

And in the config file for kplex, will that fifo be listed as [serial]?

 No, "file", but as a word of warning see the discussion with kim who seems to be having problems with the output from file interfaces.  I hope to have more info on that in a couple of days.

You can also output using a "pty" interface which something else can read like a serial port

[joachim bakke]

Thanks again Keith,

I will definitely try that. And I will follow the other thread too. Great work!

[joachim bakke]

It seems the Nordic summer is passing and we have time to improve on our boat electronics as we wish without paying too much.
I'm having some trouble with the wireless access point. It shows some errors on startup. I basically tried to follow the adafruit guide, but it doesn't fly entirely. Would itbe possible to have a look at your pi image?

[strip...@gmail.com]

I'm having some trouble with the wireless access point. It shows some errors on startup. I basically tried to follow the adafruit guide, but it doesn't fly entirely. Would itbe possible to have a look at your pi image?

I put some notes up on setting up an access point here:

http://www.stripydog.com/kplex/examples/piap.html

It's probably not the best way to do it but might be of interest.  Possibly the most important thing I found is to update the (closed source) pi firmware.  This didn't used to be done with a standard apt-get update/upgrade but is performed using the rpi-update. rpi-update was included in the last raspbian image I downloaded (maybe 2 months ago?) but the access point was *very* unreliable until I updated the firmware.  It's still not "mission critical" quality.

[Kim Bøndergaard]

joachim, you are welcome to see how I've solved my access point setup. I'm not using a big distro like raspbian but, a minimal distro I'm in full control of due to the oe-lite build framework.
This mean I can write a fully configured image into an SD card from my PC

Having read a little more about freeboard I'm not sure it'll be that easy to port into my distro, but maybe some of my configurations can help you with your wireless stuff.

You can see my work at https://gitorious.org/nmeaap/nmea-ap

Most configurations are stored in the folder recipes/configs:

dnsmasq-conf for the dns service, directing every dns request into my device
hostapd-conf for the accesspoint stuff.
udhcpd-conf for DHCP services
lighttpd-conf for web - at the momet mostly for verification of the dns configuration. All erroneous http requests go to my index file. Have to make it work for https too
kplex-conf for nmea serial to 'wireless (TCP)'
netbase-interface for ethernet interfaces - including wlan

[joachim bakke]

Thank you both,

The fifth time I went through the tutorial, I managed to solve the issue. It required downloading and installing a newer driver for the wifi dongle. I've now suceeded and can see the pitch, yaw, roll, heading and rate of turn from the compass to my ipad. 

Next things to do is connect the instruments on board with a USB hub, and ensure a graceful shutdown of the pi when it powers off.

Thanks again!

[joachim bakke]

Hi again,

I have tried to get the output to speak to freeboard, but it seems freeboard (java) only accepts connections like /dev/ttyUSBxx and requires a baud rate. So that rules out [file]. Do you have any hints where I can read up on this?  

[Keith Young]

I have tried to get the output to speak to freeboard, but it seems freeboard (java) only accepts connections like /dev/ttyUSBxx and requires a baud rate. So that rules out [file]. Do you have any hints where I can read up on this?  

If freeboard is expecting a serial port a pty interface from  kplex might well work.  If you're running the process with a uid which has write access on /dev you could add in an interface something like pty:mode=master,baud=57600,filename=/dev/ttyUSB10

That would create a symlink called "/dev/ttyUSB10" to a pseudo terminal master device which will be usable in many ways like a serial port.

Personally I don't like running things as root which I don't need to and calling a file "ttyUSBxx" might be confusing (i.e. it looks like a USB tty device) so if freeboard can use any arbitrary serial device name I'd be inclined to create the pty with a different location/name.  If the documentation doesn't tell you how to do that, might be worth dropping Rob who writes freeboard a line to ask if it's possible.

[Keith Young]

I've had a quick look at the freeboard site: Can't find a manual as such but I had a look at the code on github.  The notes in INSTALL.txt mention how to configure serial ports in freeboard.cfg:

https://github.com/rob42/freeboard-server/blob/444c0de7da4223b03a919eb39735478191c10757/INSTALL.txt

Seems like you can call your pseudo tty what you like as long as you configure it there (with your desired baud rate)

[joachim bakke]

Hi and thanks again,

I have established that I can use other names as well. I was just confused by the error I got from the GUI (told me the names of the USBs and did not save /dev/pty22 as serial), but when adding it in the config file, it sticks.

I have a pty open from kplex, but I can now only cat it after I chmod the pty.  My problem is that I have to do this after every boot. I have tried to change the permissions of pty's in /lib/udev/rules.d/91-permissions.rules to 755, but that still does not keep the permissions after reboot.

Sorry to trouble you with this basic linux stuff. I'd be very happy if I could solve this.

Joachim

[Keith Young]

No...this wasn't actually well thought out on my part.  The slave pty is created with the uid kplex is running as, group tty if kplex is running as root, and permissions 0620.  That sucks if you want to read the slave as a different uid.  udev isn't involved here unfortunately.  This does need a code update which I should hopefully be able to do within a week and add it to the 1.2 beta.  As a workaround you could put a chmod into the kplex startup file.

[Keith Young]

The slave pty is created with the uid kplex is running as, group tty if kplex is running as root, and permissions 0620.

To clarify, those are the defaults and not what kplex explicitly implements.  What I need to do is to add an option to change that. 

[joachim bakke]

Thank you for that. I will try and change the permissions in the start script. I understand that I am well outside my knowledge here...

[Keith Young]

Thank you for that. I will try and change the permissions in the start script. I understand that I am well outside my knowledge here...

There's several ways to do it.  Assume you've specified "filename= /var/run/ktty" which will create /var/run/ktty as a symbolic link to the actual slave device.  If you want a certain process to be able to read the pty you can chmod the file to be group readable and add them to the tty group. On many current linux distributions normal tty devices are readable by group dialout, so you could change the group of the device to "dialout" before the chmod and make sure the reading process is in group "dialout".  For example, after.....

        log_daemon_msg "Starting kplex multiplexer" "kplex"

        start-stop-daemon --start --quiet --oknodo --exec $DAEMON --startas $DAEMON --chuid $RUN_AS_USER -- -f $KPLEXCONF -o mode=background

        status=$?

      log_end_msg $status

but before the following ";;" you could add something like:

[ $status -ne 0 ] && exit $status

count=0

KPTY=/var/run/ktty

while [ ! -L $KPTY ]

do

    count=`expr $count+1`

    if [ $count -eq 3 ]

    then

       $0 stop

       exit 1

    fi

done

chgrp dialout $KPTY

chmod g+r $KPTY

Actually the while loop shouldn't be necessary as kplex will almost certainly have created the symlink before "log_end_msg" has completed, but I'm not fond of "almost" certainly :-). 

[joachim bakke]

Wow thanks a lot. That would have taken me a while to figure out

[Keith Young]

Actually I think I've fixed the permissions issue and should have a new feature of being able to specify owner/group/permissions in a new version of 1.2 beta in a couple of days

[Keith Young]

OK perhaps not as much testing as I normally do but have uploaded a new beta.  This allows you to specify the owner, group and permissions on a slave pseudo terminal.  Terminology may be a little confusing here.  For pty interfaces if you specify "mode=slave" (or nothing) it tells kplex to use a pre-existing slave pty device. The new options do not apply to this.  When you specify "mode=master" kplex creates a pseudo tty, attaches itself to the "master" side, and other programs can communicate with it via the "slave".  The new options control owner/group/permissions on this slave which is what I believe is required here.  So to re-emphasise, these new options only apply for pty interfaces with "mode=master"

Full details on the beta page but...

If you specify "owner=<user>" the pty slave's owner will be set to <user>

If you specify "group=<group>" the pty slave's group will be to to <group>

If you specify "perm=<permissions>" where <permissions> are an octal representation of the desired access permissions the pty slave's permissions will be set according.  So "perm=660" would set the slave to be readable and writable by owner and group, inaccessible by others.  A leading 0 on the permissions is ignored so this could be specified as "perm=0660" if preferred.

Of course you have to have appropriate permissions to set the owner and group.  Normally kplex would need to be running as root to set a different user and a non-root user can only specify a group to which they belong.  Specified users and groups obviously have to exist.

For compatibility with OpenCPN etc., linux users might consider using the "dialout" group.

When adding this I noticed that the current version of kplex does' remove symlinks it creates to slave ptys when it exits.  This is now fixed.

[joachim bakke]

Great work on this. I tried first to insert as you wrote, but I got /etc/init.d/kplex: 35 Illegal number 0+1+1+1.....

I then removed the while loop and got the same result as before after a reboot, but when I sudo restarted the kplex service, it all worked perfectly. 

Backing up now and will try the new beta. Thank you for the great work!

And the beta worked like a charm right out of the box. Brilliant

Joachim

[Keith Young]

Great work on this. I tried first to insert as you wrote, but I got /etc/init.d/kplex: 35 Illegal number 0+1+1+1.....

    count=`expr $count+1`

Ah. My bad.  should have been spaces around the + i.e.

count=`expr $count + 1`

(...I think...)

Anyway, glad the new version (which is  much better solution anyway: thanks for pointing out the problem in the first place) works. Let us know if there are any problems

[joachim bakke]

Works in the new beta 4. One small issue though. When i restart the kplex service, it complaints about the pty not existing. A reboot works to incorporate changes in config, but not restart.

[Keith Young]

On Thursday, January 1, 2015 at 10:46:30 PM UTC, joachim bakke wrote:

Works in the new beta 4. One small issue though. When i restart the kplex service, it complaints about the pty not existing. A reboot works to incorporate changes in config, but not restart.

Sorry for the delay in replying: I overlooked this and sorry to ask questions which you may have already answered but...can you post your  current config? restart of kplex should definitely cause it to read a new config.  Maybe the restart isn't killing an existing process for some reason?

[joachim bakke]

Sure, here we go:

[global]

failover=GP***:0:nx:30:mux

#Arduino kompass

[serial]

filename=/dev/ttyACM0

name=compass

direction=in

baud=38400

#Multiplexer

[serial]

filename=/dev/ttyUSB0

name=mux

direction=both

baud=4800

optional=yes

#Nexus server

[serial]

filename=/dev/ttyUSB1

name=nx

direction=both

baud=4800

optional=yes

#AIS

[serial]

filename=/dev/ttyUSB2

name=ais

direction=both

baud=38400

optional=yes

[tcp]

mode=server

port=10110

direction=both

#[bcast]

#device=eth0

#address=255.255.255.255

#port=55554

[file]

name=log

filename=/home/pi/kplexlogs/nmea.log

direction=out

append=yes

eol=n

[pty]

name=freeboard

mode=master

filename=/dev/pty22

baud=38400

group=dialout

perm=0660

Thanks again

[Keith Young]

Sorry: I've been unable to reproduce this.  Can you let me know the exact error message so I can work out where in the code it's happening, also how you're restarting kplex.  Thanks and sorry you're having so much trouble with this: hope we'll get you working soon.

[joachim bakke]

I haven't either after that time. Let's ignore this.

[Keith Young]

I haven't either after that time. Let's ignore this.

A closer look at the init script suggests that there might be a bit of a race condition caused by the fact that ubuntu's "start-stop-daemon" doesn't wait for a process to exit before returning with a success status.

If there is unsent data, say to a FIFO or remote tcp connection, kplex could take a couple of seconds to close down, but which time the new process could be starting up.  That's not good and potentially creates a race condition which might explain why you saw this once then neither of us could recreate it.  If so, re-running the start script should have made it work. Changing the delay between stop and start to 4 seconds should (nastily) remove the problem but the real solution is to change the startup script to ensure that a previous instance has terminated before the new one starts.  I'll see what I can do with the init script to make it better.

[joachim bakke]

I have just managed to reproduce this behavior now, and it seems a few more service restarts did the trick. But even worse, it also seemed to appear after a while.

pi@kplex ~/scripts/python $ sudo service kplex restart

[ ok ] Stopping kplex multiplexer: kplex.

[ ok ] Starting kplex multiplexer: kplex.

pi@kplex ~/scripts/python $ cat /dev/pty23

$IIVPW,,N,,*1F

$TIROT,1.8,A*32

$IIVPW,,N,,*1F

^C

pi@kplex ~/scripts/python $ python pynmea_test2.py

Traceback (most recent call last):

  File "pynmea_test2.py", line 6, in <module>

    for msg in streamreader.next():

  File "/usr/local/lib/python2.7/dist-packages/pynmea2/stream.py", line 33, in next

    return [nmea.NMEASentence.parse(line) for line in lines]

  File "/usr/local/lib/python2.7/dist-packages/pynmea2/nmea.py", line 106, in parse

    raise ParseError('could not parse data: %r' % input)

pynmea2.nmea.ParseError: could not parse data: u'30$TIROT,1.8,A*32\r'

pi@kplex ~/scripts/python $ python pynmea_test2.py

Traceback (most recent call last):

  File "pynmea_test2.py", line 2, in <module>

    pty = open("/dev/pty23")

IOError: [Errno 2] No such file or directory: '/dev/pty23'

This is basically a python test trying to parse the pty output on /dev/pty23. As you can see, it throws a kplex unrelated error first, then complains about missing pty. 

Joachim

[Keith Young]

IOError: [Errno 2] No such file or directory: '/dev/pty23'

This is basically a python test trying to parse the pty output on /dev/pty23. As you can see, it throws a kplex unrelated error first, then complains about missing pty.

 The slave pty you're trying to read from will disappear when the process holding the master exits, so if I'm assuming what is happening correctly (and apologies: please correct me if I'm wrong) this is expected behaviour..   If you want to have something which you can try to open with a python program which will just block if nothing is holding open the other end, I'd create a FIFO in the file system (with mkfifo) and use a "file" interface type instead of a pty. 

[joachim bakke]

Ok thanks. I'm learning the ropes and appreciate your help.