Connection drops with no traffic?

[Richard Montgomery]

Been working with Kplex at my home as a test to get it setup before installation.

I am out of range of any boat traffic at home, but where I will be installing it will be at the river.

My question is does Kplex drop the connection after a period of time if there is no traffic being sent to marinetraffic?

I have it setup for TCP connection. It connects normally on startup. But after awhile I get an email from marinetraffic that it is offline. There doesn't appear to be a TCP connection to marinetraffic but Kplex is still running.

I am running the dAISy Hat on my raspberry pi. Running buster OS. I have also ran rpi-update and all latest updates.

Are there logs I can look at for any messages about connection issues?

Any help or ideas would be appreciated.

Thanks

Richard

[serial]
filename=/dev/serial0
direction=in
baud=38400
[tcp]
mode=server
port=10110
direction=out
[tcp]
address=5.9.207.224
port=<hidden>
persist=fromstart
direction=out
keepalive=yes
[udp]
address=195.201.71.220
port=<hidden>
direction=out
#[udp]
#address=5.9.207.224
#port=<hidden>
#direction=out
#coalesce=yes
[global]
debuglevel=5

[Keith Young]

To see more of what is doing on either edit the kplex startup to include a "-d 7" or add a section to the config file:

[global]

debuglevel=7

Then check your system log (/var/log/syslog on raspbian iirc).  debuglevel 7 should show you all reconnection attempts.  Your config is set up to retry so my suspicion would be a "known issue" that if kplex is using a network interface which obtains its address from dhcpd and the lease is lost or a new address assigned any kplex interfaces using that address will shut down.  The solution would be to use a static address or do something fancy with dhcpcd hooks to restart kplex when a new interface address is assigned.

[Richard Montgomery]

Here is some syslog info pertaining to kplex. See anything out of place?  It seems kplex reconnected to marinetraffic when it decoded ais signals? 

Thanks

Richard

Jun 14 00:34:37 EVA-AIS systemd[1]: Starting LSB: Start kplex NMEA 0183 multiplexer...

Jun 14 00:34:37 EVA-AIS kplex[502]: Starting kplex multiplexer: kplexkplex DEBUG: kplex starting, config file /etc/kplex.conf

Jun 14 00:34:37 EVA-AIS kplex[502]: kplex DEBUG: _serial-id1: opened serial device /dev/serial0 for input

Jun 14 00:34:37 EVA-AIS kplex[502]: .

Jun 14 00:34:37 EVA-AIS systemd[1]: Started LSB: Start kplex NMEA 0183 multiplexer.

Jun 14 00:34:37 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2: initialised

Jun 14 00:34:37 EVA-AIS kplex[502]: kplex DEBUG: _udp-id3: output address 195.201.71.220, port hidden

Jun 14 04:49:23 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2 id 20000: write failed: Connection reset by peer

Jun 14 04:49:23 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2: Reconnecting (write) interface

Jun 14 04:49:28 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2: Reconnecting...

Jun 14 04:49:29 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2: Reconnected (write) interface

Jun 14 04:49:29 EVA-AIS kplex[502]: kplex DEBUG: Flushing queue interface _tcp-id2

Jun 14 08:36:35 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2 id 20000: write failed: Connection reset by peer

Jun 14 08:36:35 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2: Reconnecting (write) interface

Jun 14 08:36:40 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2: Reconnecting...

Jun 14 08:36:41 EVA-AIS kplex[502]: kplex DEBUG: _tcp-id2: Reconnected (write) interface

Jun 14 08:36:41 EVA-AIS kplex[502]: kplex DEBUG: Flushing queue interface _tcp-id2

[Keith Young]

Nothing odd there.  Seems like you're getting disconnected by marine traffic roughly every 4 hours.  I just reminded myself that kplex uses a default (30 mins) shorter than the linux default (2 hours, as recommended as the minimum default in rfc1122) before sending the first keepalive packet so despite a quick google suggesting that marine traffic regards you as offline if it hasn't seen data or a keepalive packet in the past 2 hours it marks you offline, this would not seem to be the cause.

Did you get an email from marine traffic about being offline while recording the above?  If so when did it say you went offline?

[Richard Montgomery]

Got an offline email from marinetraffic at 3:17AM during the above syslog. I received an online email at 2:12 AM the same timeframe. So roughly an hour after the online email I get the offline. 

I will keep testing. I may put it in where it will receive a lot of traffic and see if it goes offline.

Thanks

Richard

[Keith Young]

Got an offline email from marinetraffic at 3:17AM during the above syslog. I received an online email at 2:12 AM the same timeframe. So roughly an hour after the online email I get the offline. 

Do you have any documentation on what the "online"/"offline" emails are triggered by?  All I could find was this page on how they define status:

https://help.marinetraffic.com/hc/en-us/articles/218688357-How-is-the-AIS-Station-Status-defined-

Presumably from your earlier log you should have been "online" at 00:34, but even if no data sent, keepalives should have been being sent after 30 mins, well below their 2 hour limit.

Can I confirm that your connection is low volume and you may go for some 10s of minutes without data passing over the connection?  Rather than moving to a high volume location I'd like to try to get to the bottom of this.  I'm wondering if you're behind a NAT device which times out connections in a shorter period than the 30 mins before kplex sends a keepalive.  May I suggest some or all of the following?

Add:

keepidle=300

keepintvl=240

...to the stanza for the tcp connection sending to marinetraffic.  This will trigger a keepalive after 5 mins of inactivity and make the time between subsequent attempts 4 mins.  Hopefully this should get round any NAT state maintenance issues.

Add a "log file" with time tracking so we can tie up periods of inactivity with the connection dropping:

[file]

filename=<path_to_logfile>

append=yes

timestamp=s

For added points we could try monitoring with tcpdump, ie, as root or via sudo...

tcpdump -w <output_file> -l -i <name of outbound interface> tcp and dst host 5.9.207.224

[Newcastle Lifeboat]

My setup uses static ip addresses but I get these disconnections. I put it down to no traffic in area based on my location and distance from sea. Might have to look at this further.

 I’m going to set up another pi closer to the sea Where I live but it will have reduced height - due to me living nearer to sea level.

Declan 

Sent from my iPhone

On 13 Jun 2020, at 22:21, Keith Young <strip...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "kplex" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kplex+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kplex/3f3b8d00-19c9-484b-8e29-9ca75898a135o%40googlegroups.com.

[Keith Young]

On Saturday, June 20, 2020 at 11:56:34 AM UTC+1, Newcastle Lifeboat wrote:

My setup uses static ip addresses but I get these disconnections. I put it down to no traffic in area based on my location and distance from sea. Might have to look at this further.

As with Richard's config, perhaps this is an issue with NAT state tracking on your router timing out before keepalives kick in  if you have no traffic passing?  You could try shortening keepidle and keepintvl as above?

[Richard Montgomery]

My logs show quite a bit of traffic is being logged. Is there a way to log the decoded AIS data instead of the raw text? 

How about a time stamp function that would show when I logged the receiving data at my station?

The dropping of my connection to marine traffic seems to be missing a lot of my data that I would normally be sending to them. 

Thanks

Richard

--
You received this message because you are subscribed to a topic in the Google Groups "kplex" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kplex/ez-q85gTM7E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kplex+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kplex/284bc4c3-8f1a-42ed-aca3-f6d37921463ao%40googlegroups.com.

[Keith Young]

My logs show quite a bit of traffic is being logged. Is there a way to log the decoded AIS data instead of the raw text? 

How about a time stamp function that would show when I logged the receiving data at my station?

The "timestamp=s" directive I mentioned earlier when talking about logging should have added a timestamp in NMEAv4 TAG format: e.g. "\c:1423133110*5C\". The number between the c: and the checksum (preceded by "*" is the "unix" time, ie seconds since midnight UTC On jan 1st 1970, or in this case "Thu, 05 Feb 2015 10:45:10 +0000"

If you google "unix time converter" or similar there's a bunch utilities on the Interweb for doing the conversion for you (also easy enough to do in python or similar).

kplex doesn't decode AIS, you'd need to write a script for that.  There are online decoders but this looks a pretty good resource for writing your own:

https://gpsd.gitlab.io/gpsd/AIVDM.html

Remember of course that AIS sentences can be multi-part so a single sentence doesn't always have a complete AIS "message"

The dropping of my connection to marine traffic seems to be missing a lot of my data that I would normally be sending to them. 

Did you try the keepidle and  keepintvl changes?  What type of NAT router are you going through?

[Richard Montgomery]

Yes I am using the keepidle and keepintvl changes. I am using a regular home router. I used to run a scanner feed at my home with the same hardware and never had any connection drop issues like this. 

While troubleshooting this connection dropping issue I changed to static IP for my raspberry pi but didn't seem to make a difference.

Also there are no log files recorded that I can find pertaining to connection issues with marinetraffic. The only logs are decoded frames that my receiver picks up. 

In looking at /var/log there are only these errors that don't seem to pertain to kplex:

gpgconf: running /usr/bin/gpg-agent failed (exitcode=2): General error

gpgconf: fatal error (exit status 1)

No other errors that I can find and no mention of connection drops. 

Thanks

Hope that helps with some troubleshooting

Here is a copy of my entire kplex.conf :

[serial]

filename=/dev/serial0

direction=in

baud=38400

[tcp]

mode=server

port=10110

direction=out

[tcp]

address=5.9.207.224

port=<hidden>

persist=fromstart

direction=out

keepalive=yes

keepidle=300

keepintvl=240

[udp]

address=195.201.71.220

port=<hidden>

direction=out

#[udp]

#address=5.9.207.224

#port=<hidden>

#direction=out

#coalesce=yes

[global]

debuglevel=9

[file]

filename=/home/pi/ais_logs/log

append=yes

timestamp=s

direction=out

[Keith Young]

- Does the file /home/pi/ais_logs/log exist?  I should have specified that whilst kplex will create the file if it doesn't exist, it won't create the directory

- Are you still getting the emails about going offline?  In which case, what traffic do you see being logged with a timestamp between 2 hours and 2.5 hours before the time on that message?

- Assuming you are still seeing disconnects/reconnects in the kplex logs, how do these relate to the traffic in your log of AIS sentences?

Feel free to email me any logs with anything sensitive redacted.

And can I check here...you have two destinations, one tcp and one udp.  Are they both ceasing to function?

I think we may need to look at what's going on at the tcp level with tcpdump

The reason for asking about what kind of router was so we could try to find how it manages connection state, although if you now have keepalive packets firing every 5 minutes it would be unlikely that this is timing out (I've been confirming this works as expected this evening and it does).

[Richard Montgomery]

Thanks Keith,

Yes I created the path for my log file. And it does populate. I don't get much ais traffic. The last traffic recorded in the log file is from June 27th. On reboot I get a connection to marinetraffic as shows using netstat -ant. After awhile (sorry don't know how much time passes) I check and it doesn't show a connection:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:10110           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:65535         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:64000         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:39498         127.0.0.1:22            ESTABLISHED
tcp        0      0 192.168.11.9:43074      5.9.207.224:<hidden>       ESTABLISHED
tcp        0    192 127.0.0.1:22            127.0.0.1:39498         ESTABLISHED
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::29998                :::*                    LISTEN
tcp6       0      0 :::29999                :::*                    LISTEN

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:10110           0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:65535         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:64000         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:39498         127.0.0.1:22            ESTABLISHED
tcp        0    192 127.0.0.1:22            127.0.0.1:39498         ESTABLISHED
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::29998                :::*                    LISTEN
tcp6       0      0 :::29999                :::*                    LISTEN

The only connection is my ssh using remote.it service. I know the PI is online this entire time as I will get an email alert instantly from remote.it anytime the connection to the internet is interrupted (including a reboot). I never get those messages during the time my marinetraffic connection drops. 

Like I said I am testing this at my house before I install it in a higher traffic area. I have it connected to an outside antenna but conditions have to be perfect before I decode any AIS traffic as I am a long way from the TN River. 

In the logs you mention I never see any connects/disconnects. Only successful decodes of AIS traffic is in there.

Maybe I am overlooking something significant and just don't know where to look for more information.

Thanks

Richard

--

You received this message because you are subscribed to a topic in the Google Groups "kplex" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kplex/ez-q85gTM7E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kplex+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kplex/97e01f25-165f-4a57-b166-8845bee135ceo%40googlegroups.com.

[Keith Young]

In the logs you mention I never see any connects/disconnects. Only successful decodes of AIS traffic is in there

The log file /home/pi/ais_logs/log will contain only NMEA data.  Any messages about disconnects and reconnects will still go to syslog file as they were doing in your earlier post.

Would it be possible to run the tcpdump I mentioned earlier fro several hours over a period where you're seeing problems? ie:

/usr/sbin/tcpdump -w <output_file> -l -i <name of outbound interface> tcp and dst host 5.9.207.224

(apt-get install tcpdump if it's not already on your system)?

you can "sanitise" it to remove references to your destination port and mail me the result, ie if you're sending to tcp port 12345:

sed 's/\.12345/\.xxxx/' <output_file> > <redacted_file_to_email>

[Keith Young]

sed 's/\.12345/\.xxxx/' <output_file> | gzip > <redacted_file_to_email>

(perhaps would make it smaller to send...) 

[Richard Montgomery]

Do you have a direct email that I can send my tcpdump wireshark file to? 

I let it run even during a disconnect email from marinetraffic so maybe there is some usable information.

Thanks

Richard