Turn off Rate limited (< 5 sec) and Location changes too fast (adaptive limit)] software

[Patrick Ryan KC6VVT]

Adaptive limiting and rate limited packets leads to an issue when there are delayed packets being reported and the map will show incorrect and delayed data points.

In this case, a balloon payload is descending (KC9PON-11) and sending packets, and the altitude decreasing clearly shows that the aprs.fi software is throwing out the true packets in favor of the delayed packets.

Here is an excerpt of the raw packets to illustrate selection error caused by aprs.fi limiting..

2013-08-16 13:39:05 CDT: KC9PON-11>APBL10,WIDE1-1,WIDE2-1,qAR,KX9A:!4053.50N/08818.26WO187/006/A=008518V366
2013-08-16 13:40:05 CDT: KC9PON-11>APBL10,WIDE1-1,WIDE2-1,qAR,N9IO:!4053.35N/08818.42WO276/011/A=006933V355
2013-08-16 13:40:06 CDT: KC9PON-11>APBL10,W9MKS-15,WIDE1*,WIDE2-1,qAo,KB9VZH-1:!4053.55N/08817.91WO233/009/A=012573V366 [Rate limited (< 5 sec)]
2013-08-16 13:40:35 CDT: KC9PON-11>APBL10,W9MKS-15,WIDE1*,WIDE2-1,qAo,KB9VZH-1:!4053.57N/08817.95WO216/008/A=011720V366 [Duplicate position packet]
2013-08-16 13:40:52 CDT: KC9PON-11>APBL10,WIDE1,W9MKS-15,WIDE2*,qAo,KB9VZH-1:!4053.48N/08817.89WO248/007/A=016173V366 [Duplicate position packet]
2013-08-16 13:41:13 CDT: KC9PON-11>APBL10,W9MKS-15,WIDE1*,WIDE2-1,qAo,KB9VZH-1:!4053.55N/08817.99WO229/008/A=010910V366 [Duplicate position packet]
2013-08-16 13:41:35 CDT: KC9PON-11>APBL10,WIDE1-1,WIDE2-1,qAR,N9IO:!4053.19N/08818.60WO210/006/A=004528V355
2013-08-16 13:42:02 CDT: KC9PON-11>APBL10,W9MKS-15,WIDE1*,WIDE2-1,qAo,KB9VZH-1:!4053.53N/08818.04WO237/015/A=010073V366DePaul University Ballooning 1 [Duplicate position packet]
2013-08-16 13:42:05 CDT: KC9PON-11>APBL10,WIDE1-1,WIDE2-1,qAR,N9ZZK:!4053.17N/08818.67WO196/007/A=003727V355 [Location changes too fast (adaptive limit)]
2013-08-16 13:42:39 CDT: KC9PON-11>APBL10,WIDE1-1,WIDE2-1,qAo,KB9VZH-1:!4053.53N/08818.16WO231/006/A=009272V366
2013-08-16 13:43:35 CDT: KC9PON-11>APBL10,WIDE1-1,WIDE2-1,qAo,KB9VZH-1:!4053.50N/08818.26WO187/006/A=008518V366 [Duplicate position packet]
2013-08-16 13:43:35 CDT: KC9PON-11>APBL10,W9AZ-1,WIDE1*,WIDE2-1,qAR,N9IO:!4053.15N/08818.82WO276/010/A=001634V355 [Location changes too fast (adaptive limit)]
2013-08-16 13:43:38 CDT: KC9PON-11>APBL10,W9MKS-15,WIDE1*,WIDE2-1,qAo,KB9VZH-1:!4053.35N/08818.42WO276/011/A=006933V355 [Location changes too fast (adaptive limit)]

at time stamp 13:43:35 we see that that is the lowest altitude of A=001634, but the software discards it for "Location changes too fast" due to the delayed packets above it.

What is better, a clear but FALSE map, or a cluttered but ACCURATE map? I chose the latter, and let the user interpret the track!

Please turn off Rate limited (< 5 sec) and Location changes too fast (adaptive limit)] software. Duplicate position packet filtering seems to work fine.

73 de Pat KC6VVT

[Heikki Hannikainen]

On Sat, Aug 17, 2013 at 7:27 PM, Patrick Ryan KC6VVT <kc6...@gmail.com> wrote:
> Adaptive limiting and rate limited packets leads to an issue when there are
> delayed packets being reported and the map will show incorrect and delayed
> data points.
>
> In this case, a balloon payload is descending (KC9PON-11) and sending
> packets, and the altitude decreasing clearly shows that the aprs.fi software
> is throwing out the true packets in favor of the delayed packets.

The original problem is not aprs.fi filtering packets, but the delayed
packets (and possibly too high packet transmit rate - 30 seconds
only?). I would suggest working towards fewer delayed packets. Here
are a few things that work great for flying things such as balloons:

- Do not use a digipeater path at all when flying high. All good
trackers support altitude-based configuration switching, so that you
can drop the WIDE elements completely from the path once you've gained
a little altitude (and re-enable them automagically at landing). Your
packets are going to be received by a lot of igates directly, so
there's no need to digipeat! This will very effectively reduce network
load and the amount of delayed packets caused by digipeaters. If your
current APRS tracker does not support altitude-based config switching,
get a tracker which does (and tell your current vendor that you're
switching for a better tracker due to an important missing feature).

- Reduce your packet transmit rate.

- Normal APRS packets which have no sequence numbers or timestamps are
really hard to filter out of duplicates when receiving. Transmit
position packets which have either:
1) Base91 comment telemetry
(http://he.fi/doc/aprs-base91-comment-telemetry.txt), which happens to
have a packet sequence number, which can be used by aprs.fi to figure
out which packets were duplicates
2) Timestamps, which can be used by aprs.fi to figure out which
packets were duplicates (takes more space)

- The most important part: Find the igates which are delaying packets.
Talk (politely!) to the owners, help them fix their configurations
*or* kindly request them to take their igates offline, so that they do
not hurt the network. Demonstrate the technical problem caused by
their igates.

> What is better, a clear but FALSE map, or a cluttered but ACCURATE map? I
> chose the latter, and let the user interpret the track!
>
> Please turn off Rate limited (< 5 sec) and Location changes too fast
> (adaptive limit)] software. Duplicate position packet filtering seems to
> work fine.

Sorry, but I can't do that. Most of the time the current filters work
just fine. They're in place for multiple reasons:

1) To optimize the performance of my servers by reducing amount of
stored data (reducing RAM memory requirements, required hard disk
space, and at the same time making things run faster in general).
Without filters like this, a few stations and parts of the network
with badly working igates/digipeaters consume a larger proportion of
my server capacity than well-working network. I only have a couple of
servers with 8 CPU cores each, and only 32 GB RAM per server, and some
20K visitors daily to serve with a 300GB database on the back, so it
needs to run fast.

2) To reduce the performance hit by buggy software which suddenly puts
out packets at a really fast rate. It happens every now and then on
the APRS-IS. I just need to start dropping packets at some point for
those clients.

3) To improve the client-side performance of the map. Especially with
mobile clients (or slow browsers such as IE) the map draws really
slowly when there's a huge amount of points drawn on the path. If one
position packet per 5 seconds isn't enough, then something is
*seriously* wrong over there.

4) To encourage you to set up a better tracker configuration (slower
transmit rate, no digipeaters at high altitude, sequence numbers or
timestamps), so that network load caused by your balloon becomes
smaller. 30 second interval at 30000 feet with WIDE1-1,WIDE2-1 path is
pure madness.

- Hessu

[Lynn W Deffenbaugh (Mr)]

On 8/19/2013 2:42 AM, Heikki Hannikainen wrote:
> On Sat, Aug 17, 2013 at 7:27 PM, Patrick Ryan KC6VVT <kc6...@gmail.com> wrote:
>> What is better, a clear but FALSE map, or a cluttered but ACCURATE map? I
>> chose the latter, and let the user interpret the track!
>>
>> Please turn off Rate limited (< 5 sec) and Location changes too fast
>> (adaptive limit)] software. Duplicate position packet filtering seems to
>> work fine.
> Sorry, but I can't do that. Most of the time the current filters work
> just fine.

You can always run your own APRS-IS client that receives all
APRS-IS-delivered packets, or better, put it on RF and receive ALL
packets, and plots/records them as you want. But when you observe a
track where the final packet of the balloon is consistently a few
thousand feet in the air from the delivery of the last delayed packet,
you'll come to appreciate software that at least attempts to eliminate
these known-to-be-bad packets.

Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32
(which also attempts to filter delayed duplicate packets from its track
display)

[Matti Aarnio OH2MQK]

On Saturday, 17 August 2013 19:27:38 UTC+3, Patrick Ryan KC6VVT wrote:

...

at time stamp 13:43:35 we see that that is the lowest altitude of A=001634, but the software discards it for "Location changes too fast" due to the delayed packets above it.

What is better, a clear but FALSE map, or a cluttered but ACCURATE map? I chose the latter, and let the user interpret the track!

Please turn off Rate limited (< 5 sec) and Location changes too fast (adaptive limit)] software. Duplicate position packet filtering seems to work fine.

I see several problems there.

• Excessive use of digipeaters causing the message to be digipeated TWICE at a very large area.
• High datarate
• Lack of timestamp data within the packets
• Lots of deployed digipeaters with buggy software that pulls up packets from long delay limbo

BoB is telling that in USA thy shall use digipeater path:  WIDE1-1,WIDE2-1

For high-altitude balloons the no digipeat is usually the best way.  No WIDE-anything.

If you had a time stamp at the location beacon, you would have noticed that packet had stayed at some limbo for 40-90 seconds.

Using the "no digipeat" option avoids also this limbo effect.

 

73 de Pat KC6VVT

73 de Matti, OH2MQK

[Heikki Hannikainen]

On Mon, Aug 19, 2013 at 4:58 PM, Matti Aarnio OH2MQK <oh2...@sral.fi> wrote:
> If you had a time stamp at the location beacon, you would have noticed that
> packet had stayed at some limbo for 40-90 seconds.
> Using the "no digipeat" option avoids also this limbo effect.

Not completely, though - some igates delay packets too. Usually gates
with Kantronics KPC-3+ TNCs:
http://blog.aprs.fi/2011/03/kantronics-kpc3-considered-harmful.html

Timestamping and slightly longer transmit interval helps with that.

- Hessu

[Ray Doerr]

What kind of Timestamping can I add to my Tracker Packets so delayed packets from a DigiPeater are ignored instead of being processed while other good ones are thrown out?  I'm using compressed MIC format.

Thanks

Ray Doerr

N519RV

--
You received this message because you are subscribed to the Google Groups "aprs.fi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to aprsfi+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Lynn W Deffenbaugh (Mr)]

What kind of Tracker are you using?  TT3, TT4, Tracker2, OT2m, OT3, custom built, something else?  "Tracker" is a bit too generic to give a specific answer.

Lynn (D) - KJ4ERJ - Author of APRSISCE for Windows Mobile and Win32

[Tom Hayward]

On Mon, Aug 19, 2013 at 1:27 PM, Ray Doerr <rdoe...@gmail.com> wrote:
> What kind of Timestamping can I add to my Tracker Packets so delayed packets
> from a DigiPeater are ignored instead of being processed while other good
> ones are thrown out? I'm using compressed MIC format.

On my tracker's configuration program, there's a section called
"Reporting Options". Under that section there's a checkbox for "Time"
that, when checked, adds a timestamp to the packet.

As Hessu suggested, you can also use Base91 comment telemetry which
has a sequence number in the packet.

This question is not really related to aprs.fi and you'll probably get
better answers asking a mailing list specific to your tracker.

Tom KD7LXL

[Ray Doerr]

My Tracker is my own design so I can control whatever I want in the packet.  I currently use MIC compressed format and I include a timestamp in the text a long with a counter so I can see exactly which packets are retransmitted late by a bad or mis-behaving DigiPeater.  The question is what kind or timestamp/format in the packet could I supply that the APRS-IS Servers would parse this timestamp and/or sequence number so better be able to weed out these bad/late packets and yet preserve the good ones?

Thanks

Ray Doerr

N519RV

[Tom Hayward]

On Mon, Aug 19, 2013 at 2:13 PM, Ray Doerr <rdoe...@gmail.com> wrote:
> My Tracker is my own design so I can control whatever I want in the packet.
> I currently use MIC compressed format and I include a timestamp in the text
> a long with a counter so I can see exactly which packets are retransmitted
> late by a bad or mis-behaving DigiPeater. The question is what kind or
> timestamp/format in the packet could I supply that the APRS-IS Servers would
> parse this timestamp and/or sequence number so better be able to weed out
> these bad/late packets and yet preserve the good ones?

Check out chapter 6 of the APRS spec, "Time and Position Formats".

Tom KD7LXL

[Heikki Hannikainen]

On Tue, Aug 20, 2013 at 12:13 AM, Ray Doerr <rdoe...@gmail.com> wrote:
> My Tracker is my own design so I can control whatever I want in the packet.
> I currently use MIC compressed format and I include a timestamp in the text
> a long with a counter so I can see exactly which packets are retransmitted
> late by a bad or mis-behaving DigiPeater. The question is what kind or
> timestamp/format in the packet could I supply that the APRS-IS Servers would
> parse this timestamp and/or sequence number so better be able to weed out
> these bad/late packets and yet preserve the good ones?

The Mic-E compressed format does not have a timestamped variant, so
I'd recommend adding Base91 comment telemetry - even without any
telemetry values, with just the sequence counter in there. Adds 4
bytes to the length of the packet, less than the timestamps in
plaintext APRS packets.

http://he.fi/doc/aprs-base91-comment-telemetry.txt

APRS-IS servers will not parse any timestamps or sequence numbers -
they just pass drop out any duplicate copies of packets appearing
within 30 seconds, based on the uniqueness of the packet contents and
source/destination callsigns.

aprs.fi will look at the timestamp, and comment telemetry sequence
numbers, and I think it should parse comment telemetry without any
values, too.

- Hessu

[Mark Potosnak]

This balloon was one of our's. Pat, just tried contacting you via Facebook too. We're trying to figure out how to configure our transmitter correctly.