On abuse of the 'raw packets' display

[Heikki Hannikainen]

I was looking at the server logs a bit tonight, and noticed that the raw
packets display is again being abused quite a lot.

In particular, quite a few smart folks have, again, figured that it would
be a great idea to download raw APRS-IS packets programmatically (using
software they've written) from the aprs.fi USER interface (the raw packets
view). This is forbidden by the TOS (https://aprs.fi/page/tos) and a
longer reasoning can be found in the FAQ (https://aprs.fi/page/faq):

"""
Downloading data for application use using the user interface (for
example, fetching the /info/ page just to get the current coordinates of a
station) wastes both human and computer resources.

First, you need to write a parser to extract the data from the HTML (and
then fix your parser every time I change my HTML layout). Second, aprs.fi
needs to do the user interface magic (language and timezone selection,
user-friendly template formatting, session set-up) for every request, all
of which is unnecessary.

Third, the user interface often generates much more information than you
actually need. The position API responds in a few milliseconds since the
current positions are generally available in memory, while the
corresponding /info/ page can take tens or hundreds of milliseconds to
generate because it goes and finds the nearby stations, and the
digipeaters and igates used, which often requires disk access. All of this
extra overhead consumes CPU time, which in turn heats up the computer
room, consumes electricity, destroys tropical rain forests, accelerates
global warming, and kills kittens. And baby seals. """

The correct way to obtain a raw APRS-IS feed is to connect to the APRS-IS
(http://aprs-is.net/Connecting.aspx) - you'll get a lightweight TCP
session with all the packets in real time. For free, from the very same
place where I get them from, and much faster!

If you choose to fetch the packets by hammering on my USER INTERFACE, as
opposed to an API, you'll create unnecessary extra load on my servers, and
make me a little bit upset.

I've now blacklisted a few IP addresses, so if you find you can no longer
use aprs.fi, this might be why. Some smart guy has figured I might do
this, and is using the Tor network to hide his own IP address, so I'll
probably have to block Tor exit nodes altogether. Oh, and PIA (Private
Internet Access) VPN too.

If the abuse gets too heavy, I might be forced to make the rate limits
much tighter or remove the raw packets list.

- Hessu

[jr4...@gmail.com]

Too many 421 errors.

Dr.OM.

I am a 144MHz dx lover in JA using APRS for propagation check.
I have about 40 digipeaters all over Japan to find E-skip or Ducting on 2m.
I always check aprs raw packets to find abnormal propagation.

Nowadays I always see 421 errors.

aprs.fi: 421 - There are too many connections from your internet address

Maybe I am on your black list.

I have no programming skill so use excel to check many packets when in good condx.

From now 144MHz dx condition will be better and more time have to check them.  

Pse tell me how to avoid errors or any better solutions with no programming.

Thank you.

de JR4ENY/1 Awa 144MHz ES lover


2019年3月18日月曜日 7時01分11秒 UTC+9 Heikki Hannikainen:

[Heikki Hannikainen]

Hello,

Yes, you have been violating the terms of service, by using the user
interface intended for debugging, by humans, as an API. Adusted abuse
prevention parameters blocked your use of the user interface from Excel.

You might not have been aware of it, but your use caused quite high extra
load to the aprs.fi service, since the user interface is not designed for
such use.

Unfortunately your use case is not supported by aprs.fi. Sorry!

Supporting it would require additional programming effort from me. Maybe
later I can look into supporting tropo DX alerts in a more efficient way;
it would certainly be fun and useful.

At the moment I'm mostly concentrating on server upgrades which are quite
mandatory; need to turn off the old blade chassis hardware soon to
conserve electricity and cooling power. Need to move the service to the
new blade servers first.

> --
> You received this message because you are subscribed to the Google Groups "aprs.fi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to aprsfi+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

- Hessu

[Asbjørn Hauge]

Hi Hessu, 

I just started seeing the 421 too, but that's just by normal browsing of raw packets when looking into some infrastructure quirks while configuring a bunch of trackers we use for SAR. I'm not sure how much strain normal use puts on your system, but I surely wouldn't call my usage anything but normal use. 

It's rare, but for real SAR situations we would use raw packet display to easily see status messages from our trackers so we can keep track of battery voltage. A raw packet display would be filtered with callsign "LE1*", which in turn could display all of our 50+ trackers used. I believe the 421 error would trigger quickly in that case, so I wonder what other easily available solutions we can use are. Reading status text off a map would be impossible with so many units, so a list is mandatory. APRSIS32 can, theoreticaly, be used, but the raw display is not structured in any way and I don't know of any other software we can connect to APRS-IS and do the same as aprs.fi's raw packet display. Any suggestions are welcome.

-- 

Best regards / 73 de

Asbjørn Hauge, LA1HSA

søndag 17. mars 2019 23.01.11 UTC+1 skrev Heikki Hannikainen følgende:

I was looking at the server logs a bit tonight, and noticed that the raw
packets display is again being abused quite a lot.

   - Hessu

[Nosey Nick VA3NNW]

It's rare, but for real SAR situations we would use raw packet display to easily see status messages from our trackers so we can keep track of battery voltage. A raw packet display would be filtered with callsign "LE1*", which in turn could display all of our 50+ trackers used. I believe the 421 error would trigger quickly in that case, so I wonder what other easily available solutions we can use are.

Connect direct to the APRS-IS servers.

First, make sure you know what callsign and SSID YOU are going to use (not your trackers, but YOU doing the fetching), and your APRS passcode (probably a 5-digit number? If you don't already know yours, email me off-thread with proof of your name and callsign and I'll send you one).

Next, check http://www.aprs-is.net/javAPRSFilter.aspx and work out what filter you want. I'm not entirely clear whether you want FROMCALL, TOCALL, or OBJECT NAMES of "LE1*", but assuming you want all messages FROM "LE1*", then it looks like "p/LE1" might be what you mean? Note that some support "*", some are already "prefix" matches, and you can combine a few, so maybe "p/LE1 o/LE1* g/LE1* u/LE1*" to catch all the things you MIGHT have meant?

In a text file / notepad / similar, prepare a line like...

user YOURCALL-SSID pass 12345 filter p/LE1

... except inserting the values of your callsign-ssid, your passcode, and your chosen filter.

Every Windows or Linux machine comes with a "telnet" command. Open up a terminal,  "command prompt", "DOS prompt", "start, run command, cmd", or whatever.

Type "telnet rotate.aprs.net 14580", hit return. It should fairly quickly tell you something like "Connected to rotate.aprs.net" and a server version, maybe "# aprsc 2.1.4-g408ed49" or "# javAPRSSrvr 4.2.1b12" or something. Fairly quickly, cut and paste the "user blah pass blah filter blah" line you prepared, hit return. It will probably tell you you have been verified, and respond with a software version, a date/time, a name for the server, and a bunch of other stuff, maybe:

# logresp YOURCALL-SSID verified, server FIFTH
# aprsc 2.1.4-g408ed49 15 Apr 2019 13:22:19 GMT FIFTH 45.63.21.153:14580

Once in a while it will repeat the version/date/time/name line, but it will ALSO, in among them, show your filtered raw packets. I've been connected for a while and (with the above "p/LE1" filter) I've captured:

LE1ATC>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:>010000zFW=v1.0 (27.2.2010) / VBATT=6.9
LE1ATA>APPT10,LD2BI,WIDE1*,WIDE2-2,qAR,LD2GI:/132900h5956.16N/01101.46E[326/027/A=000609
LE1ATA>APPT10,LD2BI,WIDE1*,WIDE2-2,qAR,LD2GI:/132912h5956.22N/01101.33E[298/027/A=000622
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133018h5956.43N/01100.82E[219/017/A=000577
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133029h5956.42N/01100.81E[299/000/A=000578
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133101h5956.42N/01100.81E[299/000/A=000578 NRRL Samband Oslo
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133133h5956.42N/01100.81E[299/000/A=000578
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133217h5956.34N/01100.52E[214/027/A=000561
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133228h5956.30N/01100.41E[234/023/A=000573 NRRL Samband Oslo
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133240h5956.25N/01100.28E[236/026/A=000546
LE1ATA>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:/133315h5956.04N/01100.06E[166/022/A=000517
LE1ATA>APPT10,LD1BK*,WIDE2-2,qAR,LD3HV:/133448h5955.65N/01100.38E[084/018/A=000546
LE1ATA>APPT10,LD1BK*,WIDE2-2,qAR,LA1HSA-2:/133533h5955.66N/01100.79E[110/018/A=000616
LE1ATE>APPT10,WIDE1-1,WIDE2-2,qAR,LA1HSA-2:>010000zFW=v1.0 (27.2.2010) / VBATT=7.1

If all this "telnet" stuff sounds too complicated, look at ANY APRS software that can speak to APRS-IS. You'll probably still need your callsign, ssid, passcode, and a filter though. APRS.fi is not the only APRS-IS viewer out there.   :-)

Nick VA3NNW

[Asbjørn Hauge]

Hi Nick, 

Thank you for the very detailed suggestion. I'm very aware of how APRS and APRS-IS works, but it could be nice for other to see too :)  I'm also aware of the data being available directly in the APRS-IS feed and fetching, sorting, storing and displaying that data may be the only viable approach I have. The point, or problem, is that it's not necessarily me that will be keeping track of battery status. Any ham in the field or at home backing up the operation should be able to perform this task, wheter he/she uses a Windows PC, Mac, Linux, Android phone or an iPhone. In turn this mean it should be web based since the know-how to use telnet for this may be way out of reach for most. This is why the aprs.fi raw packet web page is so nice just for this purpose, it's really sad to see these restrictions come into play now. Also, looking at the APRS-IS stream will not give you historical data unless you collect it all the time.

Looks like I have to dig up something of my own so it can be presented on a web page. Should have some time looking into it during Easter.

I understand Hessu and that scripts fetching a lot of data all the time is straining the system, but I hope something between the old and new "limit" is possible.

-- 

Best regards / 73 de

Asbjørn Hauge, LA1HSA

--

[Ray Rischpater]

If it helps, I threw together a little (and I mean really little) project last Christmas that connects to the APRSIS stream, does some filtering, and stores it in memory for later display on the Web. My application was for map data (kf6gpe.org/where), but it might be a starting point for you.

https://github.com/kf6gpe/pyaprs-stationservice

73 de KF6GPE

Ray Rischpater, KF6GPE | kf6gpe.org | @kf6gpe

[Heikki Hannikainen]

Hello,

I've looked at the logs again. I'm a bit torn between a few options:

* Relax the limiters
* Make them even more strict
* Remove the raw packets display altogether
* Add a captcha or other robot control in front of it

The difficult thing is that I have better things to do than battle the
abuse. If those silly people did not do this, I might spend the time to
create something useful.

According to the log file, the current rate limit denied access to the raw
packets display 19279 times yesterday. You can probably guess that there
were not *that* many actual people trying to look up raw packet data
manually from there *that* many times. There may be a few false positives
but... 19k? No.

Some of the abuse limiting is based on having rate limits per IP address.
So, the heaviest abuser is using the Tor onion routing network, because it
allows him to have a lot more different IPs. So, I'll probably block Tor
for now.

Dear Tor-using raw-packet-abusing 1000-packets-at-a-time
every-few-seconds-fetching person: If you're reading this, please try:

telnet rotate.aprs.net 10152

type "user foobar" and press Enter. Observe a nice real-time feed of all
APRS packets in nice text format. Faster than polling my user interface.
Easier to parse, no HTML messing it all up.

- Hessu

[Heikki Hannikainen]

On Tue, 16 Apr 2019, Heikki Hannikainen wrote:

> * Relax the limiters
> * Make them even more strict
> * Remove the raw packets display altogether
> * Add a captcha or other robot control in front of it

Tom Hayward pointed out one more option, which is pretty good:

* Require logging in to aprs.fi to access the raw packets list

This way I can have a rate limiter per user account, which can be more
relaxed so that normal use should not be affected.

It'll make automated use of the view difficult (but not impossible), as
you need to login first. The signup view has a captcha so automated
signups are difficult (but not impossible).

It'll be slightly more difficult in normal use if you're not logged in
already, but the login cookies have a long lifetime so you don't need to
do that often.

- Hessu

[Lynn W Deffenbaugh (Mr)]

+1 on that idea!  I use the raw packets to debug new devices and devices
that have unexpectedly changed behavior as well as to look at some
remote stations and their corresponding iGates.  Currently, I don't stay
logged in to aprs.fi, but this will be a small price to pay to keep the
raw packets display available for interactive use.

Lynn (D) - KJ4ERJ

[Rusty Dekema]

From my perspective (as an occasional user of the raw packet list),
requiring a login to access it seems like a good option to me. It
seems like it should solve the abuse problem and also have minimal (or
no) impact on legitimate/normal users.

Cheers, and thanks for all your work with aprs.fi,
Rusty Dekema

[Eric Wilhelm]

Hello,

I am a television meteorologist in Ohio. I was fumbling around with Excel trying to set up a graphic for use on TV that shows rain totals from local personal weather stations. My intention was to have Excel grab data from about a dozen stations once every hour or two. While trying to set up the Excel macro correctly, I did hit the APRS site several times in a row and consequently my IP address was blocked. I am looking for written permission to grab this data, as the Terms of Service says I should do. I am not looking to abuse the system or download a huge amount of data. Any help you can provide would be appreciated.  

[Heikki Hannikainen]

Hello,

I think it'd be best if you used the API, which is designed for precisely
this purpose: fetching data using an Application such as Excel.

https://aprs.fi/page/api

The API can give the data in JSON or XML format, and Excel seems to be
able to fetch data from an URL in XML format:

https://blogs.msdn.microsoft.com/hovsep/2007/07/11/excel-2007-how-to-connect-to-and-import-data-from-an-xml-web-service/

This should be much more reliable, as it will not break when I change the
layout of the User Interface. It also consumes much less of my server
resources.


On Tue, 30 Apr 2019, Eric Wilhelm wrote:

> Hello,I am a television meteorologist in Ohio. I was fumbling around with

> --
> You received this message because you are subscribed to the Google Groups
> "aprs.fi" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to aprsfi+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>

- Hessu