Grid square "rarity" maps based on pskreporter data?

[Mouser Williams]

I'm interested in producing maps of grid "rarity" based on pskreporter's database on a per-band basis (and possibly per-mode), but I don't believe the current API allows for this. This would be a useful tool for people planning expeditions to under-served grids. These data exist for the 6m band in the form of the FFMA leaderboard (see http://www.arrl.org/files/file/FFMA/FFMA_Survey_2010.pdf ) but the data are only based on lifetime miss reports from the FFMA leaderboard members and are likely not representative of actual signal density from a given grid.  

I'd be willing to do the leg work to create the maps if I could get access to some SQL results from the database (something like  SELECT COUNT(*) FROM reports WHERE grid='DM02' and band='23cm'; -- I don't know how the database is structured, obviously.. hah). Just the 488 CONUS grids would be valuable by themselves, or the full 32,400 Maidenhead grid square set if you're feeling spicy...

This would be useful even if it was only run once on, say, a year's worth of data (does pskreporter even keep records back that far?). Or, if the load on the database to run these queries turns out to not be ridiculous, it could be an interesting dynamic feature for the website -- a map of signal source density by grid square over a user-definable period of time, band, mode, whatever.   

Is this idea infeasible? Would anyone else care about access to such maps? Does this already exists somewhere and I'm just a huge clod?

Just wanted to get a conversation started; let me know what you think

Cheers,

-Richard

KB1IKC

[Philip]

Unfortunately, I only keep the last 10 days or so online. However, I did run the query over the last 24 hours that you wanted and put the result in https://pskreporter.info/locators.txt

Is this the sort of data that you wanted?

Philip

[Mouser Williams]

Yes that's exactly what I was looking for; thank you!  Having more time coverage would make the data more useful; any chance something like this could be run in a cron job once a day or once every 10 days and archived or summed up for a year?  :-D 

Cheers,

-Richard

KB1IKC

[Paman Leo.]

thank you and good luck to you

--
You received this message because you are subscribed to the Google Groups "PSK Reporter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psk-reporter...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/149333f4-124b-4030-a116-a8ae4c658242n%40googlegroups.com.

[Mouser Williams]

OK so this is a really rough first cut, but this is the all-band totals for every grid square in the world over the 24 hour period you pulled the data for.  I colored all grids with zero reports as black, then a linear color scale from red to green between 1 and 1000 reports.  Everything >= 1000 is the same color in this particular representation.  

(I hope that picture gets through the posting process here...)

A few observations: 

* Almost no activity in Africa (outside of South Africa) and very little in south Asia

* A surprising amount of active grid squares off the eastern seaboard of the US (though almost all of them are single reports)

* A surprising amount of activity in the arctic, but again single reports for the most part

On Tuesday, January 25, 2022 at 8:15:06 PM UTC-7 Philip wrote:

[Wes Cosand]

Richard, this looks interesting.  I wonder what the error rate is for FT8 grid squares.  This seems to indicate that it is higher than I might have anticipated

Wes WZ7I

--

You received this message because you are subscribed to the Google Groups "PSK Reporter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psk-reporter...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/0db6a9c0-9612-47f7-821c-bf138bb8692an%40googlegroups.com.

--

Wes Cosand
6299 Route 413
PO Box 349
Pipersville, PA 18947

[Mouser Williams]

Yeah I agree. I think data averaged over a longer time would probably show some universal low-level noise due to bit errors that make it through the FT8 message authentication features.  

The truly rare grids that I'd like to draw out using this technique might actually be buried in the error noise, unfortunately. 

-R

You received this message because you are subscribed to a topic in the Google Groups "PSK Reporter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/psk-reporter/Z2jBJzzoFgA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to psk-reporter...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAEZgdkZ78h13%3DRpMnZt6jMwJB86RGyg0ACWiFfACurgaDgseZg%40mail.gmail.com.

[Philip Gladstone]

I'll try and generate a week of data. Many of the single reports could be false decodes... 

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAOERBK%3DCk-OkhYn1Drt8inMi_X_thqOQabgZXifdtzwx67Z1_w%40mail.gmail.com.

[Russ]

About FT8 errors, I have always used only WSJT-X, but recently I started playing with MHSV due to a request from a friend.  Running they both side by side, I see about 8 times as many errors with MHSV.  For example, just listening to white noise all night, with WSJT I typically see 1 error every other night. On average.  But with MHSV I usually see 4 or 5 errors every night.  This is listening to six meters, dead band so no real signals were present.

 

MHSV requires about 20 to 25 Db higher audio input level than WSJT-X, and that may affect the error ratio.  Anyway, if there was a way to separate MHSV reports and ignore them, the error rate on your experiment might be significantly reduced.

 

73, Russ K2TXB

 

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAOERBK%3DCk-OkhYn1Drt8inMi_X_thqOQabgZXifdtzwx67Z1_w%40mail.gmail.com.

[Philip Gladstone]

I think that it depends on whether you have the deep deciding enabled or not.

Philip

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/221001d81528%244d91de30%24e8b59a90%24%40comcast.net.

[Philip Gladstone]

I'm going to build the data for the last week, but I am going to filter out the grids that only get reported once on a band in a 20 minute slot. This should eliminate most of the false decodes. It'll be interesting to see what the new chart will look like.

Philip

[Russ]

I used deep decode in WSJT-X.  There is no deep decode in MHSV, for FT8, but I have 3 stage decoding turned on.  I will have to investigate to find out if that is the same as ‘deep’.

 

Thanks, Russ K2TXB

 

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAMd_wWq5BSeP44xXY8N669Cp%3DzS3pDZVkXBc76D2-gjTQyN0zA%40mail.gmail.com.

[Russ]

Investigation shows that 3 stage decoding is probably the same as Deep decoding.  I cannot find anywhere that says that, but the descriptions are similar.  It would be better if developers used same words to describe the technical aspects of their programs…

 

Russ

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/225b01d81530%24425cef70%24c716ce50%24%40comcast.net.

[Philip Gladstone]

This should contain the current data going back to 20th Jan.

https://pskreporter.info/cgi-bin/gridusage.pl

Philip

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/227901d81539%241b1b4200%245151c600%24%40comcast.net.

[Mouser Williams]

Here's the data again with 0=black.  The upper threshold is 10,000 counts in this map:

Here are the same data but with the full range. Unfortunately Tokyo is such an outlier on the high side that it renders the color scale mostly useless:

And so here is the full-range data but in log scale: 

I'm really surprised at how much activity there is in central Russia... I just don't hear those folks very much. Possibly a symptom of my crap antennae...

There's still a halo of "noise" around North America and I suppose it's possible these are /MM stations but I think decoding errors is the more likely culprit.  I wonder if JT would find these data useful for fine-tuning their bit error acceptance rate? 

-Richard

KB1IKC

[David Ackrill]

" I'm really surprised at how much activity there is in central Russia "

If this is a snapshot of all PSK spots, then Central Russia is in easy reach of those of us in Europe on a number of HF bands and even some VHF ones at times..

Dave (G0DJA)