MQTT for Pskporter?

[Mike Karliner]

I've been mulling on the subject of a public message broker for PSKR around 10 years, and I've finally decided to poke my head above the parapet. :-)

Philip, would you be at all interested in providing a raw, 'fire hose' of PSKR reports for use by an MQTT broker?

For those who aren't aware, MQTT is a ubiquitous protocol used in the Internet of Things for reporting and telemetry. It's biggest advantage is that it's extremely lightweight, even a Raspberry Pi Zero can handle hundreds of messages per second.

Also MQTT brokers can be bridged and cascaded so that only one broker would need to be fed from the core PSKR system, which could be then 'fanned out' by slave systems that might or might not do their own filtering and processing for specialist applications.

A broker might offer a few different topics/services, 

- Single callsign feed, that is just reports featuring a given callsign.

- Single country  feed, reports from a country or state.

- And many other filters that I'm sure you could think of.

The advantages are many,

- Fully real time reporting

- Low bandwidth consumption all round, as only JSON report payloads would be sent to a browser,

- No database queries, just realtime filtering, so low broker loads, which translates to it being economical for people to join the federation of brokers and distribute the load and layer their own filtering and queries to the raw feed. For those who want persistence, databases can subscribe to the feed.

Feedback and critcism welcome.

Mike

[Robert Sillett]

Hi, Mike.

Just to clarify, you propose to have a single feed from PSKReporter to your site (infrastructure), then you will provide the MQTT broker to which multiple downstream clients will subscribe?  If so, that should keep the bandwidth requirements for PSKReporter itself quite low.  

That would open up many possibilities.  One could subscribe to your feed and then filter or even aggregate the PSKReporter data.  For example, one could create a program that would process the queue and save off your own callsign, thus giving you a local history of days, weeks, or longer.  

Bob, W4RLS

[Mike Karliner]

Hi Bob, yes, you have it pretty much in a nutshell.

Mike

[Mike Karliner]

One program that would make processing the MQTT feed nice and easy is node-red. (https://nodered.org/)

Node red provides a drag and drop, low to no code interface for processing realtime

data, like MQTT. It's free, runs on pretty much all OS's and even has dashboard 

elements. It's also used extensively in IoT and home automation projects.

Mike

[Philip Gladstone]

How many of you would subscribe to an MQTT feed of all the spots?

Would anybody be prepared to run/operate/host the broker? 

Philip

--
You received this message because you are subscribed to the Google Groups "PSK Reporter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to psk-reporter...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/3c1eb44c-ee95-40d9-98ae-083b0a035d3cn%40googlegroups.com.

[Tom M0LTE]

Me, and yes.

To be honest with the access I’ve already got I could build this out, with your permission Philip?

Tom M0LTE

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAMd_wWrU9Ng038dj58pH%2Bpng3xhJYZg_N1x03c_M-GLNdwS3Pg%40mail.gmail.com.

[Philip Gladstone]

Tom -- that was rather what I was thinking....

Philip

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAL4ggqvooWFbrm%2BhbAQRGfELzdkkaww%2BKTS4Qg_D_F8LBNyoTw%40mail.gmail.com.

[Tom M0LTE]

All- watch this space.

Thanks Philip.

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAMd_wWrqD56LLS7-vbOx9ZZBwWnOxR-xge-3u-oCqFQzmU9wPA%40mail.gmail.com.

[Jim Miller]

What would the average ham do with this?

73

Jim ab3cv 

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAL4ggqs9iLUd-9%3DagB_bOpoL1W5-Tj56UEW5QYX%2BdUspWFebyw%40mail.gmail.com.

[Steve Meuse]

I think this would be an opportunity to utilize a hosted service, like AWS HiveMQ (there are others). 

Lots of us run dedicated servers, etc, but if you're going to do it, do it at scale, make it someone else's problem. 

-Steve

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CAMd_wWrU9Ng038dj58pH%2Bpng3xhJYZg_N1x03c_M-GLNdwS3Pg%40mail.gmail.com.

--

-Steve

[Philip Gladstone]

Unfortunately, the network bandwidth charges from AWS are excessive. At the current rate, a full feed would be around $200/year in bandwidth charges (traffic exiting us-east-1). 

If you look at the pricing for (say) cloudmqtt.com, it appears to be much cheaper than using AWS. But they use AWS, and I get the feeling that if you are paying them $19/mth for up to 100 clients, and you are costing them $750/mth in bandwidth charges (50 full feeds), then the relationship will be troubled.

Most services seem oriented around IOT where the message rates are low. The pskreporter feed would exceed 500 messages/second today and will probably only get worse in the future....

Philip

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/CA%2B3sbiPXv0LwzAJyH9Uf_X9bfqq5ZwXjAtg7ODLnkr8cAdD14Q%40mail.gmail.com.

[Mike Karliner]

Seeing as I kicked this off, I obviously need to put my hand up, although

M0LTE seems to have beaten me to it. Tom, nice to meet you and I

assume we are in the same time zone. Would you like an off group

chat to see how we  could/might work together?

My vision would probably be less a full feed to everyone who wants it

as opposed to single callsign topics and single Maidenhead square

topics. Full feeds could be restricted to secondary brokers.

 Running the actual infrastructure is probably trivial compared

to working out managment and the prevention of misuse, which hosted

services can't/won't help you with.

PS:

As I've never posted here before this week, I'd like the register

my enormous gratitude and admiration for Philip. Running

the service all these years and scaling it to its present form

is quite a feat.

Mike G8LKD

[Mike Karliner]

Hi Jim,
This is a suggestion for a base infrastructure, so in it's raw form, it's of limited

interest to non-programmer hams. However, it would make it really easy to create

a lot of really useful addons to share in the community. It would also allow true realtime reporting of

spots, as there would be no database in the middle of the reporting pipeline.

Mike G8LKD

[Tom M0LTE]

Hi Mike, Philip and all

Mike, great to meet you too. I have been consuming the firehose for some time for local archive / big data playing / self-learning. I can see the community value here and I’m certainly interested in collaborating. It appears (from a direct email) that Philip, you and I are all thinking along the same lines with regards to topics and subdivision of the feed. I’ve been thinking of doing this for some time and seeing the interest here has sparked me into action. Doesn’t mean it needs to be just me, of course :)

So as to avoid polluting this list with potentially lots of traffic, I have created a public group for those who are interested in collaborating: 

https://groups.io/g/pskr-mqtt

Please feel free to join.

Thanks

Tom

To view this discussion on the web visit https://groups.google.com/d/msgid/psk-reporter/b98db3fe-9d24-432e-aa94-d899a783af2en%40googlegroups.com.

[Tom M0LTE]

Still waiting for groups.io approval of the new group, which should be a matter of hours away. So no posts yet.