XENC.HABHUB.ORG Sunset

[Anthony Stirk]

Hi all,

 

I’m putting this on the tools mailing list before I go on the more public list.

 

Nevis/Uputronics have hosted the Habhub platform since 2011 but the physical server (XENC) underlying the Habhub infrastructure is now very old and the operating systems on it haven’t had security updates since 2018. Unfortunately most of the people who architected the system originally now have jobs/have moved on from the hobby and are understandably unable to offer the time to develop/maintain the platform moving forward. It has unfortunately become unmaintained.

 

So with regret I’m going to have to give a date at which point I intend to shut the server down and retire it. That date will be 30^th December 2022. This date assumes the system isn’t compromised should this occur it will be shutdown immediately.

 

I understand there are a number of projects that offer similar functionality to Habhub but lack some of the features. I’m hoping I’m giving enough time for people to relocate the services on these servers to modern maintained platforms.  I’m happy to assist in anyway I can to smooth this transition.

 

If you have comment or would like to discuss rather than clogging everyone’s mailboxes up please come chat on #habhub irc.libera.chat.

 

Cheers

Anthony

Anthony Stirk​ Managing Director T: 01274 550911 E: Anthon...@nevis.co.uk  |  W: www.nevis.co.uk Nevis Computers is a limited company registered in England and Wales. Registered number: 3356647.   Registered office: York House, 16 Wool Gate, Cottingley Business Park, Bingley, West Yorkshire, BD16 1PE.    Emails may be monitored for the purposes of security and staff training

[Mark Jessop]

I guess I should try and explain more about the platform Michaela and I have been working on for a while now.

We started up SondeHub in about 2019 as a platform for storing and mapping meteorological radiosonde traffic. Originally we started out feeding radiosonde data into Habitat, though this very quickly proved to be not a good idea, so Michaela set up a new database system based on Elasticsearch in AWS. Things have since built out a *lot* from there:

- We forked the Habhub tracker and switched it to Leaflet (since using the Google Maps API was just not really practical due to the API load limits) - https://sondehub.org/

- We containerised the Tawhiri Downloader and Predictor, and re-pointed the downloader at the AWS S3 mirror of the GFS model (faster and free traffic for us)

Yes, hosting in AWS is certainly not free. We've been funding this mostly out of our own pockets and from Patreon donations, but last year we were able to secure a grant from Amateur Radio Digital Communications, which should fund us for the next 12 months or so. We were also told to 'think bigger' in our next grant submission, so it looks like further funding should be possible, especially if it ends up being the prime tracking site for amateur radio high-altitude balloon launches.

One of our 'stretch goals' in the grant submission was to extend the database for amateur radio balloon traffic, and we've made a lot more progress in that space than we had originally intended, including:

- Database changes to support amateur radio traffic (noting that AR traffic is a fairly small fraction of what we are currently handling in radiosonde traffic)

- APRS-IS importer.

- Sondehub-Amateur tracker, building off the changes made for the SondeHub tracker: https://amateur.sondehub.org/

Currently the only telemetry sources feeding the sondehub-amateur database are the APRS-IS importer, and my Horus Binary decoder software. The database is modelled off the Sondehub db, so it expects 'deconstructed' telemetry packets, not UKHAS strings. There's certainly a lot more work to do to make it 'feature comparable' with the existing Habhub tracker and Habitat database - some points off the top of my head:

- There is currently no concept of a payload/flight document, nor a way to parse UKHAS strings.

- There are no hysplit predictions (though we have been experimenting with using the Tawhiri float-profile predictions as a substitute)

- There is no easy way to export telemetry in useful formats (e.g. CSV, KML).

Still with additional work, I hope it might be considered as a viable replacement for the current Habitat database. 

I've started a repository on github mainly for the point of tracking issues relating to what parts of Habitat need to be moved here: https://github.com/projecthorus/habitat-server-transition/issues/1

I invite contributions to the discussion on what needs to be moved / replaced / rebuilt. There are certainly parts of this that are 'low hanging fruit' and could be tackled by a keen individual with JS / CSS / HTML experience (e.g. the predictor). Other parts like the payload/flight document and UKHAS string parsing will require additional thought.

Cheers,

Mark VK5QI

--
You received this message because you are subscribed to the Google Groups "UKHAS Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ukhas-tools...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ukhas-tools/DU2PR08MB7375CCACB55035D411A198AFBDC99%40DU2PR08MB7375.eurprd08.prod.outlook.com.

[Anthony Stirk]

Further to this mail unfortunately this date has been brought forward. We’ve slowly been retiring all the stuff in the rack over the last year and its got to the point where we’ve been asked to complete this by 2^nd of October.

 

We are attending 30^th September to complete this work. We’ll have the unit back in our offices so I can power them up to recover data etc from them but this will only be temporary and they won’t be in production.

 

Apologies its come to this but our hand has been forced (likely due to the electricity costs).

 

Anthony

 

Anthony Stirk​ Managing Director T: 01274 550911 E: Anthon...@nevis.co.uk  |  W: www.nevis.co.uk Nevis Computers is a limited company registered in England and Wales. Registered number: 3356647.   Registered office: York House, 16 Wool Gate, Cottingley Business Park, Bingley, West Yorkshire, BD16 1PE.    Emails may be monitored for the purposes of security and staff training

From: Anthony Stirk
Sent: 10 May 2022 09:47
To: ukhas...@googlegroups.com
Subject: XENC.HABHUB.ORG Sunset

[Steve Randall]

Thanks for letting us know Anthony - I had hoped for a little more time to investigate and suggest what folk should use as an alternative - but ultimately it doesn't matter as the servers will go anyway - just less time to the inevitable anyway.  I'm planning on putting a post to [UKHAS], [GPSL] & [picoballoon] unless anyone else wants to?

     Steve

Virus-free.www.avg.com

--

You received this message because you are subscribed to the Google Groups "UKHAS Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ukhas-tools...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ukhas-tools/DB9PR08MB9828554B4DA2980BF1CF08CEBD7B9%40DB9PR08MB9828.eurprd08.prod.outlook.com.

[VE6SRV]

There’s something missing that gets overlooked a lot, and that is a heartfelt thank you to everyone that has contributed over the years for making habhub available to all of us. 

A lot of this technology and the tools available can get taken for granted. We have some excellent tools available to us that are available for free. Some is putting in monetary and well as sweat equity to make these tools available. 

Just last weekend we flew a payload. I was able to pull up a webpage and enter data in a couple fields to get my burst predictions. When the balloon took off, I was able to get real time location information and live updated landing location information. 

The tracker failed at burst, but I was able to sit close to the predicted landing location, and as the payload got within 3 km, I was able to get the last 4 position reports before the payload landed. I was the only station to hear those packets. That was because I was in the right spot to hear them. Those live updated landing predictions made sure I was in range as the upper winds had changed slightly, pushing the payload a few miles west. The tracker stopped again upon landing. 

We recovered the payload because of the online live landing predictions. 

Be sure that the work that has been put into all of these online tools is highly appreciated!

To view this discussion on the web visit https://groups.google.com/d/msgid/ukhas-tools/CAHTNikhMTkidgVyy0mkNCWbVzMLG%2Bqd55Oh4-YkpM0D-e%2BshLg%40mail.gmail.com.

--

James
VE6SRV

[Anthony Stirk]

No issues from me. I know Mark has been working on implementing some of the functionality in sondehub and we’ve had a number of very kind offers of hosting it but before I let the hardware or the VM’s go I want to know its going somewhere that is part of the community and understand what they are undertaking.

 

Whilst its been a very easy child to baby sit due to the incredibly professional way it was put together by a group of unpaid amateurs but I suspect its next steps are going to be troublesome.

 

I’d be interested to know what something like this would cost to run under say AWS, I don’t have experience of this but I suspect give the resources it sucks up prices would be significant. 6-7 VM’s with 128Gb RAM between them with dual CPU’s (can’t remember now!) quite a bit of disk space and 30Gb a day transfer ?

 

Cheers

 

Anthony

 

Anthony Stirk​ Managing Director T: 01274 550911 E: Anthon...@nevis.co.uk  |  W: www.nevis.co.uk Nevis Computers is a limited company registered in England and Wales. Registered number: 3356647.   Registered office: York House, 16 Wool Gate, Cottingley Business Park, Bingley, West Yorkshire, BD16 1PE.    Emails may be monitored for the purposes of security and staff training

To view this discussion on the web visit https://groups.google.com/d/msgid/ukhas-tools/CAHTNikhMTkidgVyy0mkNCWbVzMLG%2Bqd55Oh4-YkpM0D-e%2BshLg%40mail.gmail.com.

[Mark Jessop]

As an update on where we are at with sondehub-amateur, compared to my last update (further up in this thread):

- predict.sondehub.org (and the tawhiri API For that matter) now has CSV and KML export support. It still needs prediction parameters in the URL to allow link sharing: https://github.com/projecthorus/leaflet_predictor/issues/1

- We now have a level of flight statistics and export at grafana.v2.sondehub.org.

  - There is a 'flight selector' dashboard at - https://grafana.v2.sondehub.org/d/VeLSoQqnz/flights?orgId=1 which lets users select recent flight data. Once selected, they end up at a dashboard like this one: https://grafana.v2.sondehub.org/d/HJgOZLq7k/basic?orgId=1&var-Payload=SP6ZWR-4FSK&time=1661601600000&time.window=21600000&from=1661590800000&to=1661626800000

- From here they can export data as CSV, KML or JSON (further down the page)

- We hope to add a link to the dashboard from the tracker site. 

The big functionality that we are missing for continuity of service is the UKHAS parser. It should be possible to have this as a standalone service, accepting data and forwarding it on to the sondehub-amateur database (the parser's output is already pretty close to what is required as input to sondehub), but to do this I think we're going to need help from the original authors, and I'm not sure how much time they have...

The other functionality required is pretty much there at amatuer.sondehub.org

As mentioned above, we have funding from ARDC - this will keep us going for the next 10 months, and they have indicated that we should 'think bigger' in our next grant request. There are also some other parties that have indicated they would like to donate, however there are some issues in actually obtaining those funds (mainly tax related...)

73

Mark VK5QI

To view this discussion on the web visit https://groups.google.com/d/msgid/ukhas-tools/DB9PR08MB9828CCEDB63A646B6B0F0DA8BD7B9%40DB9PR08MB9828.eurprd08.prod.outlook.com.

[David Akerman]

I'm working through my receiver programs, adding support for uploads of telemetry and listener data to the Sondehub/amateur system.  This makes it easy for LoRa (and other - see below) flights to be tracked on the Sondehub map without any reliance on the habhub servers.

Since I can't use payload documents without introducing a reliance on habhub, my software assumes that the UKHAS string is

$$PAYLOAD_ID,COUNTER,TIME,LATITUDE,LONGITUDE,ALTITUDE,.....

I think everyone does that so I don't see an issue.  Any other fields are ignored.

My Pi HAB LoRa Gateway does this now, in V1.9.0, obviously for LoRa flights only.  It's on github now.

My HAB Base - A windows program for managing one or more receivers - also does it from V1.6.1.  This is also on github now.  HAB Base can connect to various receivers including dl-fldigi so this gives another way to get RTTY trackers onto Sondehub.

I've added the same support to my Android apps but I have a small bit of work to do (supporting the latest Android version) before I release those into the Google Play store.

Dave

To view this discussion on the web visit https://groups.google.com/d/msgid/ukhas-tools/CABD2Lt3KzkzhA5Q00YOaBkr68NaEGD4yu634tk6ZYWckP49udw%40mail.gmail.com.