NROD & Darwin archives now online

petermount

unread,

Jan 23, 2023, 6:31:49 PM1/23/23

to A gathering place for the Open Rail Data community

I've finally found some time to sort the online archives I generate from the various feeds and they are now available at https://archive.area51.dev/

You can either download files individually or batch them up into a single download.

As this is hosted on a server located at home it doesn't have any egress charges that the original S3 one had (although that still exists for now).

Please note: this is still a work in progress as I need to finish off the house keeping side of things, which might mean some downtime - I'll try to keep that to a minimum.

Let me know if you hit any problems with it.

Peter

Legolash2o

unread,

Jan 25, 2023, 8:32:06 AM1/25/23

to A gathering place for the Open Rail Data community

That's great news, do you have any plans to share VSTP messages please?

Since I have a big collection of schedule data as well, there might be a way to collaborate to spread the costs between different mirrors, or some other means.

petermount

unread,

Jan 25, 2023, 9:10:53 AM1/25/23

to A gathering place for the Open Rail Data community

At the moment it has everything I collect but as I need to redo my connections (the extra 4 weeks will be handy here as this is the busiest time of the year for me work wise), I'll be trying to include those and a few other feeds as well.

Douglas Fraser

unread,

Jan 25, 2023, 9:35:09 AM1/25/23

to A gathering place for the Open Rail Data community

Thanks, Peter. Any possible ETA on when you'll turn off the S3 service?

Doug

Martin Thorpe

unread,

Jan 25, 2023, 9:41:55 AM1/25/23

to openrail...@googlegroups.com

Another vote of thanks from me too — I downloaded a lot of the TD messages last night although there were some batches that seemed to be missing. Peter, shall I email you directly with what I couldn’t retrieve?

I have a 200Mbit/s symmetric connection at home (with no upload quota) and plenty of disk space so I’d be happy to host files if that would be helpful.

Martin

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/eef21993-fbdd-4ffe-9c1f-7ae67327e7a0n%40googlegroups.com.

Douglas Fraser

unread,

Jan 25, 2023, 9:44:18 AM1/25/23

to A gathering place for the Open Rail Data community

I have been collecting NR data for a few years now - here at UEA we are using DARWIN data in the AI module (prediction of arrival delays) and 'theoretically' some research ... but it's been a slog. Anyways, I have loads of it, pretty much everything, and am in the midst of making ORC files for Hive based tables instead of these giant blobs of XML...

Given UEA is a university and is supposed to act in the public good, I will see what I can do to possibly open this up to the public (non-commercial, of course) or contribute. So any effort that springs up, please let me know. Awhile back we talked with some people working for the RDG (I think) who are creating a Data Marketplace, but I don't recall any more details

Doug

Legolash2o

unread,

Jan 25, 2023, 10:32:25 AM1/25/23

to A gathering place for the Open Rail Data community

When at some point you need an archive of the other feeds, let me know and I can share. :)

petermount

unread,

Jan 25, 2023, 1:14:39 PM1/25/23

to A gathering place for the Open Rail Data community

S3 will remain for some time as it's the main backup of the archive.

It'll probably become private at some point but not for a couple of months at least until I've got this one stable.

petermount

unread,

Jan 25, 2023, 1:16:18 PM1/25/23

to A gathering place for the Open Rail Data community

Email me directly with which ones you had trouble downloading - it might be an index problem which I did try to fix yesterday so might be related.

If you still have the link (or the UID in the link) handy that will be useful

RailAleFan

unread,

Jan 26, 2023, 7:27:30 AM1/26/23

to A gathering place for the Open Rail Data community

Things seem to have gone a bit quiet on the RDM front but it sounds like a possible channel to publicise curated data sets that could save a lot of wheel re-invention.

Since publishing nodeDarwinArchiver I have archived almost the entire Summer 2022 timetable period in a very easy to consume format (JSON encoded OpenLDBSVWS response objects) - most likely useful for summary analytics but also straight forward to import and generate historic departure boards. I have a few dates uploaded here which I created out of interest to see how the disruption due to UK record high temperatures on 19th July played out across the network;

https://railalefan.co.uk/labs/da-browser.php

Cheers

On Wednesday, January 25, 2023 at 2:44:18 PM UTC sye1...@gmail.com wrote:

Hayden Sutherland

unread,

Jan 30, 2023, 9:56:11 AM1/30/23

to A gathering place for the Open Rail Data community

Doug

The RDG Rail Data Marketplace could be the place to make this data available...
Email the team at raildatam...@raildeliverygroup.com to discuss further