NRDP Phase 3 LIve

[David Higginbottom]

Dear NRDP User,

 

 Following on from the email that was sent yesterday regarding the launch of NRDP v3, this is to confirm that NRDP v3 is now live and can be accessed via https://opendata.nationalrail.co.uk/

 

There is also one correction to the email sent yesterday – see below (change highlighted in bold):

  

We have previously advised that before the end of February we would launch our new version of the National Rail Data Portal (NRDP v3).

We are pleased to announce that this is now ready to go live and will be launched on Thursday 21^st February at 10am (GMT).

 

The data services that will be included on launch will be:

• Knowledge Base
• Darwin v16
• Weekly feed of Fares, Routing guide and timetable data
• Historical Service Performance (HSP)

 

And these will be followed by:

• Connecting Train Identifiers (CTI)
• GPS data enhancement to Darwin

 

We would also like to bring to your attention the following: -

 

As a result of the changes being made for NRDP Phase 3, the way that Knowledgebase (KB) real-time incidents are consumed is changing.

 

Previously, messages were placed in the same messaging queue as Darwin push port messages.

In NRDP v3 this will change and the KB incident messages will be published on a separate message topic.

Users will be able to access the credentials and connection information necessary to subscribe to these messages via the updated NRDP web portal “My Feeds” page.

 

When existing NRDP users who consume real-time KB incidents are migrated, they will automatically be subscribed to the “Knowledgebase Real Time Incidents” feed.

 

Please note that NRDP v2 will still be in operation in parallel to NRDP v3 until the end of May 2019 and therefore you do not need to make any changes now to continue using the existing NRDP v2 subscriptions.

You will need to migrate to the new NRDP v3 subscriptions before the end of May 2019.

 

If you experience any issues please contact dsg_nrdp...@caci.co.uk

 

The v16 specification can be found here: https://wiki.openraildata.com/index.php?title=Darwin:Push_Port_XML_Schemas

 

Kind regards

 

David Higginbottom

Head of Support

Digital Systems Group

 

W: www.caci.co.uk

CACI Limited, 5th Floor, 8 St Paul’s Street, Leeds LS1 2LE

 

Follow @open_rail_feeds on twitter for service announcements and outage information

P  Please consider the environment before printing this e-mail

 

 

This electronic message contains information from CACI International Inc or
subsidiary companies, which may be confidential, proprietary,
privileged or otherwise protected from disclosure.  The information is
intended to be used solely by the recipient(s) named above.  If you are not
an intended recipient, be aware that any review, disclosure, copying,
distribution or use of this transmission or its contents is prohibited.  If
you have received this transmission in error, please notify us immediately
at postm...@caci.co.uk
Viruses: Although we have taken steps to ensure that this e-mail and 
attachments are free from any virus, we advise that in keeping with good 
computing practice the recipient should ensure they are actually virus free.

CACI Limited. Registered in England & Wales. Registration No. 1649776. CACI House, Avonmore Road, London, W14 8TS

[petermount]

Thanks David

On Thursday, 21 February 2019 10:38:19 UTC, David Higginbottom wrote:

Dear NRDP User,

 

 Following on from the email that was sent yesterday regarding the launch of NRDP v3, this is to confirm that NRDP v3 is now live and can be accessed via https://opendata.nationalrail.co.uk/

 

There is also one correction to the email sent yesterday – see below (change highlighted in bold):

  

We have previously advised that before the end of February we would launch our new version of the National Rail Data Portal (NRDP v3).

We are pleased to announce that this is now ready to go live and will be launched on Thursday 21^st February at 10am (GMT).

 

The data services that will be included on launch will be:

• Knowledge Base
• Darwin v16
• Weekly feed of Fares, Routing guide and timetable data
• Historical Service Performance (HSP)

 

And these will be followed by:

• Connecting Train Identifiers (CTI)
• GPS data enhancement to Darwin

 

We would also like to bring to your attention the following: -

 

As a result of the changes being made for NRDP Phase 3, the way that Knowledgebase (KB) real-time incidents are consumed is changing.

 

Previously, messages were placed in the same messaging queue as Darwin push port messages.

In NRDP v3 this will change and the KB incident messages will be published on a separate message topic.

Users will be able to access the credentials and connection information necessary to subscribe to these messages via the updated NRDP web portal “My Feeds” page.

 

When existing NRDP users who consume real-time KB incidents are migrated, they will automatically be subscribed to the “Knowledgebase Real Time Incidents” feed.

 

Please note that NRDP v2 will still be in operation in parallel to NRDP v3 until the end of May 2019 and therefore you do not need to make any changes now to continue using the existing NRDP v2 subscriptions.

You will need to migrate to the new NRDP v3 subscriptions before the end of May 2019.

 

If you experience any issues please contact dsg_nrd...@caci.co.uk

 

The v16 specification can be found here: https://wiki.openraildata.com/index.php?title=Darwin:Push_Port_XML_Schemas

 

Kind regards

 

David Higginbottom

Head of Support

Digital Systems Group

 

W: www.caci.co.uk

CACI Limited, 5th Floor, 8 St Paul’s Street, Leeds LS1 2LE

 

Follow @open_rail_feeds on twitter for service announcements and outage information

P  Please consider the environment before printing this e-mail

 

 

[Rob West]

Is it live live yet?

Getting this error message:

User DARWIN******-****-****-****-*****ffe3 is not authorized to read from: queue://darwin.pushport-v16

I'll try again later.

Rob

On Thursday, 21 February 2019 10:38:19 UTC, David Higginbottom wrote:

Dear NRDP User,

 

 Following on from the email that was sent yesterday regarding the launch of NRDP v3, this is to confirm that NRDP v3 is now live and can be accessed via https://opendata.nationalrail.co.uk/

 

There is also one correction to the email sent yesterday – see below (change highlighted in bold):

  

We have previously advised that before the end of February we would launch our new version of the National Rail Data Portal (NRDP v3).

We are pleased to announce that this is now ready to go live and will be launched on Thursday 21^st February at 10am (GMT).

 

The data services that will be included on launch will be:

• Knowledge Base
• Darwin v16
• Weekly feed of Fares, Routing guide and timetable data
• Historical Service Performance (HSP)

 

And these will be followed by:

• Connecting Train Identifiers (CTI)
• GPS data enhancement to Darwin

 

We would also like to bring to your attention the following: -

 

As a result of the changes being made for NRDP Phase 3, the way that Knowledgebase (KB) real-time incidents are consumed is changing.

 

Previously, messages were placed in the same messaging queue as Darwin push port messages.

In NRDP v3 this will change and the KB incident messages will be published on a separate message topic.

Users will be able to access the credentials and connection information necessary to subscribe to these messages via the updated NRDP web portal “My Feeds” page.

 

When existing NRDP users who consume real-time KB incidents are migrated, they will automatically be subscribed to the “Knowledgebase Real Time Incidents” feed.

 

Please note that NRDP v2 will still be in operation in parallel to NRDP v3 until the end of May 2019 and therefore you do not need to make any changes now to continue using the existing NRDP v2 subscriptions.

You will need to migrate to the new NRDP v3 subscriptions before the end of May 2019.

 

If you experience any issues please contact dsg_nrd...@caci.co.uk

 

The v16 specification can be found here: https://wiki.openraildata.com/index.php?title=Darwin:Push_Port_XML_Schemas

 

Kind regards

 

David Higginbottom

Head of Support

Digital Systems Group

 

W: www.caci.co.uk

CACI Limited, 5th Floor, 8 St Paul’s Street, Leeds LS1 2LE

 

Follow @open_rail_feeds on twitter for service announcements and outage information

P  Please consider the environment before printing this e-mail

 

 

[Allan Blundell]

Also can anyone access the FTP site? currently getting timeout errors. I expect some firewall errors are happening?

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Hicks]

Hi Rob

Advisory messages are disabled - I assume you're using OpenWire, so add 'jms.watchTopicAdvisories=false' like this:

  tcp://localhost:61616?jms.watchTopicAdvisories=false

I'll update documentation on the Wiki.

Peter

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

OpenTrainTimes Ltd. registered in England and Wales, company no. 09504022.

Registered office: 13a Davenant Road, Upper Holloway, London N19 3NW

[Peter Hicks]

Hi Allan

It works OK for me when *not* using passive-mode FTP - but when using passive mode, the server-side port isn't translated (Amazon EC2 makes extensive use of address translation):

ftp> passive

Passive mode on.

ftp> ls

227 Entering Passive Mode (10,2,1,164,39,116).

[ freezes ]

Can anyone confirm if this is the same on datafeeds.nationalrail.co.uk?

Peter

[Rob West]

Cheers Peter,

I was actually trying with Stomp, I'll try OpenWire later. The watchTopicAdvisories setting doesn't seem to make a difference, but its early days.

Rob

[Allan Blundell]

Thank You, changing to Active Mode made it work.

Can connect passively with datafeeds.nationalrail.co.uk

[Rob West]

OK I now have it working. My mistake was fairly trivial, with the old push port I had to subscribe to the queue, now its a topic. 

So in stomp the subscribe is like this:

connection->subscribe('/topic/darwin.pushport-v16')

rather than

connection->subscribe('/queue/<guid>')

as it was before.

Rob

[Rail Ale Fan]

Thanks all!

Looking good so far.  Using my existing v12 startup (timetable+snapshot+catchup) and then switching to v16 for live no code changes required at all.

However regarding the timetable / reference data, is there a good reason why this isn't on the FTP server?

And if not, could it be ? :)

Cheers!

On Thursday, February 21, 2019 at 10:38:19 AM UTC, David Higginbottom wrote:

Dear NRDP User,

 

 Following on from the email that was sent yesterday regarding the launch of NRDP v3, this is to confirm that NRDP v3 is now live and can be accessed via https://opendata.nationalrail.co.uk/

 

There is also one correction to the email sent yesterday – see below (change highlighted in bold):

  

We have previously advised that before the end of February we would launch our new version of the National Rail Data Portal (NRDP v3).

We are pleased to announce that this is now ready to go live and will be launched on Thursday 21^st February at 10am (GMT).

 

The data services that will be included on launch will be:

• Knowledge Base
• Darwin v16
• Weekly feed of Fares, Routing guide and timetable data
• Historical Service Performance (HSP)

 

And these will be followed by:

• Connecting Train Identifiers (CTI)
• GPS data enhancement to Darwin

 

We would also like to bring to your attention the following: -

 

As a result of the changes being made for NRDP Phase 3, the way that Knowledgebase (KB) real-time incidents are consumed is changing.

 

Previously, messages were placed in the same messaging queue as Darwin push port messages.

In NRDP v3 this will change and the KB incident messages will be published on a separate message topic.

Users will be able to access the credentials and connection information necessary to subscribe to these messages via the updated NRDP web portal “My Feeds” page.

 

When existing NRDP users who consume real-time KB incidents are migrated, they will automatically be subscribed to the “Knowledgebase Real Time Incidents” feed.

 

Please note that NRDP v2 will still be in operation in parallel to NRDP v3 until the end of May 2019 and therefore you do not need to make any changes now to continue using the existing NRDP v2 subscriptions.

You will need to migrate to the new NRDP v3 subscriptions before the end of May 2019.

 

If you experience any issues please contact dsg_nrd...@caci.co.uk

 

The v16 specification can be found here: https://wiki.openraildata.com/index.php?title=Darwin:Push_Port_XML_Schemas

 

Kind regards

 

David Higginbottom

Head of Support

Digital Systems Group

 

W: www.caci.co.uk

CACI Limited, 5th Floor, 8 St Paul’s Street, Leeds LS1 2LE

 

Follow @open_rail_feeds on twitter for service announcements and outage information

P  Please consider the environment before printing this e-mail

 

 

[Peter Hicks]

Hello

It's available in an Amazon S3 bucket instead.  You'll get a Push Port message when the new timetable and reference data is published, and you can pick them up directly without having to poll an FTP server in the hope there's new data available.

Peter

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rail Ale Fan]

Hi Peter,

It's just a shame that S3 support has been made a requirement for something that could quite easily be provided over FTP.

As Push Port is now indicating that a new timetable has been published there's no need to poll FTP, S3 doesn't add anything in this scenario and just creates an additional propitiatory protocol support requirement on the client....

Cheers!

[Peter Hicks]

Hello

There are two issues here:

1. There will be a period of time from the Push Port sending out the "new data available" message and NRDP downloading it and copying it on to an FTP server.  If the timetable/reference data message is passed on to users, they'll try to download a file that may not yet exist - and because the messages are sequenced, either the flow of messages will need to be stopped whilst NRDP downloads the files and puts them on the FTP server, or the "new data available" message needs to be re-sequenced, which would be a bit messy

2. If everyone's automating the process of downloading a new timetable when it gets published, there will be at least one connection to the FTP server per user to download the file, and I doubt it's going to be able to handle several hundred connections at once!  Having the data on S3 means Amazon can deal with that problem

Thinking more about it, the data in the S3 bucket could be made available via HTTP without the contents of the bucket being viewable - but I'm not sure what wider implications this could have for other users of Darwin.

Peter

--

You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rail Ale Fan]

Hi David,

Please can you confirm that v16 functionality is frozen and that we have no choice but to implement an Amazon S3 client in order to access timetable / reference data?

Thanks

On Thursday, February 21, 2019 at 10:38:19 AM UTC, David Higginbottom wrote:

Dear NRDP User,

 

 Following on from the email that was sent yesterday regarding the launch of NRDP v3, this is to confirm that NRDP v3 is now live and can be accessed via https://opendata.nationalrail.co.uk/

 

There is also one correction to the email sent yesterday – see below (change highlighted in bold):

  

We have previously advised that before the end of February we would launch our new version of the National Rail Data Portal (NRDP v3).

We are pleased to announce that this is now ready to go live and will be launched on Thursday 21^st February at 10am (GMT).

 

The data services that will be included on launch will be:

• Knowledge Base
• Darwin v16
• Weekly feed of Fares, Routing guide and timetable data
• Historical Service Performance (HSP)

 

And these will be followed by:

• Connecting Train Identifiers (CTI)
• GPS data enhancement to Darwin

 

We would also like to bring to your attention the following: -

 

As a result of the changes being made for NRDP Phase 3, the way that Knowledgebase (KB) real-time incidents are consumed is changing.

 

Previously, messages were placed in the same messaging queue as Darwin push port messages.

In NRDP v3 this will change and the KB incident messages will be published on a separate message topic.

Users will be able to access the credentials and connection information necessary to subscribe to these messages via the updated NRDP web portal “My Feeds” page.

 

When existing NRDP users who consume real-time KB incidents are migrated, they will automatically be subscribed to the “Knowledgebase Real Time Incidents” feed.

 

Please note that NRDP v2 will still be in operation in parallel to NRDP v3 until the end of May 2019 and therefore you do not need to make any changes now to continue using the existing NRDP v2 subscriptions.

You will need to migrate to the new NRDP v3 subscriptions before the end of May 2019.

 

If you experience any issues please contact dsg_nrd...@caci.co.uk

 

The v16 specification can be found here: https://wiki.openraildata.com/index.php?title=Darwin:Push_Port_XML_Schemas

 

Kind regards

 

David Higginbottom

Head of Support

Digital Systems Group

 

W: www.caci.co.uk

CACI Limited, 5th Floor, 8 St Paul’s Street, Leeds LS1 2LE

 

Follow @open_rail_feeds on twitter for service announcements and outage information

P  Please consider the environment before printing this e-mail

 

 

[Rail Ale Fan]

Hi all,

I've managed to get S3 access to the timetable / reference data working but only after a lot of trial and error (Amazon Web Services newbie) as it doesn't look like the details provided under the "Darwin File Information" section of the My Feeds page is consistent with AWS terminology.

The parameters provided are "S3 URL", "S3 Bucket" and "Region" (eu-west-1).

The value for "S3 URL" excluding s3:// is actually the bucket name (below as $bucket).

The value for "S3 Bucket" is actually just part of the object key, to be prefixed along with "/" to the timetable or reference filename (below as $prefix).

As eu-west-1 is not the AWS default for s3.amazonaws.com, it must be specified using the sub-domain s3-eu-west-1.amazonaws.com

As $bucket contains a period it cannot be used as a sub-domain of s3-eu-west-1.amazonaws.com (wildcard SSL issue).

[https URLs below intentionally broken with "http_"]

So the current object key list

http_://s3-eu-west-1.amazonaws.com/$bucket/?list-type=2

And today's timetable and reference for example;

http_://s3-eu-west-1.amazonaws.com/$bucket/$prefix/20190225020746_v8.xml.gz

http_://s3-eu-west-1.amazonaws.com/$bucket/$prefix/20190225020746_ref_v3.xml.gz

Hope this helps save other AWS newbies a bit of time!

Cheers

[petermount]

The URL they provide is technically correct, just depends on the client in use. The official ones & the better 3rd party ones will support it.

For example using s3cmd it will accept their url as long as you specifically set the region.

So the following line does work:

s3cmd --access_key=##### '--secret_key=#####' --region=eu-west-1 get s3://darwin.xmltimetable/PPTimetable/20190226020747_v8.xml.gz

Peter

[Olivier Bohrer]

Thank you Rob it really helped! This is indeed one of the differences when switching to the new endpoint.

Here is a gist of a consumer which works with Camel & ActiveMQ: https://gist.github.com/obohrer/a6ce492adb2464a30f1bea2599684236

It contains a few tricks which caused pain while migrating (queue --> topic and the disabling the advisory topic).

[UEA Network Rail Study]

Hi, I came across this message and thought I'd post this information, for posterity / anyone who will need this info:

https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/

Amazon is downgrading/deemphasizing the path style URLs that Rail Ale Fan describes.  They will be moving to virtual-host type URLs, e.g.

https://$bucket.s3.amazonaws.com/$prefix/$file

I see now in the blog post they are NOT going to discontinue support for path style URLs for buckets made before Sept 2020 (unlike when I first read about this).

But buckets made after that date will only support virtual-host URLs

I recently tried using the virtual-hosts style in accessing the DARWIN S3 PPTimetable files, but it wasn't working - I think it was a problem with the client I was using. In any case, I will be switching to virtual-host at some point just in the interests of being fastidious/neurotic

Doug

[Rail Ale Fan]

Hi Doug,

The virtual-hosts style of access doesn't support bucket names containing period so darwin.xmltimetable can't be used in that mode - it is mentioned in the article you linked to (some mind boggling stats there!) but it does say that Amazon are working on it so I don't think it's any problem with your client...

Cheers

[UEA Network Rail Study]

Ach! I should have read your message (better).  Yes, the bucket is really darwin.xmltimetable....  not PPTimetable (which I was trying).  

They really do need to update the "My Feeds" page on the portal to be correct.

doug

[Nick Lamb]

Hi i’m Struggling with the amazon bucket
I’ve retested as normal with nationa rail
I have an acccess key & a secret key and I’d like to access a single file to download automatically however I cant figure out how to construct the URL

I’ve tried s3-eu-west-1.amazonaws.com/Darwin.xmltimetabe/?list-type=2 which gives me an access denied message (expected as I don’t add my credentials)
Ideally I want to code something in python using a URL with my keys - I want to avoid BOTO as this requires a lot of messing with my web server which is stable and I don’t want to jinx it

Is there a simple way to code the URL and file name ?
My current process is to use FTP and unzip the file and read the XML file, but I’ve desovered the new file refreshes at 10am and this could be problematical as I’m going to mis some data unless I time by job to pull the file just before its refreshed
Any guidance to get me on the the amazon bucket would be most helpful

[Peter Hicks]

Hi Nick

I think you have to use an S3 client rather than an HTTP(S) connection - at least, I've never accessed a non-public S3 bucket using HTTP.

What's the problem with using Boto?  I've install v2.49 on a machine here with "pip install boto", and it installed in a couple of seconds without pulling in any other dependencies, or seeming to need a C/C++ compiler to create and libraries.  If your web server is that sensitive, maybe you want to look at using a separate server, or a local virtual machine for your testing.

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To post to this group, send an email to openrail...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/ab7d40c8-07f2-45de-8989-4531ea031b6f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nick Lamb]

Thanks for replying so quick, my nervousness is that to use pip on my 1&1 server I need to upgrade my phyhon install and as my website is built using a number of python scripts as is the backbone that builds the database I don’t want to temp fate, the only need I have for this file is to build a list of rids,uids,stations & departure times. These drive other processes to get the train running data
I may take the plunge with the install when I’m feeling a little braver 😀

[Chris Northwood]

Nick, you might find the S3 docs useful which show you how to access it directly: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectGET.html. There are links that show you how to construct the correct authentication headers, as well as a correct URL (you need to include the bucket name in the hostname too)

On Tue, Jun 18, 2019 at 8:58 PM Nick Lamb <nickl...@gmail.com> wrote:

Thanks for replying so quick, my nervousness is that to use pip on my 1&1 server I need to upgrade my phyhon install and as my website is built using a number of python scripts as is the backbone that builds the database I don’t want to temp fate, the only need I have for this file is to build a list of rids,uids,stations & departure times. These drive other processes to get the train running data
I may take the plunge with the install when I’m feeling a little braver 😀

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send an email to openrail...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/a1675e9b-ff44-4d11-afda-f212f7338761%40googlegroups.com.