Programmatic download of SCHEDULE feed.

[Tim Millington]

Hi,

I'm trying to implement and regular download of SCHEDULE but can't seem to grab the feed programmatically.

I think my problem maybe in the redirect to the page:

   https://datafeeds.networkrail.co.uk/ntrod/spring_security_login

Which I get, even if specify the login credentials in a manual request string, as in: 

   https://<id>:<pwd>@datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full

Anyone had any success?

I am currently trying with PHP curl and using 

        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

I am aware this won't forwards the credentials on (any ideas?)

I have tried with the credentials in the url and also with:

        curl_setopt($ch, CURLOPT_USERPWD, $uid.":".$pwd);

[Jonathon Hurley]

Do you have to do this in PHP or do you have access to the command line?  If you do, then the command-line syntax for downloading via cURL is straightforward (see Jules Self's solution on the wiki at http://nrodwiki.rockshore.net/index.php/SCHEDULE#Downloading_via_curl).  

This could be set to run as a cron job (in Linux) or via some other task manager.

It also deals with the issue of the SCHEDULE file being big, and streams this to a file on disk.

What OS are you using?

Regards,

Jonathon

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send an email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Stuart Bridger]

I use cookies. Send j_username and j_password as POST parameters to the authentication URL. Save the cookies and then send a GET request to the file URL.

Will dig out my code from when I wrote this in PHP when I get home tomorrow if you haven't got it sorted by then.

[Stuart Bridger]

Probably not the most elegant of code but at the time I just wanted to get something working! Hope it helps!

<?php

class GetFileTask extends Shell {

  public function execute($type) {

    switch($type) {

      case 'full':

        $file_url = 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full';

        $output_filename = TMP.'cif_schedule_full.gz';

        break;

      case 'update':

        $day_string = strtolower(date('D'));

        $file_url = 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-'.$day_string;

        $output_filename = TMP.'cif_schedule_'.$day_string.'.gz';

        break;

      default:

        $day_string = $type;

        $file_url = 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_UPDATE_DAILY&day=toc-update-'.$type;

        $output_filename = TMP.'cif_schedule_'.$type.'.gz';

        break;

    }

    $auth_url = "https://datafeeds.networkrail.co.uk/ntrod/j_spring_security_check";

    $auth_params = 'j_username=*******&j_password=******';

    $output_file = fopen($output_filename, 'w');

    $json_filename = substr($output_filename, 0, -2).'json';

    $ch = curl_init();

    curl_setopt($ch, CURLOPT_COOKIEJAR, 'ntrod.txt');

    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

    curl_setopt($ch, CURLOPT_VERBOSE, true);

    curl_setopt($ch, CURLOPT_POST, true);

    curl_setopt($ch, CURLOPT_POSTFIELDS, $auth_params);

    curl_setopt($ch, CURLOPT_URL, $auth_url);

    curl_exec($ch);

    curl_setopt($ch, CURLOPT_HTTPGET, true);

    curl_setopt($ch, CURLOPT_URL, $file_url);

    curl_setopt($ch, CURLOPT_FILE, $output_file);

    curl_exec($ch);

    curl_close($ch);

    fclose($output_file);

    $this->uncompressFile($output_filename, $json_filename);

    return $json_filename;

  }

  private function uncompressFile($srcName, $dstName) {

    $sfp = gzopen($srcName, "rb");

    $fp = fopen($dstName, "w");

    while ($string = gzread($sfp, 4096)) {

      fwrite($fp, $string, strlen($string));

    }

    gzclose($sfp);

    fclose($fp);

  }

}

?>

[Tim Millington]

Thanks for the help.

Both Stuart's PHP solution and the command line solution work nicely, so I'm making progress. :)

Out of interest, how often do you guys download the full SCHEDULE as opposed to the updates?

[Phil Wieland]

On Thursday, July 11, 2013 2:24:34 PM UTC+1, Tim Millington wrote:

Out of interest, how often do you guys download the full SCHEDULE as opposed to the updates?

At the moment (Still developing.) only when I make a change to the database layout.  Once things are more stable I guess the answer will be "occasionally".

When the bugs I have reported are fixed (Activity information missing from schedule locations and Delete Association have cif_stp_indicator field empty.) I'll need a reload as well.

The associations in my database are in a right mess due to the bug, but I'm not using them for anything yet.  I also occasionally see a Delete Schedule which seems to match two schedules in my database but I haven't investigated that one yet so it's just as likely to be a bug at my end.

[Jonathon Hurley]

Good to hear you've managed to get the downloading working!

Stuart - how about adding the code to the PHP page on the wiki?

As with Phil, I only download it periodically at the moment whilst I'm playing with the data feeds. However, I think I'll end up downloading it daily as there are some STP schedules that affect freight, and I'm using the schedule data to interpret train movement messages. (The only downside of this is lots of SQL JOINs and headaches with optimising indexes :-( )

Best wishes,

Jonathon

--

[Stuart Bridger]

Will do - have just requested an account

[John Dickinson]

Hi Stuart

Bit of a long shot this as I know this post was live over a year ago.

I am trying to use your code to automatically download the CIF files (I can do it manually).

I'm a novice to PHP - have copied your code but struggling to make it work -my first error is................

Fatal error: Class 'Shell' not found in C:\xampp\htdocs\autopaupdate.php on line 82

If anybody can help would be obliged. Suspect when I get past the Shell error other errors will appear !!

Thanks

John

[Stuart Bridger]

That was part of a larger script...at the time I think I was using CakePHP which has a Shell script. It allows you to call the PHP script from the command line with arguements.

 

Try the code below, unfortuantely I'm at work so can't test it. Make sure you have cURL installed or it won't work.

 

<?php

  echo "Connecting to Network Rail Datafeeds and downloading full schedule";

 

  $file_url = 'https://datafeeds.networkrail.co.uk/ntrod/CifFileAuthenticate?type=CIF_ALL_FULL_DAILY&day=toc-full';
  $output_filename = TMP.'cif_schedule_full.gz';

 

  $auth_url = "https://datafeeds.networkrail.co.uk/ntrod/j_spring_security_check";
  $auth_params = 'j_username=*******&j_password=******';

 

  $output_file = fopen($output_filename, 'w');
  $json_filename = substr($output_filename, 0, -2).'json';

 

  $ch = curl_init();

  curl_setopt($ch, CURLOPT_COOKIEJAR, 'ntrod.txt');
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_VERBOSE, true);
  curl_setopt($ch, CURLOPT_POST, true);
  curl_setopt($ch, CURLOPT_POSTFIELDS, $auth_params);
  curl_setopt($ch, CURLOPT_URL, $auth_url);

 

  curl_exec($ch);

 

  curl_setopt($ch, CURLOPT_HTTPGET, true);
  curl_setopt($ch, CURLOPT_URL, $file_url);
  curl_setopt($ch, CURLOPT_FILE, $output_file);

 

  curl_exec($ch);

 

  curl_close($ch);
  fclose($output_file);

 

  uncompressFile($output_filename, $json_filename);

  echo "Full schedule downloaded and saved as "+$json_filename;

 

  private function uncompressFile($srcName, $dstName) {
    $sfp = gzopen($srcName, "rb");
    $fp = fopen($dstName, "w");
    while ($string = gzread($sfp, 4096)) {
      fwrite($fp, $string, strlen($string));
    }
    gzclose($sfp);
    fclose($fp);
  }

?>

Cheers, Stu

On Tuesday, 9 July 2013 18:24:11 UTC+1, Tim Millington wrote:

[John Dickinson]

Hi Stuart

Many thanks for taking the time to reply

Have tried out the code.

My first error was as below: 

Parse error: syntax error, unexpected 'private' (T_PRIVATE) in C:\xampp\htdocs\autopaupdate.php on line 113

On the back of the above I removed the private

When running again I got the following:-

Notice: Use of undefined constant TMP - assumed 'TMP' in C:\xampp\htdocs\autopaupdate.php on line 83
0 

Hence I put a ' and a ' around TMP to make 'TMP'

Now when I run I get (with the zero on the end)

Connecting to Network Rail Datafeeds and downloading full schedule0 

When I check my files I have the following files downloaded but both with zero bytes.

TMPcif_schedule_full.gz and TMPcif_schedule_full.json

I'm subscribed to the feed and also when I check PHPinfo() I do appear to have curl enabled

Any more help greatly appreciated

Many thanks

John

[Stuart Bridger]

You can just get rid of the TMP - that was just a constant to the system's temporary folder.

 

The '0' at the end of the line means that the code didn't get as far as initiating the $json_filename variable. I suspect there is a problem trying to open a file for write access (fopen).

 

Didn't get chance last night but should be able to give this a try on my server last night. I haven't done much with PHP for a while, I started to learn Java instead.

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.

To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Dickinson]

Thanks for the reply Stuart. I'll continue to have a play around with the code. PHP does seem to have some funny characteristics

[Stuart Bridger]

If you can modifiy your PHP.ini settings then check values for display_errors and error_reporting - it's worth turning these all the way up while you're writing / debugging code.

[John Dickinson]

Okay thanks Stuart

[Stuart Bridger]

Ran the script last night and it seemed to work fine.

 

I was running it on a linux server - not sure if there any additional issues that could be down to xampp. Setting display_errors to 1 or on should point you in the right direction.

[John Dickinson]

Hi Stuart

Success................

Looks like my XAMPP was struggling potentially with cookies. Instead used my hosting server to run the script and that worked.

Interestingly still get a zero value for echo "Full schedule downloaded and saved as "+$json_filename; and the script seems to display the Network Rail login page

However it does work so thanks very much for your time on this. I have now learnt about CURL and FOPEN

Thanks again

Regards

John

[Steve Macey]

Hi John/Stuart,

Sorry to dig up an old thread, but looking at the other threads I think the issue is with the ssl which can be fixed by adding this line:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

That seemed to fix the problem for me, and it gets rid of the zero, but I'm also getting the log in page and was wondering if this causes any issues and whether you figured out a way to stop it.

Thanks.

[Phil Davies]

For anyone who's tried this from Java, I managed it with HttpsURLConnection using Basic authorization on the initial open but no authentication on the redirect.

(I just caught the redirect as an exception on the initial open, probably not best practice).

I also had to set a lax HostnameVerifier on the initial connection.

-Phil