Struggling to curl CORPUS with PHP

[Kieran Fletcher]

Hi there,

I'm currently trying to automate CORPUS imports into a mysql database.

Annoyingly, it seems to always fail logging in, even though I can easily download via the web using the exact same credentials.

Attached is the code and the result, entering the username and password in the response fails as it's denied by my web server.

Thank you,

Kieran

[Peter Hicks]

Hi Kieran

> On 20 Sep 2022, at 20:02, Kieran Fletcher <kierana...@gmail.com> wrote:
>
> I'm currently trying to automate CORPUS imports into a mysql database.
> Annoyingly, it seems to always fail logging in, even though I can easily download via the web using the exact same credentials.
>
> Attached is the code and the result, entering the username and password in the response fails as it's denied by my web server.

I don’t know how PHP handles things, but when you use curl and supply the username and password separated by a colon, it’ll add an Authorisation header with the value set to ‘Basic’ and a base64-encoded string of the username, a colon and the password. Is the curl extension for PHP doing this? Can you get curl under PHP to send the equivalent of debugging information somewhere?

Also as a side-note, you probably don’t want to disable CURLOPT_SSL_VERIFYPEER or CURLOPT_SSL_VERIFYHOST because they sound like they’ll disable the protection you get from using SSL/TLS.


Peter

[Ben]

Hi Kieran,

You will need to define the "Cookie Jar" to allow NROD to set the session cookie and redirect to the AWS S3 bucket. This should help you out.

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, 'https://datafeeds.networkrail.co.uk/ntrod/SupportingFileAuthenticate?type=CORPUS');
    curl_setopt($ch, CURLOPT_VERBOSE, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    // update here per requirements... jobs/threads....
    curl_setopt($ch, CURLOPT_COOKIEJAR, tempnam('/tmp', 'nrod-cookie'));
    curl_setopt($ch, CURLOPT_USERPWD, $username . ":" . $password);
    $result = curl_exec($ch);
    curl_close($ch);

    $json = gzdecode($result);

    var_dump(json_decode($json, true));

Ben