Using Node.js to connect to a feed

[Steven Collins]

Hi all,

I am new around here and have gotten stuck accessing any of the feeds. I am attempting to connect to the darwin queue that I have subscribed to however aver many attempts and variations I am still unable to authenticate. I have tried multiple node packages (stomp-client, stompit and stomp) however they either time out, do not connect or say that my username as password are not correct. 

I tried to following multiple wiki pages and multiple code examples however i still get the same result.

Which username and password am I meant to be using as I believe I have tried all the one listed on the data portal as well as my own login credentials?

thanks in advance 

[Peter Hicks]

Hello

Use d3user and d3pass (or password, check what it says on the portal).  Your queue should start with /queue/d3...

What error message are you getting?

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steven Collins]

Hi,

I am currently getting a "connect ECONNREFUSED 54.247.175.93:61613"

thanks,

[Peter Hicks]

Hi Steven

You're connecting to the wrong service - try datafeeds.nationalrail.co.uk... I think you're using datafeeds.networkrail.co.uk!

Peter

[Steven Collins]

Hi Peter,

Thank you that worked like a charm :) why do they have to be similar names :/

I now have a body of data which is compressed so i am looking into how to decompress it so i can acualy use the data.

thanks,

[Peter Hicks]

Hi Steven

You'll probably need to go back in time and ask the marketing/branding people :-)

Which language are you using?  There's gzip/deflate support for most languages.

Peter

[Allan Blundell]

Use zlib library that is part of node js
https://nodejs.org/api/zlib.html should help you inflate the data to xml so it can be read

[Steven Collins]

HI,

I have been attempting to inflate the data using zlib without much success, i am probably just using it wrong

[Steven Collins]

Ok, so i am having trouble inflating the stream to get any readable data out (sorry for all to problems and questons :s )

I am currently using Node.JS and in the process of connecting to these services I am making a node package to do it.

I am currently using stomp-client npm package in order to connect to the queue i am running this code:

    const destination = '/queue/queuename';

    const client = new StompClient('datafeeds.nationalrail.co.uk', 61613, 'user', 'password');

    console.log('connecting to DARWIN');

    client.connect(function(sessionId) {

      console.log('Trying to connect...');

      client.subscribe(destination, function (body, headers) {

        console.log('decompressing stream');

        console.log(zlib.Inflate(body));

      });

    });

70% of the time it will run the subscribe function and return data which is a complete mess (presumably compressed) or it just hangs trying to connect.

eitherway when i run the zlib.Inflate function i get a "Unrecognized Command ''" error but i am not sure as to what is causing this...

[Allan Blundell]

Try using the unzip command that is mentioned in the documentation im sure with a quick read of the docs inflate isnt for gzip, unzip is

Allan

[Allan Blundell]

var zlib = require('zlib');

console.log(zlib.createUnzip().pipe(body));

I'm not a JavaScript expert but try that?

[Allan Blundell]

Or This..

var zlib = require('zlib');




zlib.Unzip(body,function(error,result){

if(error){

console.log("Error Unzipping");

}

else{

console.log(result);

}

});

[Steven Collins]

Hello,

I have been trying to run this for a few hours now,

the code apparantly connects to the server however hangs when attempting to get a subscription. Is there  limit on how may previous subscriptions (from testing :s) there can be?

[Allan Blundell]

As far as i know no, you can reconnect and disconnect as much as you like however you don't want to be doing that every second.

Is your subscribe function still the same?

[Steven Collins]

Hi,

I belive the subscribe function is the same, Im now currently getting this error:

"User d3user is not authorized to create: queue://queue name" 

does this mean that my queue does not exist on the server?

I am using the subscribe function which is as follows: 

  connectToQueue: (queue, callback) => {

    const dest = `/queue/${queue}`;

    client = new StompClient('datafeeds.nationalrail.co.uk', 61613, 'd3user', 'd3password');

    console.log('connecting to datafeeds.nationalrail.co.uk:61613 using d3user:d3password');

    

    client.connect(function(sessionId) {

      console.log('Connected to datafeeds.nationalrail.co.uk:61613');

      console.log(`Trying to subscribe to ${dest}`);

      client.subscribe(dest, (body, headers) => {

        console.log('Message received from subscription... Decompressing message...');

        zlib.Unzip(body, (err, result) => {

          if (err) {

            console.log('Error decompressing message');

          } else {

            callback(result);

          }

        });

      });

    });

  }

[Steven Collins]

Hello again :)

so i do think that there is something wrong with the queue as i have created a new queue and the function is now getting data (although its now crashing when it attempts to decompress)

[Allan Blundell]

Okay that's good, sorry i didn't respond sooner been looking at your code and i could only think the queue name was invalid etc, can you break point the code a line before and step into the decompress function and see if any of the variables are null?

[Steven Collins]

Yea so i have a new queue name which i can receive messages on however they are either compressed or garbled. 

I have tried piping it into the unzip function mentioned earlier but i get errors along the lines of 

Unrecodnised command:'� �W]��0 �+���

H��� �E!���H�쒸B�����* ݥޅ���4���33�c >�y?r a�E�iă��s�i ���������? C'�S� �S6��0NtA8��}� ��5�󤿏߅ �Hp���!�� �|qH?eR���8

�A�3�� ` �@鉃 �钤 � TQV4�� t�Ю��$ c��\� �i� �o ��qit(� h�*BY��,;�ax.��m�]cH Эx ` ���qįmw�qw<w� JX ��k d ��Nr�����p�> ��|��1_�s���<�

i(�nF�,c���K I�4�

�F��stz of-]ϧ�H�$���߰�)���\Y�g�'

so i am not sure what to do now...

[Steven Collins]

Ok so after some looking into it that initial string does form the start of a .gz so i am going to take a guess that the app is bombing out before the full message has arrived....

[Allan Blundell]

Use Gunzip not Unzip sorry my fault

[Steven Collins]

Hi, I have tried replacing it to Gunzip however i think there may be an error occuring from the stomp-client npm package :/

I have attached a screenshot. 

The call stack seems to be pointing to various function most notibaly the StompFrameEmitter function (not sure whats happening here)

Any ideas?

[Peter Hicks]

Hi Steven

Could you post the script you're using here so somebody else familiar with Node.JS can take a look?  Remember to take out your queue name!

Peter

[Steven Collins]

Hi Peter,

I have previously posted it but i shall post it again :)

connectToQueue: (queue, callback) => {

    const dest = `/queue/${queue}`;

   client = new StompClient('datafeeds.nationalrail.co.uk', 61613, 'd3user', 'd3password');

   console.log('connecting to datafeeds.nationalrail.co.uk:61613 using d3user:d3password');

   

   client.connect(function(sessionId) {

     console.log('Connected to datafeeds.nationalrail.co.uk:61613');

     console.log(`Trying to subscribe to ${dest}`);

      client.subscribe(dest, (body, headers) => {

       console.log('Message received from subscription... Decompressing message...');

        zlib.Gunzip(body, (err, result) => {

[Peter Bienek]

Hi Steven, 

I've also been having similar issues with the stomp-client for node. So I decided to switch to stompit instead.

See my solution below.

"use strict";
const stompit = require('stompit');
const zlib  = require('zlib');
const connectOptions = {
    'host': 'datafeeds.nationalrail.co.uk',
    'port': 61613,
    'connectHeaders':{
        'host': '/',
        'login': 'd3user',
        'passcode': 'd3password',
        'heart-beat': '5000,5000'
    }
};

stompit.connect(connectOptions, function(error, client) {

    if(error){
        console.log('Unable to connect: ' + error.message);
        return;
    }

    const subscribeParams = {
        'destination': '/queue/QUEUE_NAME_HERE',
        'ack': 'client-individual'
    };


    client.subscribe(subscribeParams, function(error, message){

        const read = function(){
            let chunk;
            while(null !== (chunk = message.read())){
                zlib.gunzip(chunk, function (error, response) {

                    if (error) {
                        console.log(message.headers)
                        console.log(error)
                    } else {
                        console.log(message.headers.FilterHeaderLevel)
                        console.log(response.toString());
                    }
                });
            }
        };

        message.on('readable', read);

        message.on('end', function(){
            client.ack(message);
        });
    });
});

[Steven Collins]

I shall have a look at this tomorrow morning :)

I have been doing some other coding and have managed to get a full node solution for connecting to the networkrail datafeeds with no problems, it does seem to be an issue with the data being gziped from national rail before it is sent...

While I am coding and learning this I am making it into a couple of npm packages to save some other people the potential problems and instead just install a node module :)

[Steven Collins]

Just a quick update, I have now managed to connect to Darwin using Peters code :) worked like a charm. 

I'm currently rounding off the npm packages to connect to Trust and Darwin so if anyone wants to give them a test then please let me know :)