Empty Push Port Queue?

[Erin Morrow]

Hello,

I'm new to the open rail data scene and am attempting to setup connections to both National and Network Rail feeds. I am using Python and stomp.py to build my client setup. So far I have had good success connecting to Network Rail's TD feed and am getting the data I'm looking for from my queue.

I have had less success in getting data from the National Rail Push Port feed. Building on the example code from the OpenRailData wiki (on Github) my client appears able to connect but my message queue never seems to provide any messages. I have verified that under my profile on the National Rail Data Portal that I am subscribed to all data sets and all TIPLOCs. I have also verified via the FTP logs that messages (quite a lot of them!) were being sent during the times I was connected. Has anyone seen something similar to this? I suspect I am missing some minor but crucial detail. My code is provided below along with some sample output. (Times are in GMT-5)

Thank you,
Erin

import stomp, gzip, time, xml
from io import StringIO
from datetime import datetime

class MyListener(stomp.ConnectionListener):

        msg_list = []
           
        def on_error(self, headers, message):
                self.msg_list.append('(ERROR) ' + message)
                print('received an error %s' % message)

        def on_message(self, headers, message):
                fp = gzip.GzipFile(fileobj = StringIO.StringIO(message))
                text = fp.readlines()
                fp.close()
                self.msg_list.append(text)
                print('%s\n' % text)


#Conection configuration for Network Rail
conn = stomp.Connection([('datafeeds.nationalrail.co.uk', 61613)])
myListener = MyListener()
conn.set_listener('', myListener)
conn.start()
#Submit credentials
print("Connecting...")
conn.connect("d3user", "d3password", wait=False)
#Access data feed subscription
print("Subscribing to queue...")
conn.subscribe(destination="/queue/<myQueueID", id=1, ack='auto')

#Listening to queue for specified conditions
print("Listening...")
t = 0
while len(myListener.msg_list) < 1:
    print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), " - ", len(myListener.msg_list), " messages")
    try:
        time.sleep(10)
        t = t + 10
    except KeyboardInterrupt:
        print("User requested program termination")
        if conn.is_connected():
            print("Connection is live")
            conn.disconnect()
            conn.stop()
            if conn.is_connected():
                print("Connection is still live")
            else:
                print("Connection terminated by user...")
        break
   
#Deal with collected messages
print(len(myListener.msg_list), " messages collected in ", t, " seconds")
messages = myListener.msg_list
print(myListener.msg_list)

#Clean up connection if necessary
if conn.is_connected():
    print("Connection persisted until end of program")
    conn.disconnect()
    conn.stop()
    if conn.is_connected():
        print("Connection is still live")
    else:
        print("Connection terminated automatically at close")
else:
    print("Connection terminated automatically prior to close")

#Be polite    
print("Goodbye")

Connecting...
Subscribing to queue...
Listening...
2016-04-19 16:55:58  -  0  messages
2016-04-19 16:56:08  -  0  messages
2016-04-19 16:56:18  -  0  messages
2016-04-19 16:56:28  -  0  messages
2016-04-19 16:56:38  -  0  messages
...
2016-04-19 17:06:48  -  0  messages
2016-04-19 17:06:58  -  0  messages
2016-04-19 17:07:08  -  0  messages
2016-04-19 17:07:18  -  0  messages
User requested program termination
0  messages collected in  680  seconds
[]
Connection terminated automatically prior to close
Goodbye

[Peter Hicks]

Hi Erin

Can you email me privately with your queue ID?  I can have a look at what's going on from a known-good client here.

There are some cases where queues have been created but no data is actually put on them :(

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Peter Hicks]

Hi Erin

I've connected to your queue from a test client here, and I see data.

Are you sure your client is connecting?  If you're on a corporate network, you might have outbound access to TCP port 61613 blocked - the Network Rail feeds use 61618.

If you're on a UNIX system, or something with a telnet client installed, try connecting to datafeeds.nationalrail.co.uk port 61613.  You should at least get the connection opening within a second or two and see no data.  If the connection times out, or fails, you may need to get in touch with your IT folks.

Peter

On Tue, 19 Apr 2016 at 22:19 Erin Morrow <Erin....@arup.com> wrote:

Peter,

 

That was very quick! My queue ID is <<REMOVED>>.

 

Thank you for your help,

Erin

____________________________________________________________
Electronic mail messages entering and leaving Arup  business
systems are scanned for acceptability of content and viruses

[erin....@arup.com]

Hello,

I'm new to the open rail feeds scene and am working my through some basic connection issues using Python and stomp.py. So far I have successfully connected to the NetworkRail TD feed and am getting the data I'm looking for.

I am having less success with the National Rail Push Port feeds. My client appears to be connecting properly but I never seem to receive any messages in my queue. According to my National Rail Data Portal page I have subscribed to all data types and TIPLOCs. I've verified that there were messages (quite a lot in fact) during the times I was connected so I'm not sure what's going on. Has anyone else experienced this? For reference my python code is below and is adapted from the OPenRailData wiki example python code.

conn.subscribe(destination="/queue/<myqueueID>", id=1, ack='auto')

#Listening to queue for specified conditions
print("Listening...")
t = 0
while len(myListener.msg_list) < 1:
print(datetime.strftime(datetime.now(), '%Y-%m-%d %H:%M:%S'), " - ", len(myListener.msg_list), " messages")
try:
time.sleep(10)
t = t + 10
except KeyboardInterrupt:
print("User requested program termination")
if conn.is_connected():
print("Connection is live")
conn.disconnect()
conn.stop()

[Erin Morrow]

Thank you Peter. It's good to rule something out. I have just successfully tried telnet on the same wireless network and still no luck. I will keep poking at it. Looks like I may need to dig into my stomp connection...

[Erin Morrow]

Hello,

Just to close this thread out, I have solved my issues with some additional help from George Goldberg and this thread.

It turns out that Python3 requires the auto_decode parameter to be set to false or the stomp connection tries and fails to decode the zipped data in the Push Port messages.

conn = stomp.Connection([('datafeeds.nationalrail.co.uk', 61613)], auto_decode=False)

Thanks,
Erin