Instant disconnect

[Anthony Patman]

I'm using the Python code example on Github to learn about connecting to the data feeds, as part of a wider exercise to build expertise around s-class message interpretation.  I have configured the script with the details provided on the wiki and (verified) that I use to log into datafeeds.networkrail.co.uk.  stomp.py and pyxb packages are installed.

I've encountered some issues.

1. I cannot find any record anywhere of a package called "PPv16".  Python rejects the command to import it.  PIP does not recognise it and a Web search yields pages about some aeronautical engineering pipe elbow.  pyxb is something new to me but I see it isn't imported in the code and the application of PPv16 seems to be to do with decoding the messages, so I'm assuming PPv16 is a component of pyxb (since that is the specified dependency), yet Python simply does not understand the import reference.
2. The script runs fine once I've hacked past issue no. 1 (by just commenting out the import and putting 1==1 in the TRY statement), and seems to set up and execute the connection without complaining, reaching one iteration of the while True: time.sleep(1) statement but then disconnecting.  I've checked the connection details n times and they are correct.

USERNAME = '[the right one as gets me into datafeeds.networkrail.co.uk]'

PASSWORD = ' [the right one as gets me into datafeeds.networkrail.co.uk]  '
HOSTNAME = 'datafeeds.networkrail.co.uk'
HOSTPORT = 61618
# Always prefixed by /topic/ (it's not a queue, it's a topic)
TOPIC = '/topic/TD_ALL_SIG_AREA'

I am definitely missing something.  Please can someone help?

Regards,

Anthony.

[Peter Hicks]

Hi Anthony

The GitHub repository you're using is for the National Rail (RDG) data feeds.  PPv16 is a class generated from the rttiPPTSchema_v16.xsd, used to deserialize (convert) the XML messages into Python objects.

To hack this code into working for the Network Rail feeds, you probably want to change the on_message() function to just print the message, rather than uncompressing it, as all the messages on the Darwin Open Data feed are compressed.

If you can't find a Stomp client specifically for the Network Rail feeds on https://github.com/openraildata, I'll write one.  It should only take a few minutes.

Best wishes,

Peter

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/fa5658cd-a938-461d-a077-b538f5f1a210n%40googlegroups.com.

[Anthony Patman]

Thanks for the quick help, Peter.  I've tried simply commenting out the object conversion bit but actually I think no. 2 is the main issue: the connection is made then immediately dropped again.  This still happens even if I comment out everything except a single print statement in def on_message.  That part of the code is never getting 
executed because of the disconnection.  Is there a difference in how the connection itself is established between the Darwin feeds (as it was in the original code template) and the TD feed I'm trying to access?

[Evelyn Snow]

Hi Anthony

TD and Darwin aren't too different in terms of connection other than the sorts
of details you'd expect to be different (credentials, hostname, etc).

I wrote a short demonstration script in python for another mailing list member,
you may also find it useful. You'll need to install the dependencies using pip
(stomp.py==7.0.0, termcolor==1.1.0), and replace the email and password
placeholders near the top with your actual credentials.

If all goes well it should show formatted TD messages from all areas in your
terminal.

https://gist.github.com/EvelynSubarrow/53a78988371ee512e30063bf51518c3c

Evelyn

2022-06-12T08:19:33-0700 Anthony Patman <atp...@gmail.com>:

> 1. I cannot find any record anywhere of a package called "PPv16". 

> Python rejects the command to import it.  PIP does not recognise
> it and a Web search yields pages about some aeronautical
> engineering pipe elbow.  pyxb is something new to me but I see it
> isn't imported in the code and the application of PPv16 seems to
> be to do with decoding the messages, so I'm assuming PPv16 is a
> component of pyxb (since that is the specified dependency), yet
> Python simply does not understand the import reference.

> 2. The script runs fine once I've hacked past issue no. 1 (by just

> https://groups.google.com/d/msgid/openraildata-talk/760001fa-8682-4490-ad6d-d0eeb2332355n%40googlegroups.com.

[Anthony Patman]

Just reporting back on how I've got on.

I established a connection and accessed the data feed successfully.  I experimented with Evelyn's script, which did connect, and then tried to use that to diagnose why I wasn't able to connect with the original example.  Here's what made the difference: in the on_message method of the StompClient class, populating the variables header and message by including them in the arguments of the method did not work.  What did was: having a "frame" variable as the argument and then extracting the header and message values from that, as in Evelyn's script.  (I also wrote something different to lay out the messages in a basic form, as I'm learning at the moment and will get into actually doing something with them later.  Stomp Protocol 12 ACK function doesn't support the subscription parameter, so I've deleted it - what's the purpose of the acknowledgement anyway?  I'm using area TW just because it's a fairly straightforward TD, and because my colleague has already been using it for the work he's already done.)

class StompClient(stomp.ConnectionListener):

    def __init__(self):
        self._conn = conn
        self.cnt = 0

...

    def on_message(self, frame):
        try:
            headers, message = frame.headers, frame.body
            self._mq.ack(id=headers['message-id'])
            parsed = json.loads(message)
            for foo in parsed:
                for actual in foo.values():
                    # print(address)
                    if actual["area_id"] == "TW" \
                    and actual["msg_type"] in ["SF", "SG", "SH"]:
                        print(self.cnt
                            , actual["area_id"]
                            , actual["msg_type"]
                            , actual["address"]
                            , actual["data"])
                        self.cnt=self.cnt+1
        except Exception as e:
            print("\n\tError: %s\n--------\n" % str(e))

[Anthony Patman]

In fact that ACT statement still doesn't work.  After about half a dozen messages it throws an error something like:

0 TW SF 22 01
1 TW SF 18 60
2 TW SF 21 80
3 TW SF 03 B1
4 TW SF 12 10
5 TW SF 16 60

    Error: Invalid control character at: line 1 column 3005 (char 3004)

[Evelyn Snow]

Hi Anthony

Glad to hear it was useful!

Evelyn

2022-06-14T02:09:05-0700 Anthony Patman <atp...@gmail.com>:

> https://groups.google.com/d/msgid/openraildata-talk/919e3042-4dbb-476c-8546-e765336cb0een%40googlegroups.com.

[John-Lee Langford]

Did you manage to solve this issue?

I've only just started looking at using the Open Rail Data and it seems I've fallen at the first hurdle!

I'm using Stomp 8.0.1 (supporting v1.0, 1.1, 1.2 of the protocol) as this is the default version installed using the Thonny IDE on MacOS.

Thanks

[Peter Hicks]

Hi John-Lee

Have you had a look at the example code at https://github.com/openraildata/td-trust-example-python3?  This will probably help you work out what's going on.

If you're able to, uploading your client to a GitHub repository or even a Gist (https://gist.github.com/) - as long as you remove your username and password - will be really helpful and let others see the actual code you're using.  The problem there may be obvious!

Peter

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/6c7b636e-3726-483e-b246-0a09a1a5fb8cn%40googlegroups.com.

[John-Lee]

Thanks Peter

I've tried using stomp-client-python.py (https://github.com/openraildata/stomp-client-python). I have changed the USERNAME and PASSWORD variables to the ones supplied on the My Feeds page. No other changes have been made to any other part of the code. When executing, all I get is the following:

Connecting to darwin-dist-44ae45.nationalrail.co.uk
Disconnected waiting 15 seconds before exiting

Following your suggestion, I have tried td-trust-example-python3.py (https://github.com/openraildata/td-trust-example-python3). Again, I have not made any changes to the code apart from adding my email address and password to secrets.json. When executing, I get the following errors:

>>> %Run main.py
Traceback (most recent call last):
  File "/[path redacted]/main.py", line 70, in <module>
    connection.connect(**connect_headers)
  File "/[path redacted]/Library/Python/3.7/lib/python/site-packages/stomp/connect.py", line 150, in connect
    self.transport.start()
  File "/[path redacted]/Library/Python/3.7/lib/python/site-packages/stomp/transport.py", line 125, in start
    self.attempt_connection()
  File "/[path redacted]/Library/Python/3.7/lib/python/site-packages/stomp/transport.py", line 755, in attempt_connection
    self.__enable_keepalive()
  File "/[path redacted]/Library/Python/3.7/lib/python/site-packages/stomp/transport.py", line 733, in __enable_keepalive
    try_setsockopt(self.socket, socket.IPPROTO_TCP, 0x10, ka_intvl)
TypeError: try_setsockopt() missing 1 required positional argument: 'val'

I've tried running each from within my IDE (Thonny) and from the Terminal, and both give identical messages.

[Anthony Patman]

John-Lee, see my reply of  Wednesday, 15 June 2022 at 03:08:29 UTC to see how I got it working.  I found the solution to be obscure and a bit random, but it suggests that the example code often doesn't work out of the box, probably for local reasons rather than there being anything wrong with it.

[Peter Hicks]

That's interesting - but not unexpected.  I didn't include a requirements.txt in the example code, which works with stomp.py v4.1.22, but not with version 7 or 8.

Time to update the client I think!

 

Peter

To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/e1af717c-e499-48d5-9728-4c9849fb915dn%40googlegroups.com.

[Peter Hicks]

OK, fixed.  The issue is with on_message.  In version 4.x, it took the parameters 'header' and 'body', but now it takes an object called 'frame', which has methods 'header' and 'body'.

I've also taken the liberty of adding better logging (although stomp.py's logging is verbose, even at INFO level).

Peter

[John-Lee]

Thanks for that, I'm getting data through now. However, I'm also getting the following:

ERROR    module 'PPv16' has no attribute 'CreateFromDocument'