Push Port & stomp.py problems

[George Goldberg]

I'm trying to connect to the Darwin push port using python and the stomp.py stomp library [1] but running into difficulties. Connecting and heartbeats work fine, but as soon as I try to subscribe to the queue, the server terminates my connection. A stripped down version of the code I'm using is here [2]. I'm not sure this is necessarily relevant, but the exact same code works fine for the Network Rail stomp message queues (VSTP, Train Movements, etc).

Has anyone out there managed to get the push port working in python yet? If so, would you care to share a sample of your code, or to point out where I'm going wrong?

Cheers,

George

[1] https://github.com/jasonrbriggs/stomp.py
[2] https://gist.github.com/grundleborg/84f61fdf156ecaa620a5

[Chris Northwood]

Hi George,

I've successfully got it running Python, my code (not stripped down I'm afraid) is here:

https://github.com/ManchesterIO/manchester.io/blob/master/medlock/consumer.py

https://github.com/ManchesterIO/manchester.io/blob/master/medlock/medlock/importers/darwin_push_port.py

https://github.com/ManchesterIO/manchester.io/blob/master/medlock/medlock/parsers/darwin.py

Hope that helps,

Chris

--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To post to this group, send email to openrail...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Goldberg]

On 10 April 2015 at 17:16, Chris Northwood <cnort...@gmail.com> wrote:

Hi George,

I've successfully got it running Python, my code (not stripped down I'm afraid) is here:

Hi Chris,

Thanks very much for those links.

https://github.com/ManchesterIO/manchester.io/blob/master/medlock/consumer.py

What are you using for "NATIONAL_RAIL_CONSUMER_ID" in this file? Is it some value from the web portal? Or does it not matter what it is?

--
George

[Joel Haasnoot]

Hi George,

The stomp.py documentation isn't very good, but my code is similar to Chris'. Running through the stomp.py code for the subscribe code, it looks like this 'id' parameter is just added to the message headers. Likely this is so you can identify the source queue.

My "bare bones" code to connect and receive messages is here: https://gist.github.com/joelhaasnoot/551c57cb09cd1ce93317

HTH, 

Joel Haasnoot

--

[George Goldberg]

On 10 April 2015 at 19:32, Joel Haasnoot <joelha...@gmail.com> wrote:

Hi George,

The stomp.py documentation isn't very good, but my code is similar to Chris'. Running through the stomp.py code for the subscribe code, it looks like this 'id' parameter is just added to the message headers. Likely this is so you can identify the source queue.

My "bare bones" code to connect and receive messages is here: https://gist.github.com/joelhaasnoot/551c57cb09cd1ce93317

Thanks very much for that Joel. I had the exact same problem as with my code, which got me exploring all the other possibilities.

Cutting a long story short, it turned out to be a python 3 vs. python 2 problem, namely that stomp.py attempts to decode the entire message as UTF-8 under python 3, which of course doesn't work since the Push Port data is gzipped and thus contains invalid UTF-8 character sequences. However, the exception was silently being swallowed elsewhere in stomp.py, and the result was it was seeing 0 bytes coming back from a socket read, and interpreting this as the connection being closed by the remote party.

I've got a hack in place to workaround this for now, and I'll try and work out the best way to fix stomp.py when I have a moment and send them a patch. In the meantime, watch out if you are using python 3 and the push port!

--

George

[Erin Morrow]

George,

Would you mind sharing your hack? I've discovered the same issue with my code and would prefer not to switch Python version just to solve this issue.

Thank you,
Erin

[George Goldberg]

Hi Erin,

On 20 April 2016 at 17:25, Erin Morrow <erin....@arup.com> wrote:

Would you mind sharing your hack? I've discovered the same issue with my code and would prefer not to switch Python version just to solve this issue.

This has been fixed upstream in stomp.py for almost a year now, so no need for my hack unless you are using an ancient version. However, what you do need to do is make sure you pass "auto_decode=False" when instantiating the stomp Connection object as otherwise it tries to decode the message payload as a UTF-8 string (which obviously doesn't work because it's binary gzip data) and silently fails.

George

[Erin Morrow]

Perfect. Thank you George. It is working now.