TRUST/TD firewalling

62 views
Skip to first unread message

Evelyn Snow

unread,
Jul 13, 2022, 4:39:27 PMJul 13
to openrail...@googlegroups.com
Hello

I've been having some trouble connecting to TD to work on the python TD
example on the NROD github organisation, I've found that the remote end will
give me an ERROR frame and hang up the connection, with the body claiming
that the given client ID is still connected (I've made quite sure it's not),
and from that moment onwards, for several hours, any attempts to reconnect
to the STOMP server will fail, as will attempts to ping the box, or to access
the web interface on it, although I'll note that attempts to do so from
remote servers with different IP addresses succeeded in this timeframe.

I've not seen this behaviour previously, although I've observed it twice today.

I'm unsure whether the failure to recognise that the previous connection died
is an ActiveMQ bug (which it could be, since my last successful connection was
using a client which doesn't support heartbeating), but the latter behaviour,
rejecting all connections, appears to be some sort of ill-advised firewall
rule, to speculate, it could be based on an assumption that an ERROR frame
in response to a CONNECT frame is necessarily an authentication failure.

I'd be grateful for any further information anyone has on either issue.

Evelyn

Peter Hicks

unread,
Jul 13, 2022, 5:05:52 PMJul 13
to openrail...@googlegroups.com, Tom Cairns
Hi Evelyn

It sounds like CACI are still using fail2ban and if they see a "Client already connected from..." message, are triggering an iptables rule which blocks the client for several hours.

The original intent of this was to soft-block new clients from an IP address for a minute or two if they tried to connect twice, meaning that if you did have a client connected and you started another one, your new client would trigger an iptables rule to be added which stops any *new* connections for a period of time, leaving your existing connection working fine.  How do I know?  I proposed this some years ago - but with a short time and a softer ban... but it seems to have escalated to a much harsher level for no reason, and without the rules being communicated to users.  Which isn't awfully helpful.

Combined with this, I think there's a bug in the version of ActiveMQ that CACI is running, where a Stomp connection will get disconnected but ActiveMQ will still think a client is connected.  I'm not sure at what level the problem is - networking stack or application - but I think @Tom Cairns has had issues with it and reported having no luck getting the root cause fixed.

One option is not to use a durable subscription or rotate your client ID each time you reconnect.  That might turn the platform into the equivalent of a Portaloo at a music festival, so I'd recommend against it.  What I'd say is you might find it much easier to set up a copy of ActiveMQ locally with a static username/password (if you need it), and use the version of Camel embedded within ActiveMQ to bridge the NROD topics over OpenWire, which seems not to have this issue.  You can then connect to a broker locally over Stomp.


Peter


--
You received this message because you are subscribed to the Google Groups "A gathering place for the Open Rail Data community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openraildata-t...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/openraildata-talk/20220713203918.i6c7xzvzupabvilp%40evelyn.moe.

Evelyn Snow

unread,
Jul 13, 2022, 5:50:07 PMJul 13
to openrail...@googlegroups.com
Hi Peter

It's interesting that they've resolved to block all connections from the client
instead of only those related to message queues, and for the length of time
that they are - to me that seems absurdly disproportionate, and that's the
reason I assumed it might be security related.

It's really quite disappointing, particularly the lack of documentation,
although I'm not sure I should be too surprised at this point.

I did previously attempt to set up an ActiveMQ server using the guide on the
wiki but found that the software is rather unwieldy, and wasn't able to make
much progress with it. It's a shame there isn't an AMQP interface to the
queues, since that'd make RabbitMQ's shovel plugin a fairly approachable
alternative.

At any rate, thank you very much for the information, I'll make sure to note
this exciting footgun properly on the wiki.

Evelyn

2022-07-13T22:05:39+0100 Peter Hicks <peter...@poggs.co.uk>:
> https://groups.google.com/d/msgid/openraildata-talk/CAKGkhzZsvfUmV-x_Ogq8xU_1f%3DF-wsyidi%3D4VF_-WCEdGZ2boA%40mail.gmail.com.

Tom Cairns

unread,
Jul 13, 2022, 7:29:45 PMJul 13
to Peter Hicks, openrail...@googlegroups.com

Indeed, I never got anywhere with having that resolved. I actually moved over to a Camel/ActiveMQ bridge as it seemed the only way to adequately resolve it.

 

I do have other problems with the Camel/ActiveMQ bridge such as my VSTP queue dropping repeatedly due to invalid frames being sent from NROD but that’s a different problem… but I do have a separate Camel running to bridge (& convert) the data before it hits my ActiveMQ (Artemis) brokers.

 

Tom

Reply all
Reply to author
Forward
0 new messages