VPN clients connecting to XNAT experience hung application and resume operation after 300 seconds

[Martin Skoffer]

Hi 

We have a strange issue on our XNAT installation; running a web client via a VPN (Wireguard or Windows VPN) results in unstable behavior and a lot of hung connections.
Especially when deleting subjects manually from projects, and in other scenarios where you hand over tasks to the server side. It is rarely when selecting projects or using the XNAT viewer.
A few subjects are deleted, and the page freezes.
After exactly 300 seconds everything is back to normal. Well, the delete failed and you can try again, and the same thing happens.
At the same time, a web client not connected via VPN, but living on the same vlan as the xnat server is working just fine.
So it seems that the connection is hung and waiting for a timeout to happen.
Tomcat status page is also non-responsive from the VPN client (but working well from the same vlan), so it is not just the XNAT application but tomcat itself as well.
So it seems like the server is ok but the connection is not.

Ofcourse we are also looking at the VPN connection itself, it just doesn't seem to be anything obvious. Same thing happens for both types of VPN, which also points to the VPN being ok (I think). The VPN connection is working fine for everything else besides XNAT/Tomcat.

Any hints what to look for ?

Thanks
Martin

[Martin Skoffer]

Additional info:
Ubuntu 20.04 LTS
XNAT 1.8.6.1 (issue also present in 1.8.4.1)

Tomcat 9.0.62

Java 1.8.0_311

[Martin Skoffer]

More additional information:
The tomcat management pages also stop responding ! Which leaves XNAT out of the equation as far as I can tell.
But I am still very interested in experiences from other users.

[Rick Herrick]

Hi Martin,

I didn’t want to leave you hanging on this so I’m replying but I honestly have no idea what’s going on. It’s clearly an issue involving the VPN but besides that I don’t know that we’ve seen a problem quite like that.

One thing I am wondering: when you say Tomcat stops responding when accessed through the VPN, does it just stop responding to those requests? Or does it also stop responding to requests made by clients behind the firewall/on the same VLAN?

If the former, then the issue is definitely related to the VPN. It seems that most of the calls that you’re seeing freeze up are those requiring non-GET calls to the server, e.g. POST to forms, DELETE to REST API calls, etc., which implies some differences in handling those requests that may be affecting the VPN.

If the latter… I have no idea honestly :) The only way I can think of to diagnose something like that would be to use a tool like Wireshark to analyze the network traffic. You’d want to look at the transactions on both the client side and between the VPN gateway and your server.

Another question that raises is whether you have a front-end proxy server like nginx, Apache HTTPD, or HAProxy in front of the Tomcat. In that case, calls on the same VLAN would go client -> proxy -> Tomcat, while calls through the VPN would go client -> gateway -> proxy -> Tomcat. The main point would be that, if there is a proxy, you’d also want to look at traffic between the gateway and the proxy and between the proxy and Tomcat.

All of which could be further complicated by encryption, both in the VPN tunnel and if you have SSL termination set up on the proxy or on Tomcat. But you can at least monitor if there’s traffic in those channels...

Rick Herrick

Senior Software Developer

rickh...@flywheel.io

------ Original Message ------

From "Martin Skoffer" <martin...@gmail.com>

To "xnat_discussion" <xnat_di...@googlegroups.com>

Date 3/10/2023 3:07:07 AM

Subject [XNAT Discussion] Re: VPN clients connecting to XNAT experience hung application and resume operation after 300 seconds

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/fc15ec67-3a59-4270-8516-f1b88d8c56bdn%40googlegroups.com.

[Martin Skoffer]

Hi Rick 

Thanks a lot for the suggestions. It has been puzzling for us as well.
We have tested some more today, and it is definately related to the VPN!
Having a session that "freezes" over VPN, instantly picks up operations again if you abort the VPN and the machine is already on the inside of the firewall (so the machine is able to connect to XNAT both with and without VPN).
I was not able to perform that test before today, as all our machines are outside the firewall normally. 

Meaning, that it must be some kind of routing or firewall rules on the router that is the blocking factor. 

I have been looking into firewall rules today, and (not being a network specialist) it COULD be that I am not allowing connections from the internal vlans towards the VPN addresses.
I just confuses me, that in a DELETE operation for instance, it seems that deleting the first 2-5 studies works fine and then it freezes. I get the green "complete" for the first items that were deleted.
But testing it again, if I only select a single study to DELETE, it still gives me the green "complete" and says that the operation was successful, but it hangs. So it does not work even when it says so.

And if the XNAT server is not allowed to connect to the client for pushing updates, it makes sense that it seems to be hanging. Not sure exactly how the tomcat server communicates with the web client.

While a VPN connection is hanging, all the connections from machines within the firewall are working fine with XNAT/tomcat. So the server is fine.

Again, thanks a lot for your reply and suggestions. I will keep digging ...
Have a nice weekend.    

Martin