http.client.RemoteDisconnected: Remote end closed connection without response

[Tatjana Schmidt]

Hi everyone,

I have a script that I use to download data from XNAT and it works without issues, however since it takes around an hour per subject, I wanted to run it as an HPC array job, i.e. 100 subjects in parallel. It seems to work fine for the first few subjects, however after a few it fails and returns the error "http.client.RemoteDisconnected: Remote end closed connection without response". For those subjects where it works, it takes an extremely long time in some cases, e.g. 6h for a set of data that takes 30 minutes when it's not run as an array job. 

I'm wondering if there is an issue with 100 instances attempting to log in at once and thus the connection is closed? Also if anyone has an idea how to do this more efficiently I'd be happy to hear suggestions. 

Thank you in advance!

[akluiber]

Some thoughts:

- Create and use an alias token (https://wiki.xnat.org/documentation/generating-an-alias-token-for-scripted-authenticat, https://wiki.xnat.org/xnat-api/user-alias-token-api)

- Use JSESSION IDs from $XNAT_HOST/data/JSESSION to reduce the number of concurrent open sessions (https://wiki.xnat.org/documentation/generating-and-reusing-a-jsession-id-for-scripted-)

- You could increase the the user session timeout and alias token expiration if needed in Site Administration/Security

- You could run a timer on your processing and grab a new session id if the process took longer than the timeout