Update catalogue for large scans (general question by beginner)

[Roman Belenya]

Hi,

I'm looking for advice on how to update catalogue metadata for large scans (>20k files).

In our group, we have very long fMRI scans that have many DICOM files. When I try to update the catalogue from the interface (Manage data -> Update file data), it always fails. If I try to post a curl request, it also fails with the 504 Error: Gateway Time-out error. No XML files are then created in the archive directory for these scans.

Here's an example of my request:

curl -X PUT "https://server/xapi/archive/catalogs/refresh/All" -H "accept: */*" -H "Content-Type: application/json" -d "[ \"/archive/projects/PROJECT/subjects/SUBJECT/experiments/EXPERIMENT/scans/SCAN\"]"

Should I just set up my curl request to wait for a (much) longer time, or do I change the time-out setting somewhere on the server side? I'm running xnat 1.8.4

Thanks!

Roman

[Herrick, Rick]

Hi Roman,

 

That 504 error is not from XNAT or Tomcat, but from a front-end proxy like nginx or something similar (i.e. the “gateway”). On nginx you can set a number of properties to control this sort of thing:

 

• proxy_connect_timeout
• proxy_send_timeout
• proxy_read_timeout

 

Really what should happen on the XNAT side is that these calls should fire off then return immediately. It doesn’t make much sense to have to wait for these calls because they don’t return anything useful anyway. About the only thing that can happen is something goes wrong and an exception gets thrown so you’d get HTTP status 4xx or 5xx instead of 200, but that could be handled just by creating an ID that you could use to request updates on status or outcome of the operation.

 

That doesn’t happen today, but I added XNAT-7130 to track this as a feature/improvement.

 

In the meantime configuring the proxy to allow for longer timeouts should help with this. You may also need to add some command-line options to curl if this is really taking that long. --connect-timeout is the obvious one, but that really just controls the timeout for establishing the initial connection. --max-time is probably more what you need because that controls how long curl will wait for the operation to finish. I don’t know what the default value for that is, however. There’s also --expect100-timeout, but I don’t think that will help in your situation.

 

HTH

 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

Email: jrhe...@wustl.edu

Phone: +1 (314) 740-5961

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of Roman Belenya <romab...@gmail.com>
Sent: Wednesday, June 29, 2022 6:31:26 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Update catalogue for large scans (general question by beginner)

 

* External Email - Caution *

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/46462af1-95ae-4dcd-ba8a-fb11bcec2649n%40googlegroups.com.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.