Sessions stuck in "Building now" state on upload to prearchive

[Jasen Kloeppel]

I have sessions getting "stuck" in the "Building now" state when uploading them through DQR.  To get rid of them I cycle xnat and then I can delete them from the prearchive.  I can't find anything in the logs to explain what is going on with these sessions.  

Below are some screen shots of what I see.

If I select the session above and click details I see the following.

[Timothy Olsen]

When you cycle XNAT, do they show up in ERROR state?  If you go in the prearchive folder in the backend, anything odd about the data?  Are there files in the folder for that session?  

Tim

[Jasen Kloeppel]

Sorry for the delayed response.  Currently we have a few sessions "stuck" in the Building now state.  They have been like this for 5 days.  Below are some screen shots of what I currently see.  The only method to clear these sessions is to restart Tomcat which then puts these sessions in ERROR state so I can delete them from the prearchive.

[Tim Rosenow]

I had this problem a few times when I was uploading large quantities of data at once, and each time I tracked it down to one of two scenarios:

1. I had used up all of the disk space on the drive that the prearchive resides on (was hard to track down as by the time I checked it, the drive had been partly cleared so it didn't appear full).

2. I was trying to upload a duplicate of a scan *while the original was in the "Building Now" state* (note this didn't always cause the error, only occasionally: usually duplicates are handled appropriately).

I solved 1 by not uploading >1TB of scans at once, but uploading in batches. I solved 2 by tweaking the delays I built in to the batch uploads in combination with the "Session Idle Check Interval" and "Session Idle Time" parameters. In our case this hasn't happened since (because this never actually happens in practice, only when I was uploading half a decade of historical data).

[Jasen Kloeppel]

I have been playing with the  "Session Idle Check Interval" and "Session Idle Time" parameters but so far that hasn't solved my issue.  What values did you end up using that seemed to help with your issue?

Thanks,

[Tim Rosenow]

For general use, they are now back on the defaults (60000ms and 5mins). During the initial large import process, I split the data into 30 GB batches. Each batch took around 10 mins or so to upload (via REST), so I ensured that the Session Idle Time was >15 mins (so the entire batch was loaded in before the session builder ran). I then left 30 mins in between upload batches to ensure the session builder was completely finished before the next upload.

I was being overly conservative because I was in no rush and didn't want to deal with any possible errors, and this was a once-off scenario. Our data was poorly curated which is why there were so many conflicts - we had a huge directory of duplicate scans, etc. I would imagine that if you are getting these errors in daily use (i.e. not in a massive bulk upload) then it's possible your problem is triggered by something else.