Data Stuck in pre-archive

[suyash Bhogawar]

Hello All,

I have a problem where sessions are getting "stuck" in the prearchive with an "archive pending" status, and they can't be reset from the web interface. Due to this i cannot archive my data pushed to XNAt from console.

I am using XNAT 1.6.4. 

Does any one how to solve this problem?

Thanks!

Suyash 

[Herrick, Rick]

Do you see any errors in the XNAT logs that correspond with the times that XNAT tried to archive the sessions?

One thing you can try is to drop the prearchive table. You need to stop your Tomcat server, then go into the database, something like this:

psql -U xnat

drop table xdat_search.prearchive;

Restart Tomcat and see what happens. The prearchive table is rebuilt on each XNAT start. You may also be able to see log messages when that happens that would indicate something else going on.

There’s also the possibility that something’s messed up in the session XMLs that are being built for the sessions. You can try deleting all of the XMLs in your prearchive folder while you have your Tomcat shut down, something like:

cd /wherever/you/have/your/prearchive

find . -name *.xml –exec rm ‘{}’ \;

It’s possible those will be re-built when the prearchive table is re-built anyway, but without getting into some murky code I can’t tell for sure and can’t remember off the top of my head…

-- 

Rick Herrick

Sr. Programmer/Analyst

Neuroinformatics Research Group

Washington University School of Medicine

(314) 740-5961

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

---

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[suyash Bhogawar]

Thank you Rick for your detailed response.

As per your email i am trying to locate XNAT logs. Can you please tell me where can i find them?

Many Thanks!

Suyash

[Herrick, Rick]

They are in your deployed XNAT application in the folder logs. So if your Tomcat is located (example uses Debian/Ubuntu paths, change as necessary for your deployment) in /var/lib/tomcat7, the path to the logs would be:

/var/lib/tomcat7/webapps/xnat/logs

[suyash Bhogawar]

Thank you Rick for your response.

But unfortunately i wasnt able to find any logs under var/lib/tomcat6/webapps .

Which other place can i search?

By any chance is it possible that if i delete files from pre-archive through terminal without restarting tomcat server? Will this work?

Many Thanks!

Suyash

[Herrick, Rick]

Well, it will definitely remove the files, so it will work in that sense. But it won’t clear the prearchive entry in the database, which will then just sit there in your UI.

Note that you don’t need to be able to access the logs to shut down Tomcat, drop the prearchive table, delete the files, and re-start Tomcat. The logs would be useful in figuring out why the sessions are stuck and hopefully preventing it from happening again.

Regarding the log files, do you see the folder /var/lib/tomcat6/webapps/xnat? There really should be a logs folder under that. If not, you can look at the file WEB-INF/conf/log4j.properties underneath the xnat webapp folder. Look for a line like this:

log4j.appender.turbine.file=${applicationRoot}/logs/turbine.log

That ${applicationRoot}/logs/turbine.log is the default value and maps to /var/lib/tomcat6/webapps/xnat or wherever your XNAT application is hosted. If the path is something beside that default value, you can look there for the logs. If it’s the default value and there’s no /var/lib/tomcat6/webapps/xnat/logs folder and no logs in there, you have something bigger going wrong. If that’s the case I wouldn’t be surprised if there’s some kind of permissions or ownership problem.

[suyash Bhogawar]

Thank you rick for the detailed message. 

I checked og4j.appender.turbine.file for the default value. yes its set for default value and hence i check if logs are under /var/lib/tomcat6/webapps/xnat. But as you pointed logs folder is not here. 

What should I do in this case?

i suspect its been the same way from day one. Does this issue also cause XNAT to be very slow?

very Grateful!

Suyash

[Herrick, Rick]

The only thing I can imagine at this point is a problem with write permissions. The Tomcat process runs as a particular user, by default the same name as the service, e.g. tomcat6. Depending on your OS, this is configured in various places. On Debian/Ubuntu, you can usually find it in /etc/default/tomcat6.  If you look at that file, it will have lines like this:

 

TOMCAT6_USER=tomcat6

TOMCAT6_GROUP=tomcat6

 

That user has to be able to write into the xnat folder under the Tomcat webapps folder. You can try a couple things to see if that’s the case:

 

·         Type “ls -ld /var/lib/tomcat6/webapps/xnat” and see what you get. Trying this on the XNAT 1.6.4 virtual machine, I get this:

 

xnat@xnatdev:~# ls -ld /var/lib/tomcat7/webapps/xnat/

drwxrwxr-x 16 xnat xnat 4096 Nov 11 11:02 /var/lib/tomcat7/webapps/xnat/

 

This works because the Tomcat user and group are set to xnat on the VM. If you see something else, that indicates that another user (probably whatever login you used to build XNAT) owns the folder  and your Tomcat user can’t write to it.

 

·         You can also try writing to the folder as that user. Try something like this:

 

xnat@xnatdev:/home$ sudo -H -u tomcat7 mkdir /var/lib/tomcat7/webapps/xnat/logs

mkdir: cannot create directory ‘/var/lib/tomcat7/webapps/xnat/logs’: Permission denied

 

That sudo command runs the specific command (in this case, mkdir) as the user indicated by the -u option. You’d want to substitute your Tomcat user (probably tomcat6) for the tomcat7 shown in this case.

 

If this does seem like the problem, it should be fairly easily fixable. Just run the following command (assuming your Tomcat user is tomcat6):

 

chown -R tomcat6.tomcat6 /var/lib/tomcat6/webapps/xnat

 

After that, your Tomcat user should be able to write into that folder.

 

If this is the case, then it’s also possible that you have the same thing going on with your prearchive and archive folders and that’s why the sessions are getting stuck: when you go to do the move,  the Tomcat user is denied permission to move the file. You can run the same chown command from above but just substitute the other paths in your installation for archive, prearchive, etc.

[suyash Bhogawar]

I will ofcourse go through the above steps mentioned by you. But just that you the know, 

the problem of data getting stuck in pre-archive is not for all projects. i have 7 projects and data gets stuck in pre-archieve only for one project. Rest all other projects data can be archived.

Here is the screen shot of the error.

 

-

Suyash

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/RXBVkZ7mfZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

--

Regards 

Suyash Bhogawar

Graduate - Bio-medical Engineering

202-594-4582

[suyash Bhogawar]

Hi Rick, 

I am getting following error after running the commands you suggested.

-sh-4.1$ sudo -H -u tomcat6 mkdir /var/lib/tomcat6/webapps/xnat/logs

sudo: unknown user: tomcat6

sudo: unable to initialize policy plugin

-sh-4.1$ chown -R tomcat6.tomcat6 /var/lib/tomcat6/webapps/xnat

chown: invalid user: `tomcat6.tomcat6'

Can you please point out if i have made any mistake?

Many Thanks!

Suyash

Cornell University

[Herrick, Rick]

Yes, you need to check that name, tomcat6, against the value for TOMCAT6_USER in /etc/default/tomcat6.conf. Whatever value is set there is what you need to use in place of tomcat6 in all of those commands. This assumes you have tomcat6 on Ubuntu/Debian, because if you don’t, the configuration file won’t be in /etc/default/tomcat6.conf. Instead it’ll be in /etc/default/tomcat7.conf (Tomcat 7 on Debian/Ubuntu) or /etc/tomcat6/tomcat6.conf (Tomcat 6 on Fedora/RHEL/CentOS). So you need to figure out both the Tomcat user and group to make those commands work.

If the user IS set to tomcat6 and that user is unknown, then the problem is outside of XNAT: your system is misconfigured or corrupted somehow.

[suyash Bhogawar]

Hi Rick, 

I was able to locate XNAT installation and found logs folder. However there are too many different pre-archive logs saved for different dates. 

I even restarted our tomcat server but still the files stuck in pre archive with status pending are not dropped.

I have attached all recent log files with this email. I went through the prearchive log but wasn't able to interpret it if theres any error with it.

Can you please point out any error that is causing few subjects stuck in pre archive?

Thanks!

Suyash

--
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/RXBVkZ7mfZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.

To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion.
For more options, visit https://groups.google.com/d/optout.

--

[Herrick, Rick]

Just so this doesn’t get lost in all the notes I made below, what version of XNAT are you running and on what platform?

This looks like something to do maybe with available storage space or memory. There were some errors in the application log about PostgreSQL being unable to create connections:

2015-05-04 03:24:53,097 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-1] ERROR org.quartz.core.JobRunShell - Job users-jobs.clear-expired-alias-tokens threw an unhandled Exception: 

org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is org.postgresql.util.PSQLException: Connection rejected: could not fork new process for connection: Cannot allocate memory

That was early in the day, but it’s conceivable issues stemming from that affected calls later on.

Then later in the day, there are errors that are coming from Tomcat:

2015-05-04 14:40:46,256 [http-apr-8443-exec-10] ERROR org.restlet.XNATComponent.Server - An exception occured writing the response entity

ClientAbortException:  java.io.IOException

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:413)

        <snip>

Caused by: java.io.IOException

at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:205)

        <snip>

That is some kind of system level error where Tomcat is failing to properly write. Since this is while receiving a response entity (i.e. a file or data payload of some sort), it’s really hard to say what kind of issue might be involved. It does NOT look like the problem there is related to memory: I’d guess disk space or write/permissions issues.

Then it looks like the prearchive builder continues to think data is arriving for sessions and never builds them, until it does, at which point:

2015-05-04 14:40:08,001 [http-apr-8443-exec-5] ERROR org.nrg.xnat.helpers.prearchive.PrearcTableBuilder - 

org.nrg.xnat.helpers.prearchive.PrearcDatabase$SyncFailedException: Session already exists, retry with overwrite enabled

at org.nrg.xnat.helpers.prearchive.PrearcDatabase$10.extSync(PrearcDatabase.java:517)

Then there appear to be some successes:

2015-05-03 02:58:52,678 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2] DEBUG org.nrg.xnat.helpers.prearchive.SessionXMLRebuilderJob - Creating JMS queue entry for admin to archive /prearchive/projects/FFP/20150502_193940203/RNS1811187

2015-05-03 02:58:53,038 [org.springframework.scheduling.quartz.SchedulerFactoryBean#0_Worker-2] INFO  org.nrg.xnat.helpers.prearchive.SessionXMLRebuilderJob - Built 1 of 38

2015-05-03 02:58:53,078 [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] INFO  org.nrg.xnat.helpers.prearchive.SessionXmlRebuilderRequestListener - Received request to process prearchive session at: /prearchive/projects/FFP/20150502_193940203/RNS1811187

2015-05-03 03:15:56,889 [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] DEBUG org.nrg.xnat.helpers.prearchive.SessionXmlRebuilderRequestListener - Processing queue entry for admin in project FFP to archive /prearchive/projects/FFP/20150502_193940203/RNS1811187

2015-05-03 03:15:56,891 [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] INFO  org.nrg.xnat.helpers.prearchive.SessionXmlRebuilderRequestListener - Listener completed session XML rebuild request.

There are some strange errors in your xdat.log:

Caused by: java.lang.NumberFormatException: For input string: "2703291894"

at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)

at java.lang.Integer.parseInt(Integer.java:495)

at java.lang.Integer.valueOf(Integer.java:582)

That should parse as best I can tell, but the line numbers in the source code are off from what I currently have

[suyash Bhogawar]

Thank you Rick!

I am running XNAT on 

Distributor ID: CentOS

Description: CentOS release 6.5 (Final)

Release: 6.5

Codename: Final.

I haven't heard confirmation about XNAT version from our programmer who had installed it. 

My notes say its 1.6.4. 

I have other question about XNAT_RESTAPI. 

Documentation  shows its in bin folder of 'xnat_builder' but i wasn't able to locate this folder in the XNAT installation on our Tomcat server. Any suggestions..?

Thanks!

Suyash

On Tue, May 5, 2015 at 2:45 PM, Herrick, Rick <herr...@mir.wustl.edu> wrote:

This message is eligible for Automatic Cleanup! (herr...@mir.wustl.edu) Add cleanup rule | More info

[Herrick, Rick]

You should use XnatDataClient instead of XNATRestClient:

https://wiki.xnat.org/display/XNAT16/XnatDataClient

This is located in the bin folder of your XNAT builder.

[suyash Bhogawar]

Thank you Rick for the suggestion. 

I have to spend some time to learn it properly and start using it for accessing/downloading data from XNAT sever. 

However, our aim is to push data to XNAT from a PC other than MRI console. 

Currently we push data from our console at Cornell MRI facility. But as this is slow and at times slows down other recon scripts running, we are looking into options to upload data from other PC/dicom catcher. This dicom PC will be devoted to push data to XNAT and run few pre processing or recon scripts. 

As we have just started using XNAT, it would be great if you can suggest us few options for this exercise. 

By now you must have discovered from my naive questions about the level we are at. It would be also great if you can point us to a video tutorial or training videos if available. We here are trying best to convince our PI's to use XNAT. 

Many Thanks for your help!

Suyash 

Programmer Analyst 

Cornell MRI Facility