Large project deletion

[Alexis]

Dear all,

I have a project with a project I have deleted. To summarize : my main
project contains 400~500 subject, each of them having ~20 subject
assessor ( behavioural tests ), 2 mrsessions of ~6 scans, each scan
having a nifti conversions stored as a mrassessor. I would like to
delete this project to re-insert it with another ( I don't mean
changing the xnat id, but every data/files have been changed so I
really need to delete everything ).

The point is that I selected in the interface : Project -> delete ->
delete entire project -> Ok ( I didn't selected to delete the files ).
I though it would take a while, but 24h later, the web page is still
waiting.

So I stopped it. I cannot see any more the project and it's data in
the interface ( and so I'm not proposed to delete anything ) . But
when I looked in the database : in xdat_projectdata , the project has
disappeared, but in xnat_experimentdata still remain a lot of records
which project is empty.

Could a timeout cause this ? Is there a way I can clean this without
complete drop the database ?

Thanks

Alexis

[Timothy Olsen]

That doesn't sound like the intended behavior. 

 

Which branch of the code are you running?

 

The code for deleting data is not very robust (mainly because we don't use it much, especially not removing large projects).  However, it usually wouldn’t fail in such frustrating fashion, and it was already on my list for review.  Within XNAT deleting data is especially slow.  Speeding it up is possible, but would probably require more time then I have now.

 

The sequence of events the Delete Project option goes thorough is...

START

  For each subject in project

     For each experiment in subject

         For each assessor in experiment

             Delete assessment

         End

 

         Delete experiment

     End

 

     Delete subject

  End

 

  Delete project

END

 

The code is structured in this way to prevent an sql statement that is GB in size and to provide for security checks at each level (shared subjects wouldn’t need to be deleted).  Each step involves updating metadata and history tables with associated triggers.  This is part of the speed problem.

 

The java code for this is in the DeleteProjectData action. 

 

In even the last released code, the project may be deleted even if the deletion of an individual subject, expt or assessment failed.  I'll need to review this.  However, the hanging page is surprising to me.  The last step of the process is to delete the project.  So the fact that your project was deleted, leads me to believe that the process completed, and the action should have completed.

 

Were there any errors in your xdat.log, sql.log, or catalina.out?

 

I have a more recent version of the code available that I can check in.  It primarily included more thorough security review for non standard structures (shared subjects without shared experiments, etc).   However, I don't think it will be any quicker.  Ideally, it would prevent the deletion of the project if previous steps failed.  Let me know which branch you are on.

 

 

Regarding fixing your post delete database...

 

It may be easier to roll back to a backup if you have one available (we keep nightly copies of our db).  If you don’t have a recent backup, then you’ll need to fix the database.

 

I would create a dummy project through the website.  Then run some SQL to add any orphaned subjects or experiments to that project.  Something like…

 

UPDATE xnat_experimentData SET project=’DUMMY_ID’ WHERE project IS NULL;

 

UPDATE xnat_subjectData SET project=’DUMMY_ID’ WHERE project IS NULL;

 

FYI, if your file paths are relative, the references to the file system will be broken unless you use a project with the same id as the previous one.

 

Once they are added in the database, you can return to the web site to navigate the contents (see what got deleted and what didn’t).

 

 

Tim

 

Another option for this would be to delete the data via the REST interface.  I know one of my test scripts creates a project with 100 subjects (10 MR each), and then deletes it.  I’ve never timed it, but it always completes successfully.  I have a more recent copy of that code as well.

[Gao, Yong]

Hi,

I was trying to install 1.4rc2 on my new Thinkpad w700 running 64-bit Vista. The
setup step produced the error below. I have 64-bit jdk.

Anyone has seen this or has a clue?

There was no problem with my older Thinkpad.

Thanks,

Yong

----------------------------------------------------------------------------
C:\xnat>bin\setup.bat -Ddeploy=true
Using XDAT Installation: C:\xnat
Using JAVA_HOME: C:\Program Files\java\jdk1.6.0_13
.
Verify java version (with 'java -version')
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
__ __
| \/ |__ _Apache__ ___
| |\/| / _` \ V / -_) ' \ ~ intelligent projects ~
|_| |_\__,_|\_/\___|_||_| v. 1.0.2

Plugin cache will be regenerated
build:start:

xdat:setup:
[echo] HOME: C:\xnat
xdat:testDB:

xdat:verifyPLPGSQL:

xdat:copyMavenJars:
[echo] HOME: C:\xnat
[echo] USER HOME: C:\Users\yong

[echo] INCLUDE XNAT TEMPLATE: true
[echo] ROOT SCHEMA: C:\xnat\schemas
xdat:modifyTurbine:
[delete] Deleting 1 files from C:\xnat\plugin-resources\plugins
[delete] Deleting 39 files from C:\xnat\plugin-resources\cache
[delete] Deleted 27 directories from C:\xnat\plugin-resources\cache

xdat:setupLib:
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
[copy] Copying 1 file to C:\xnat\lib
java.io.IOException: Cannot run program "command.com": CreateProcess error=2,
The system cannot find the file specifie
at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
at java.lang.Runtime.exec(Runtime.java:593)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:64
6)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at
org.apache.tools.ant.taskdefs.Execute.getProcEnvironment(Execute.java:180)
at
org.apache.tools.ant.taskdefs.Property.loadEnvironment(Property.java:455)
at org.apache.tools.ant.taskdefs.Property.execute(Property.java:359)
at org.apache.tools.ant.Task.perform(Task.java:341)
at org.apache.commons.jelly.tags.ant.AntTag.doTag(AntTag.java:185)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:135)
at
org.apache.maven.jelly.tags.werkz.MavenGoalTag.runBodyTag(MavenGoalTag.java:79)
at
org.apache.maven.jelly.tags.werkz.MavenGoalTag$MavenGoalAction.performAction(Mav
enGoalTag.java:110)
at com.werken.werkz.Goal.fire(Goal.java:639)
at com.werken.werkz.Goal.attain(Goal.java:575)
at com.werken.werkz.WerkzProject.attainGoal(WerkzProject.java:193)
at
org.apache.maven.jelly.tags.werkz.MavenAttainGoalTag.doTag(MavenAttainGoalTag.ja
va:127)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:135)
at
org.apache.maven.jelly.tags.werkz.MavenGoalTag.runBodyTag(MavenGoalTag.java:79)
at
org.apache.maven.jelly.tags.werkz.MavenGoalTag$MavenGoalAction.performAction(Mav
enGoalTag.java:110)
at com.werken.werkz.Goal.fire(Goal.java:639)
at com.werken.werkz.Goal.attain(Goal.java:575)
at com.werken.werkz.WerkzProject.attainGoal(WerkzProject.java:193)
at
org.apache.maven.jelly.tags.werkz.MavenAttainGoalTag.doTag(MavenAttainGoalTag.ja
va:127)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:135)
at
org.apache.maven.jelly.tags.werkz.MavenGoalTag.runBodyTag(MavenGoalTag.java:79)
at
org.apache.maven.jelly.tags.werkz.MavenGoalTag$MavenGoalAction.performAction(Mav
enGoalTag.java:110)
at com.werken.werkz.Goal.fire(Goal.java:639)
at com.werken.werkz.Goal.attain(Goal.java:575)
at
org.apache.maven.plugin.PluginManager.attainGoals(PluginManager.java:671)
at org.apache.maven.MavenSession.attainGoals(MavenSession.java:263)
at org.apache.maven.cli.App.doMain(App.java:488)
at org.apache.maven.cli.App.main(App.java:1239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.jav
a:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.werken.forehead.Forehead.run(Forehead.java:551)
at com.werken.forehead.Forehead.main(Forehead.java:581)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find
the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.<init>(ProcessImpl.java:81)
at java.lang.ProcessImpl.start(ProcessImpl.java:30)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:452)
... 45 more
xdat:buildGenericExecutables:
[copy] Copying 1 file to C:\xnat\plugin-resources\misc
[copy] Copying 1 file to C:\xnat\plugin-resources\misc
xdat:editGenericExecutable:
[copy] Copying 1 file to C:\xnat\bin
[copy] Copying 1 file to C:\xnat\bin

The information in this e-mail is intended only for the person to whom it is
addressed. If you believe this e-mail was sent to you in error and the e-mail
contains patient information, please contact the Partners Compliance HelpLine at
http://www.partners.org/complianceline . If the e-mail was sent to you in error
but does not contain patient information, please contact the sender and properly
dispose of the e-mail.

[Alexis B]

Thanks Tim,

The xnat version I use is 1_4b_061008 , and I think I last updated on
last november ( the reason is that I use it in production and I wanted
to update only for a release version ). I have put below what I have
found and what I have tried :

- in the db : the project has been deleted, the subjects also, but
the experiments not, and their project is empty. Subject assessor are
also still here, and their subject_id is empty.
- What is there in the logs ?

On 16 avr, 20:27, Timothy Olsen <t...@npg.wustl.edu> wrote:
> That doesn't sound like the intended behavior.
>
> Which branch of the code are you running?
>
> The code for deleting data is not very robust (mainly because we don't use it much, especially not removing large projects). However, it usually wouldn't fail in such frustrating fashion, and it was already on my list for review. Within XNAT deleting data is especially slow. Speeding it up is possible, but would probably require more time then I have now.
>
> The sequence of events the Delete Project option goes thorough is...
>
> START
>
> For each subject in project
>
> For each experiment in subject
>
> For each assessor in experiment
>
> Delete assessment
>
> End
>
> Delete experiment
>
> End
>
> Delete subject
>
> End
>
> Delete project
>
> END
>
> The code is structured in this way to prevent an sql statement that is GB in size and to provide for security checks at each level (shared subjects wouldn't need to be deleted). Each step involves updating metadata and history tables with associated triggers. This is part of the speed problem.
>
> The java code for this is in the DeleteProjectData action.
>
> In even the last released code, the project may be deleted even if the deletion of an individual subject, expt or assessment failed. I'll need to review this. However, the hanging page is surprising to me. The last step of the process is to delete the project. So the fact that your project was deleted, leads me to believe that the process completed, and the action should have completed.
>
> Were there any errors in your xdat.log, sql.log, or catalina.out?
>
> I have a more recent version of the code available that I can check in. It primarily included more thorough security review for non standard structures (shared subjects without shared experiments, etc). However, I don't think it will be any quicker. Ideally, it would prevent the deletion of the project if previous steps failed. Let me know which branch you are on.
>
> Regarding fixing your post delete database...
>
> It may be easier to roll back to a backup if you have one available (we keep nightly copies of our db). If you don't have a recent backup, then you'll need to fix the database.
>

> I would create a dummy project through the website. Then run some SQL to add any orphaned subjects or experiments to that project. Something like...

[Alexis B]

( Sorry, I pressed the send button by mistake )

- So in the logs :
- xdat : many things like "INFO org.nrg.xft.db.DBAction -
xnat:datatypeProtocol_meta_data updated" , "INFO
org.nrg.xft.db.DBAction - xnat:datatypeProtocol Removed.", ... and
many times the error :
[http-8443-Processor3] ERROR org.nrg.xft.XFTItem -
482-org.postgresql.util.PSQLException: ERREUR: la fonction
gettext_psytool_testdata(character varying, integer, boolean, boolean,
boolean) n'existe pas
( ie: a function seems to not exist, but concerns only 1 type of
subject_assessor )

and

ERROR velocity - Left side ($data.getSession().getAttribute
("user").getUsername()) of '==' operation has null value. If a
reference, it may not be in the context. Operation not possible.
navigations//NoMenuTop.vm [line 9, column 76]


Then I tried to create a dummy project, and I modified by sql command
the xnat_experimentData project. Then I could see the data again, but
they don't have subject. If I select one, and click delete, it works
( even in the db, the line disappear ).
Now I again click on project->delete . It took some minutes and seems
to end nicely. Then in the db tables : the experimentdata are still
here, and their project is None.

So I think I won't succeed to fix the db. I'll try to see what to do
with backup, but the data I was deleting was the only available at the
moment, so deleting the db would not be a great problem. I would just
prefer not to ask people who already created an account to set again
their password ... Would it be possible that I only backup the
xdat_users table, empty/fill the db and restore only this table ?

Alexis

[Timothy Olsen]

Ah, I didn't realize the subjects were successfully deleted.

Add the dummy project again. Then add a dummy subject to that project. Then figure out what XNAT's ID (the accession) for that subject is.

Then run

UPDATE xnat_experimentData SET project='DUMMY_ID' WHERE project IS NULL;

UPDATE xnat_subjectassessordata SET subject_id='SUBJECT_ID' WHERE project IS NULL;

Then the delete project page should be functional again.

Tim

[Alexis Barbot]

I"m already trying this, but by deleting the dummy subject instead to directly delete the project ( probably there's no difference ). The only potential problem is that this means one subject with > 700 mrsessions ... I'll keep you informed !

Alexis

2009/4/17 Timothy Olsen <ti...@npg.wustl.edu>

[Gao, Yong]

More info:
towards the end, setup outputs this:

BUILD FAILED
File...... C:\xnat\maven.xml
Element... copy
Line...... 357
Column.... 97
C:\xnat\plugin-resources\webapp\xnat-templates not found.

[Gao, Yong]

There appears to be a small bug in maven.xml: c drive is hard-coded in <ant:arg
value="/c"/> (line 1267). I happened to use D

I've checked out the source again, and was able to do an installation with the
fixed maven.xml. However, the "Cannot run program "command.com" error is still
there. You won't see it if you let the set up run to the end.

Also, the set up script didn't recognize "JAVA_HOME" env viable even though it
was set. As a result, the scripts (StoreXML.bat etc) were not generated
properly.

Thanks,

Yong

[Alexis Barbot]

Hi,

Unfortunately it did not work. I first tried to delete the subject where I put all the data, and I got an error page. Then I tried to delete the project, and it said it was successful, but like before it deleted the subjects, but for the experimentdata simply put the project/subject to Null.

I tried to look at the logs, and what I found again many times the error "la fonction gettext_psytool_testdata... n'existe pas".

Well, I'm don't absolutely need to keep the history of the data, and at the moment there's only one project. The only thing I would prefer to keep are the users account. I think I will proceed like this ( as it seems to work ) :
 - from admin account, download xmls of users ( by selecting each users and click on "download xml" )
 - drop the database and recreate it ( possibly by deleting all tables/views )
 - re-create the users by storing the xmls

Alexis

2009/4/17 Alexis Barbot <alexis...@gmail.com>

[Timothy Olsen]

Alexis, you could export the users xml and then restore them.  FYI, you may run into problems if you don’t re-create your projects before storing the user XML.

 

I believe you could use the command prompt tool ‘Browse’ to export all of the user xmls.  It is preinstalled in XNAT_HOME/bin.  Pass in : -e xdat:user

 

Tim

[Timothy Olsen]

Yong,

I found a local copy of Vista to test on. Its 32-bit, but is also having some issues. It doesn't seem to like this ant command:

<ant:property environment="env"/>

It is throwing an error about '16-bit MS-DOS Subsystem'.

A little background info...

XNAT uses Maven to manage the install process (something we aren't necessarily excited about...). Maven allows the use of Ant commands. Maven 1.0.2 is included in XNAT/plugin-resources/maven.

Ant allowed the assignment of the environmental variable set to a property using the 'offending' command. This was used to access the JAVA_HOME value (env.JAVA_HOME). It appears something within the Ant assignment is breaking in Vista. My quick attempt at upgrading Maven's version of Ant (plugin-resources/maven/lib) didn't go smoothly. Maybe there is a better way to access the environmental variables in maven. It may take some time for me to resolve. You can take a stab at it, if you like.

In the meantime, it appears XNAT needs some work before it is usable on Vista.

Tim

Thanks,

Yong

Thanks,

Yong

xdat:verifyPLPGSQL:

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.tools.ant.taskdefs.Execute$Java13CommandLauncher.exec(Execute.java:646)
at org.apache.tools.ant.taskdefs.Execute.launch(Execute.java:416)
at org.apache.tools.ant.taskdefs.Execute.execute(Execute.java:427)
at org.apache.tools.ant.taskdefs.Execute.getProcEnvironment(Execute.java:180)
at org.apache.tools.ant.taskdefs.Property.loadEnvironment(Property.java:455)
at org.apache.tools.ant.taskdefs.Property.execute(Property.java:359)
at org.apache.tools.ant.Task.perform(Task.java:341)
at org.apache.commons.jelly.tags.ant.AntTag.doTag(AntTag.java:185)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:135)
at org.apache.maven.jelly.tags.werkz.MavenGoalTag.runBodyTag(MavenGoalTag.java:79)

at org.apache.maven.jelly.tags.werkz.MavenGoalTag$MavenGoalAction.performAction(MavenGoalTag.java:110)

at com.werken.werkz.Goal.fire(Goal.java:639)
at com.werken.werkz.Goal.attain(Goal.java:575)
at com.werken.werkz.WerkzProject.attainGoal(WerkzProject.java:193)

at org.apache.maven.jelly.tags.werkz.MavenAttainGoalTag.doTag(MavenAttainGoalTag.java:127)

at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:135)
at org.apache.maven.jelly.tags.werkz.MavenGoalTag.runBodyTag(MavenGoalTag.java:79)

at org.apache.maven.jelly.tags.werkz.MavenGoalTag$MavenGoalAction.performAction(MavenGoalTag.java:110)

at com.werken.werkz.Goal.fire(Goal.java:639)
at com.werken.werkz.Goal.attain(Goal.java:575)
at com.werken.werkz.WerkzProject.attainGoal(WerkzProject.java:193)

at org.apache.maven.jelly.tags.werkz.MavenAttainGoalTag.doTag(MavenAttainGoalTag.java:127)

at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:135)
at org.apache.maven.jelly.tags.werkz.MavenGoalTag.runBodyTag(MavenGoalTag.java:79)

at org.apache.maven.jelly.tags.werkz.MavenGoalTag$MavenGoalAction.performAction(MavenGoalTag.java:110)

at com.werken.werkz.Goal.fire(Goal.java:639)
at com.werken.werkz.Goal.attain(Goal.java:575)
at org.apache.maven.plugin.PluginManager.attainGoals(PluginManager.java:671)
at org.apache.maven.MavenSession.attainGoals(MavenSession.java:263)
at org.apache.maven.cli.App.doMain(App.java:488)
at org.apache.maven.cli.App.main(App.java:1239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

[Gao, Yong]

Thanks Tim.

Maybe eventually XNAT will part with Maven?
Another suggestion (it may already be on your todo list or have-tried list): if
there's a setenv script, other scripts such as StoreXML don't have to be
generated for each installation?

Yong

________________________________

[Alexis Barbot]

Ok, so for feedback I'll explain here the procedure I have tested and I'm about to apply. First precisions : I'm working with a server and a development station. The server contains the tomcat server and postgres; The development station contains the xnat files and a postgresql base ( which does not contains data : sql scripts are applied on the server ).

So :

 1- In the web interface, as admin, select one by one the users and save their xml in a folder
On the dev station :
 2- Drop database and recreate it.
 3- Run the intial script project_name.sql
 4- Run bin/update
 5- apply script project-update.sql
 6- store basics xmls ( security/security.xml and work/field_groups )
 7- Make a dump of the database ( pg_dump )
On server :
 8- Drop all tables :
  8i- \copy (SELECT 'DROP TABLE ' n.table_name ' CASCADE; ' FROM information_schema.tables as n WHERE table_schema='public') TO '/temp/drop_tables.sql'
  8ii- \i /temp/drop_tables.sql
 9- Also drop xdat_search schema : DROP SCHEMA xdat_search
 10- fill the database with the dump made before
 11- log as admin and re-create the projects
 12- Store the users xml

Alexis

2009/4/20 Timothy Olsen <ti...@npg.wustl.edu>