Extracting additional tag from DICOM file during import to XNAT

Simon Doran

unread,

Apr 24, 2013, 4:38:06 PM4/24/13

to xnat_di...@googlegroups.com

This is probably one for Kevin again ...

I want to make a customisation that is, in principle, very simple.

Our hospital routinely stores the hospital ID of the patient in the "Other Patient IDs" DICOM tag (0010, 1000). I have already defined an extension to xnat:demographicData (icr:clinicalDemographicData) and it will be straightforward to write some independent Java code to extract the correct tag and upload it using the REST client to icr:clinicalDemographicData/hospitalID in XNAT's Postgres database. I could tag this code onto the end of the autorun pipeline so that it gets executed at upload time.

However, this seems a bit of a long way round. Is there a neater way of achieving this by making some sort of trivial modification to the standard nrg code that performs the original upload?

Thanks in advance,

Simon

Archie, Kevin

unread,

Apr 25, 2013, 12:34:41 PM4/25/13

to xnat_di...@googlegroups.com

Simon,

I wish I had a simple answer for you. Unfortunately, the DICOM to XNAT metadata translation is all specified in Java code, in its own library (https://bitbucket.org/nrg/dicom-xnat, in dicom-xnat-mx). It’s generally straightforward to add a new field, except that then you have to rebuild the dicom-xnat-mx jar, substitute it into plugin-resources/repository/dcm/jars, and run update.sh. I’ve done this for the Human Connectome Project, where I’ve changed the jar name to indicate the customization version. You can see the sorts of changes I’ve made at https://bitbucket.org/hcp/dicom-xnat-intradb . The webapp changes are not as easy to show off, because the HCP build process is deeply idiosyncratic, but that’s really just change the jar, and change plugin-resources/originals/project.xml if you’re changing the jar name to indicate a customization version (which I recommend).

That’s all just for changing the fields in a data type where we’re already doing a translation. Extracting DICOM metadata into a distinct type is another kettle of fish. I can’t think of an example of doing this since we moved the DICOM and session building into XNAT (for version 1.5). I’m vaguely thinking that GradualDicomImporter might be the right place to do all this, but I don’t have concrete ideas. Tim or Rick might have something smart to say.

- Kevin

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To post to this group, send email to xnat_di...@googlegroups.com.
Visit this group at http://groups.google.com/group/xnat_discussion?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Simon Doran

unread,

Apr 26, 2013, 4:14:18 AM4/26/13

to xnat_di...@googlegroups.com

Thanks, Kevin.

I'll revert to Plan A then and let you know how it pans out.

Simon

Simon Doran

unread,

Apr 26, 2013, 5:39:16 PM4/26/13

to xnat_di...@googlegroups.com

I decided to use Java to create a Unix command to use in an XNAT pipeline.

The first half of the problem is fine. It was easy to extract the required parameters with DCM4CHE. I then thought it would be equally straightforward to use the instructions at https://wiki.xnat.org/display/XNAT16/Using+the+XNAT+REST+API to modify the desired field using REST with a query string. I started by simply trying to replicate the example in the docs:

XNATRestClient -host http://central.xnat.org -u USERNAME -p PASSWORD -m PUT -remote "/data/archive/projects/YOURTEST/subjects/1000?xnat:subjectData/demographics@xsi:type=xnat.demographicData/gender=male

However, when I ran this with my own projects and subjects, I had no success. I have a few questions:

1. In the example, should it really be xsi:type=xnat.demographicData or is it supposed to be xsi:type=xnat:demographicData?

Neither seemed to work for me.

2. Should I be able to perform the equivalent query using -m GET?

XNATRestClient ... -m GET -remote "/data/archive/projects/YOURTEST/subjects/1000?xnat:subjectData/demographics@xsi:type=xnat.demographicData/gender&format=xml"

This doesn't seem to work. Instead, the whole of the subject element is listed.

3. How do I find the contents of "xnat:demographicData" as a whole? If I try and find it by listing out the entire subject XML via

XNATRestClient ... -m GET -remote /data/archive/projects/YOURTEST/subjects/1000?format=xml

then the demographics entry I get is:

<xnat:demographics>

</xnat:demographics>

4. I thought an alternative solution might be to try and get the information as a column in a listing via:

XNATRestClient ... -m GET -remote "/data/archive/projects/TEST/subjects?columns=xnat:subjectData/demographics@xsi:type=demographicData/gender&format=xml"

but that simply ignored my request and just output the standard set of columns without any demographic data.

Is all this expected behaviour or do I have a problem with my installation (1.6.1dev running on RHEL 6 with postgres 9.2)?

The idea that there might be something more fundamental wrong is suggested by the fact that when I try to make the change interactively, it doesn't work either. If I click on "Edit" on the subject report page and try to change the Gender pull-down menu I get the message "Error Saving item" when I press the Submit button. In xdat.log, I found:

2013-04-26 21:48:45,266 [http-bio-8080-exec-8] ERROR org.nrg.xft.db.DBAction - Error saving item:

<XFTItem name="xnat:demographicData">^M

<age>NULL</age>^M

<gender>male</gender>^M

<xnat_abstractdemographicdata_id>2</xnat_abstractdemographicdata_id>^M

<yob>NULL</yob>^M

<abstractdemographicdata_abstractdemographicdata>^M

<XFTItem name="xnat:abstractDemographicData">^M

<xnat_abstractdemographicdata_id>2</xnat_abstractdemographicdata_id>^M

<extension_meta_element>^M

<XFTItem name="xdat:meta_element">^M

<element_name>xnat:demographicData</element_name>^M

</XFTItem>^M

</extension_meta_element>^M

</XFTItem>^M

</abstractdemographicdata_abstractdemographicdata>^M

</XFTItem>^M

2013-04-26 21:48:45,269 [http-bio-8080-exec-8] ERROR org.nrg.xft.db.DBAction - Error saving item:

<XFTItem name="xnat:subjectData">^M

<demographics_xnat_abstractdemographicdata_id>2</demographics_xnat_abstractdemographicdata_id>^M

<id>XNAT_VALHALLA_S00002</id>^M

<label>SqijPhnt06</label>^M

<project>TEST</project>^M

<subjectdata_info>2</subjectdata_info>^M

<demographics_abstractdemographicdata>^M

<XFTItem name="xnat:demographicData">^M

<age>NULL</age>^M

<gender>male</gender>^M

<xnat_abstractdemographicdata_id>2</xnat_abstractdemographicdata_id>^M

<yob>NULL</yob>^M

<abstractdemographicdata_abstractdemographicdata>^M

<XFTItem name="xnat:abstractDemographicData">^M

<xnat_abstractdemographicdata_id>2</xnat_abstractdemographicdata_id>^M

<extension_meta_element>^M

<XFTItem name="xdat:meta_element">^M

<element_name>xnat:demographicData</element_name>^M

</XFTItem>^M

</extension_meta_element>^M

</XFTItem>^M

</abstractdemographicdata_abstractdemographicdata>^M

</XFTItem>^M

</demographics_abstractdemographicdata>^M

</XFTItem>^M

2013-04-26 21:48:45,329 [http-bio-8080-exec-8] ERROR org.nrg.xnat.turbine.modules.actions.EditSubjectAction - Error Storing xnat:subjectData

org.postgresql.util.PSQLException: ERROR: function ie_xnat_demographicdata(integer, integer, boolean, boolean, boolean) does not exist

Hint: No function matches the given name and argument types. You might need to add explicit type casts.

Position: 8

at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)

at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)

at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:254)

at org.apache.commons.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:205)

at org.nrg.xft.db.PoolDBUtils.executeQuery(PoolDBUtils.java:890)

at org.nrg.xft.db.PoolDBUtils.returnStatisticQuery(PoolDBUtils.java:367)

at org.nrg.xft.db.PoolDBUtils.ReturnStatisticQuery(PoolDBUtils.java:390)

at org.nrg.xft.search.ItemSearch.getItemsFromKeys(ItemSearch.java:396)

at org.nrg.xft.XFTItem.getPkMatches(XFTItem.java:2255)

All suggestions gratefully received.

Simon

Timothy Olsen

unread,

Apr 29, 2013, 12:29:03 PM4/29/13

to xnat_di...@googlegroups.com

Dealing with XML Path in the REST api can be problematic. But the error log at the bottom of your email is disconcerting.

org.nrg.xnat.turbine.modules.actions.EditSubjectAction - Error Storing xnat:subjectData

org.postgresql.util.PSQLException: ERROR: function ie_xnat_demographicdata(integer, integer, boolean, boolean, boolean) does not exist

It seems like the generated sql functions might be out of sync with what XNAT expects. Did you run the generated *-update.sql file last time you updated your server? It might be worth re-running the update.sh script and executing the generated update sql.

Tim

Simon Doran

unread,

May 1, 2013, 5:31:56 AM5/1/13

to xnat_di...@googlegroups.com

Tim,

Thanks very much for the hint above.

Apologies in advance for another overlong and technical e-mail!!

If you don't have time to read anything else, the bottom line is:

I still really need some help in setting an xsi:type field via REST, but once it is in there, I can extract it OK, just a little inelegantly by parsing the entire subject XML.

First, the good news. Doh! I thought that there must have been a user error there somewhere!! Re-running the update SQL did indeed sort out the error messages at the end of the last post and it seems that I am now able to store demographics successfully via the UI. I set the relevant values for a phantom and two subjects and looking at the relevant table demonstrates that it worked.

xnat_abstractdemographicdata_id | dob | yob | age | gender | handedness | ... | demographicdata_info

---------------------------------+-----+-----+-----+---------+------------+-...-+----------------------

3 | | | | male | | ... | 2

1 | | | | unknown | | ... | 1

4 | | | | female | | ... | 3

This is only a qualified success, though, as velocity.log has loads of errors associated with xnat_subjectData_edit_core_demographics.vm

2013-05-01 10:10:39,729 [localhost-startStop-1] ERROR velocity - VM #xdatStringBoxOnChange: error : too few arguments to macro. Wanted 6 got 0

2013-05-01 10:10:39,758 [localhost-startStop-1] ERROR velocity - VM #xdatSearchForm: error : too few arguments to macro. Wanted 2 got 0

2013-05-01 10:10:39,796 [localhost-startStop-1] ERROR velocity - VM #displayFileSystem: error : too few arguments to macro. Wanted 1 got 0

2013-05-01 10:10:39,797 [localhost-startStop-1] ERROR velocity - VM #auditBoxes: error : too few arguments to macro. Wanted 4 got 0

2013-05-01 10:11:12,384 [http-bio-8080-exec-10] ERROR velocity - VMProxyArg.setObject() : Programmer error : I am a constant! No setting! : dataType / "xnat:projectData"

2013-05-01 10:11:12,385 [http-bio-8080-exec-10] ERROR velocity - VMProxyArg.setObject() : Programmer error : I am a constant! No setting! : dataType / "xnat:projectData"

2013-05-01 10:11:12,386 [http-bio-8080-exec-10] ERROR velocity - VMProxyArg.setObject() : Programmer error : I am a constant! No setting! : dataType / "xnat:projectData"

2013-05-01 10:11:12,710 [http-bio-8080-exec-10] ERROR velocity - Left side ($om.getItem().getGenericSchemaElement().getPrimaryElements().indexOf("xnat:experimentData")) of '!=' operation has null value. Operation not possible. navigations//Breadcrumb.vm [line 177, column 102]

2013-05-01 10:11:15,869 [http-bio-8080-exec-8] ERROR velocity - #parse() error : null argument

2013-05-01 10:11:16,109 [http-bio-8080-exec-8] ERROR velocity - Left side ($om.getItem().getGenericSchemaElement().getPrimaryElements().indexOf("xnat:experimentData")) of '!=' operation has null value. Operation not possible. navigations//Breadcrumb.vm [line 177, column 102]

2013-05-01 10:11:24,565 [http-bio-8080-exec-9] ERROR velocity - Left side ($matches.size()) of '>' operation has null value. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit.vm [line 19, column 53]

2013-05-01 10:11:24,592 [http-bio-8080-exec-9] ERROR velocity - RHS of #set statement is null. Context will not be modified. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 32, column 1]

2013-05-01 10:11:24,593 [http-bio-8080-exec-9] ERROR velocity - Right side ($currentYear) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 36, column 101]

.

2013-05-01 10:11:24,606 [http-bio-8080-exec-9] ERROR velocity - RHS of #set statement is null. Context will not be modified. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 41, column 8]

2013-05-01 10:11:24,607 [http-bio-8080-exec-9] ERROR velocity - RHS of #set statement is null. Context will not be modified. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 98, column 26]

2013-05-01 10:11:24,607 [http-bio-8080-exec-9] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 101, column 53]

2013-05-01 10:11:24,607 [http-bio-8080-exec-9] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 102, column 63]

2013-05-01 10:11:24,607 [http-bio-8080-exec-9] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 103, column 61]

2013-05-01 10:11:24,607 [http-bio-8080-exec-9] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 104, column 77]

2013-05-01 10:11:24,607 [http-bio-8080-exec-9] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 105, column 67]

2013-05-01 10:11:24,763 [http-bio-8080-exec-9] ERROR velocity - Left side ($om.getItem().getGenericSchemaElement().getPrimaryElements().indexOf("xnat:experimentData")) of '!=' operation has null value. Operation not possible. navigations//Breadcrumb.vm [line 177, column 102]

2013-05-01 10:11:31,569 [http-bio-8080-exec-10] ERROR velocity - #parse() error : null argument

2013-05-01 10:11:31,726 [http-bio-8080-exec-10] ERROR velocity - Left side ($om.getItem().getGenericSchemaElement().getPrimaryElements().indexOf("xnat:experimentData")) of '!=' operation has null value. Operation not possible. navigations//Breadcrumb.vm [line 177, column 102]

The fact that I can set demographic info via the web interface does not sort out my previous queries:

1. XNATRestClient ... -m PUT -remote /data/archive/projects/PROJ/subjects/XNAT_S00015?xnat:subjectData/demographics@xsi:type=xnat.demographicData/gender=unknown

does not seem to work with either xnat.demographicData (as per docs) or xnat:demographicData (which is what I would have expected).

In both cases the REST call returns the subject ID on the command line, but fails with an error message (restlet.log) of form:

2013-05-01 10:16:41,403 [http-bio-8080-exec-3] ERROR org.nrg.xnat.restlet.resources.SecureResource -

org.nrg.xft.exception.FieldNotFoundException: Field not found: 'xnat:subjectData/demographics@xsi:type=xnat.demographicData/gender'

at org.nrg.xft.GenericItemObject.setProperties(GenericItemObject.java:327)

2. It would be really nice if a call such as

XNATRestClient ... -m GET -remote "/data/archive/projects/PROJECT/subjects/XNAT_S00015?xnat:subjectData/demographics@xsi:type=xnat:demographicData/gender&format=xml"

were to return just the desired element. But it does not seem to do this. Instead it returns the whole subject element. No error in restlet.log.

3. Progress on this, at least! Now that I have managed to set the xsi:type fields via the web interface, the full subject listing does what I expect:

XNATRestClient ... -m GET -remote "/data/archive/projects/PROJECT/subjects/XNAT_S00015?format=xml

<xnat:demographics xsi:type="xnat:demographicData">

<xnat:gender>female</xnat:gender> <<<<<<< Which was what I set it to in the web interface, before trying to reset it to "unknown" via the REST call

</xnat:demographics>

4. The column I want is still ignored when I try to a listing of form:

XNATRestClient ... -m GET -remote "/data/archive/projects/RPYS_RPACS01/subjects?columns=xnat:subjectData/demographics@xsi:type=xnat.demographicData/gender&format=xml"

This fails with:

2013-05-01 10:22:48,791 [http-bio-8080-exec-4] ERROR org.nrg.xnat.restlet.resources.SecureResource -

org.nrg.xft.exception.ElementNotFoundException: Element not found: 'columns=xnat:subjectData'

at org.nrg.xft.meta.XFTMetaManager.FindElement(XFTMetaManager.java:224)

Thanks in advance.

Simon

Postscript:

This is probably completely unconnected, but as it does refer to an abstract resource, I will also note this that I found in xdat.log.

2013-04-30 15:03:21,926 [http-bio-8080-exec-6] ERROR org.nrg.xdat.om.base.auto.AutoXnatAbstractresource -

org.postgresql.util.PSQLException: ERROR: column "snapshots" does not exist

Position: 241