"GM" DICOM images fail to upload to XNAT
282 views
Skip to first unread message
Orringer Lab
Mar 8, 2021, 11:57:37 AM3/8/21
to xnat_discussion
Hi XNAT Team,
I am unable to upload DICOM images of the modality "GM" (general microscopy). I believe that the upload is failing because XNAT does not support that SOP class and modality corresponding to "GM." There is some documentation (https://wiki.xnat.org/documentation/how-to-use-xnat/image-session-upload-methods-in-xnat/how-xnat-translates-dicom-metadata) regarding how to customize DICOM metadata, but the BitBucket is no longer available. How can access the BitBucket, and how can I add compatibility for the SOP class and modality corresponding to "GM," so that these DICOM images can be uploaded to XNAT?
Appreciate your help.
I am unable to upload DICOM images of the modality "GM" (general microscopy). I believe that the upload is failing because XNAT does not support that SOP class and modality corresponding to "GM." There is some documentation (https://wiki.xnat.org/documentation/how-to-use-xnat/image-session-upload-methods-in-xnat/how-xnat-translates-dicom-metadata) regarding how to customize DICOM metadata, but the BitBucket is no longer available. How can access the BitBucket, and how can I add compatibility for the SOP class and modality corresponding to "GM," so that these DICOM images can be uploaded to XNAT?
Appreciate your help.
600s...@gmail.com
Mar 10, 2021, 10:00:53 AM3/10/21
to xnat_discussion
Hi, I don't have a solution for the larger problem at the ready, but I have fixed the link to the dicom-xnat repository in the documentation. That repo is now located here: https://bitbucket.org/xnatdcm/dicom-xnat
Regards,
Will
Moore, Charlie
Mar 10, 2021, 10:49:47 AM3/10/21
to xnat_di...@googlegroups.com
My hope is that the general problem will one day be fixed in XNAT itself. The ticket tracking the issue can be found here: https://issues.xnat.org/browse/XNAT-4527 .
Thanks,
Charlie
Charlie
From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of 600s...@gmail.com <600s...@gmail.com>
Sent: Wednesday, March 10, 2021 9:00 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Re: "GM" DICOM images fail to upload to XNAT
Sent: Wednesday, March 10, 2021 9:00 AM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Re: "GM" DICOM images fail to upload to XNAT
|
* External Email - Caution * |
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/8efeb822-49d6-41fe-8fef-c91948f356e3n%40googlegroups.com.
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/8efeb822-49d6-41fe-8fef-c91948f356e3n%40googlegroups.com.
The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.
James d'Arcy
Mar 10, 2021, 10:56:57 AM3/10/21
to xnat_discussion
Hi Will
Thanks for the new repo link. The docs linked by Orringer Lab contain a lot of links that reference the old dicom-xnat repository and they remain broken. It would be very helpful if these were updated too.
Thanks
James
600s...@gmail.com
Mar 10, 2021, 7:17:56 PM3/10/21
to xnat_discussion
Yes, that was my oversight. There are actually four different repos linked from the documentation, all of which had moved. I've updated all of the links including referenced line numbers when necessary, and added quick reference links to each of the four repos to the top of the page.
Regards,
Will
Orringer Lab
Mar 11, 2021, 9:02:59 AM3/11/21
to xnat_discussion
Hi Charlie and Will,
Thank you for updating those links! What changes would need to be implemented to the code base to add support for an unsupported modality, like "GM"?
Best,
Orringer Lab
Kate Alpert
Mar 11, 2021, 4:14:20 PM3/11/21
to xnat_di...@googlegroups.com
As Charlie mentioned, a fix to XNAT-4527, which would map "rarer" modalities to existing data types in XNAT should be forthcoming.
In the meantime, you can leverage a new 1.8 feature: Customizing DICOM Study/Series classification in XNAT (proper documentation in the works) to allow import of GM data as XnatGmsessiondata/XnatGmscandata. This feature allows you to instruct XNAT to classify incoming DICOM data as a custom XnatImagesessiondata- and/or XnatImagescandata- extending data type. But, you can also use it to classify incoming DICOM data as an existing data type.
To do so, you'll need to create a plugin that includes XnatGmsessiondataBeanFactory and XnatGmscandataBeanFactory Spring components (and ensure these are in package org.nrg.dcm.xnat), jar it up, drop it into your plugins directory, restart tomcat, and test!
I am attaching a guess at what these factories might look like. You might also wish to consult the bean factories included by default (in particular, XnatPetmrImagesessiondataBeanFactory.java as that's easiest to read IMHO) to learn more about how session and scan bean factories work.
Best,
Kate
--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/81170683-cbea-434a-8216-51ea954573adn%40googlegroups.com.
Orringer Lab
Mar 12, 2021, 8:41:04 AM3/12/21
to xnat_discussion
Hi Kate,
Thanks for those details! I'll see whether I can import the GM images by implementing an XNATGmsessiondata/XnatGmscandata.
James d'Arcy
Mar 12, 2021, 12:51:00 PM3/12/21
to xnat_discussion
@Will Thanks for updating the docs.
@Kate,
I've tried to add SM support to my plugin by using your instructions/examples with the only difference being GM -> SM but I'm still not able to create an SM session:
I guess I've missed some important step so I'm looking forward to the proper docs :)
2021-03-12 17:10:44,228 [http-bio-8080-exec-7] WARN org.nrg.dcm.xnat.DICOMSessionBuilder - Session builder not implemented for SOP class(es) [1.2.840.10008.5.1.4.1.1.77.1.6] or modality(ies) [SM], trying default session builder factory
I have added org.nrg.dcm.xnat to the @ComponentScan of the plugin class but breakpoints in the XnatSmsessiondataBeanFactory::create() method are never triggered when I try to upload via the XNAT Desktop Client.
I did manage to get an SM session to import (via XNAT Desktop Client) by changing and rebuilding dicom-xnat-mx and dicom-xnat-sop then replacing those jars inside ROOT.war and restarting Tomcat7. It's a bit of a hack but it did work and was a more direct approach to XNAT-4527.
I guess I've missed some important step so I'm looking forward to the proper docs :)
ttfn
James
Kate Alpert
Mar 12, 2021, 5:02:47 PM3/12/21
to xnat_di...@googlegroups.com
James,
Looks like there was a bug that was interfering with your efforts. It is fixed now on the develop line, and hopefully it'll be patched into master shortly.
Until the full documentation is ready, have a look at https://bitbucket.org/radiologics_kate/customize-dicom-classification-plugin. In addition to the files I mentioned earlier in this thread, you'll note:
I wrongly assumed these were populated for existing data types, but it turns out, you'll need to inject them alongside the factories. I added some example "additions" if you want to populate fields beyond the ImageSession/ImageScan defaults; these can be removed if you're not interested. More such samples abound in the dicom-xnat-mx codebase, e.g., https://bitbucket.org/xnatdcm/dicom-xnat/src/master/dicom-xnat-mx/src/main/java/org/nrg/dcm/xnat/CTScanAttributes.java.
Best,
Kate
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/aa0f9cd7-6bb9-4b6a-b92a-759d3a9ec953n%40googlegroups.com.
James d'Arcy
Mar 12, 2021, 5:12:58 PM3/12/21
to xnat_di...@googlegroups.com
Hi Kate
I'll take a look at that on Monday and probably steal those classes. I'm not familiar enough with SM yet to make a call on which fields beyond stock that will be needed but it's good to know I can add them if needed.
Thanks for looking into this and fixing it :)
ttfn
James
You received this message because you are subscribed to a topic in the Google Groups "xnat_discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/xnat_discussion/mvFyD4T3djE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/CAFe%2BCqcatUF0d6%3DVypzPcmsOPz1RAysdR2mLUZhN0HiTT7ryYw%40mail.gmail.com.
Ihor Mykhalevych
May 15, 2022, 9:36:50 AM5/15/22
to xnat_discussion
Greetings!
tldr; - How to politely bulk upload with the project level anonymisation or how to fix the CR (DX) upload in general?
The story:
I'm looking for a way to upload a lot (~18K images) of strongly de-identified data to my private XNAT. These are Kaggle competition data (https://www.kaggle.com/competitions/vinbigdata-chest-xray-abnormalities-detection/data). The problem is that those dcm files do not have the Modality tag set, but only the SOPClassUID one (1.2.840.10008.5.1.4.1.1.1 and 1.2.840.10008.5.1.4.1.1.1.1).
First problem I've encountered was the prearchive ERROR:
Almost fresh dockerized XNAT setup, with default site-wide anonymisation turned on.
Bulk upload with autogenerated subject id and session label.
XNAT Desktop Client error log:
Type Message Date/Time Details
error Session upload failed (with status code: 500 - ""). 2022-05-15 12:57:59 FAILED: Unable to archive FN450_undefined_1: org.xml.sax.SAXParseException; systemId: file:///data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. You can get technical details here. Please continue your visit at our home page.
success Series uploaded 1.2.826.0.1.3680043.2.1143.5315754475745396317638352422706470748. 2022-05-15 12:57:58 undefined
application.log:
2022-05-15 10:57:59,154 [http-nio-8080-exec-8] ERROR org.nrg.session.SessionBuilder - Unable to write session XML for /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1
java.lang.NullPointerException: null
2022-05-15 10:57:59,274 [http-nio-8080-exec-8] ERROR org.nrg.session.SessionBuilder - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1 must contain exactly one session
org.nrg.session.SessionBuilder$NoUniqueSessionException: No session found
at org.nrg.session.SessionBuilder.writeSession(SessionBuilder.java:210)
at org.nrg.session.SessionBuilder.run(SessionBuilder.java:249)
at org.nrg.xnat.archive.XNATSessionBuilder.buildPetSession(XNATSessionBuilder.java:227)
at org.nrg.xnat.archive.XNATSessionBuilder.call(XNATSessionBuilder.java:174)
at org.nrg.xnat.helpers.prearchive.PrearcUtils.buildSession(PrearcUtils.java:808)
at org.nrg.xnat.helpers.prearchive.PrearcUtils.buildSession(PrearcUtils.java:771)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase$13.extSync(PrearcDatabase.java:1035)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase$13.extSync(PrearcDatabase.java:1033)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase$LockAndSync.run(PrearcDatabase.java:1596)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase.buildSession(PrearcDatabase.java:1047)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase.buildSession(PrearcDatabase.java:1015)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.postActionCommit(PrearcSessionResource.java:402)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.handlePost(PrearcSessionResource.java:267)
at org.restlet.Finder.handle(Finder.java:357)
...
dicom.log:
2022-05-15 10:57:59,059 [http-nio-8080-exec-8] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Building session for 1.2.826.0.1.3680043.2.1143.3781799682068820516421578260039507767
2022-05-15 10:57:59,086 [http-nio-8080-exec-8] WARN org.nrg.dcm.xnat.DICOMSessionBuilder - Session builder not implemented for SOP class(es) [1.2.840.10008.5.1.4.1.1.1.1] or modality(ies) [null], trying default session builder factory
2022-05-15 10:57:59,090 [http-nio-8080-exec-8] INFO org.nrg.dcm.xnat.DefaultXnatImagesessiondataBeanFactory - Did not find a DICOM session type but did find scan type "dxScanData" for SOP class UIDs, returning generic session bean: [1.2.840.10008.5.1.4.1.1.1.1]
2022-05-15 10:57:59,152 [http-nio-8080-exec-8] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Warnings occurred in processing /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1; see org.nrg.sl4fj.PrintWriterLogger(/data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1/dcmtoxnat.log)
jms.log:
2022-05-15 10:57:59,437 [DefaultMessageListenerContainer-6] ERROR org.nrg.xnat.services.messaging.prearchive.PrearchiveOperationRequestListener - An error occurred processing a request from user admin to perform Archive operation on prearchive session at: /prearchive/projects/COVID/20220515_105754243/FN450_undefined_1
java.lang.IllegalStateException: org.xml.sax.SAXParseException; systemId: file:///data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at org.nrg.xnat.helpers.prearchive.PrearcDatabase._archive(PrearcDatabase.java:949)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase.archive(PrearcDatabase.java:935)
at org.nrg.xnat.helpers.prearchive.handlers.PrearchiveArchiveHandler.commitSessionToArchive(PrearchiveArchiveHandler.java:61)
at org.nrg.xnat.helpers.prearchive.handlers.PrearchiveArchiveHandler.execute(PrearchiveArchiveHandler.java:47)
at org.nrg.xnat.services.messaging.prearchive.PrearchiveOperationRequestListener.onRequest(PrearchiveOperationRequestListener.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:719)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:649)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1167)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1159)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1056)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1238)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:274)
at org.nrg.framework.services.SerializerService.parse(SerializerService.java:419)
at org.nrg.framework.services.SerializerService.parse(SerializerService.java:378)
at org.nrg.xft.schema.Wrappers.XMLWrapper.SAXReader.parse(SAXReader.java:136)
at org.nrg.xnat.turbine.utils.XNATSessionPopulater.populate(XNATSessionPopulater.java:56)
at org.nrg.xnat.archive.PrearcSessionArchiver.<init>(PrearcSessionArchiver.java:142)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase._archive(PrearcDatabase.java:946)
... 21 common frames omitted
prearchive.log:
2022-05-15 10:57:59,304 [http-nio-8080-exec-8] ERROR org.nrg.xnat.helpers.prearchive.PrearcUtils - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml is empty.
received.log:
2022-05-15 10:57:58,775 - admin:/data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1/SCANS/1_2_826_0_1_3680043_2_1143_5315754475745396317638352422706470748/DICOM/1.2.826.0.1.3680043.2.1143.3781799682068820516421578260039507767-no-value-for-SeriesNumber-no-value-for-InstanceNumber-1ebynrv.dcm
restlet.log:
2022-05-15 10:57:59,870 [http-nio-8080-exec-8] ERROR org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource -
org.nrg.action.ServerException: FAILED: Unable to archive FN450_undefined_1: org.xml.sax.SAXParseException; systemId: file:///data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at org.nrg.xnat.archive.QueueBasedImageCommit.call(QueueBasedImageCommit.java:169)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.postActionCommit(PrearcSessionResource.java:414)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.handlePost(PrearcSessionResource.java:267)
at org.restlet.Finder.handle(Finder.java:357)
...
<prearchive>/COVID/20220515_105754243/FN450_undefined_1/dcmtoxnat.log:
session attribute date has no value
session attribute time has no value
session attribute scanner/model has no value
session attribute study_id has no value
session attribute dcmPatientWeight has no value
session attribute modality has no value
session attribute dcmPatientBirthDate has no value
session attribute scanner/manufacturer has no value
session attribute dcmAccessionNumber has no value
session attribute acquisition_site has no value
session attribute fields/field has no value
session attribute operator has no value
This leads also to the "archive pending" status in the project's prearchive.
Rebuilding gives next logs:
application.log:
2022-05-15 12:08:42,446 [DefaultMessageListenerContainer-10] ERROR org.nrg.session.SessionBuilder - Unable to write session XML for /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1
java.lang.NullPointerException: null
2022-05-15 12:08:42,563 [DefaultMessageListenerContainer-10] ERROR org.nrg.session.SessionBuilder - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1 must contain exactly one session
org.nrg.session.SessionBuilder$NoUniqueSessionException: No session found
at org.nrg.session.SessionBuilder.writeSession(SessionBuilder.java:210)
at org.nrg.session.SessionBuilder.run(SessionBuilder.java:249)
...
dicom.log:
2022-05-15 12:08:42,361 [DefaultMessageListenerContainer-10] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Building session for 1.2.826.0.1.3680043.2.1143.3781799682068820516421578260039507767
2022-05-15 12:08:42,375 [DefaultMessageListenerContainer-10] WARN org.nrg.dcm.xnat.DICOMSessionBuilder - Session builder not implemented for SOP class(es) [1.2.840.10008.5.1.4.1.1.1.1] or modality(ies) [null], trying default session builder factory
2022-05-15 12:08:42,379 [DefaultMessageListenerContainer-10] INFO org.nrg.dcm.xnat.DefaultXnatImagesessiondataBeanFactory - Did not find a DICOM session type but did find scan type "dxScanData" for SOP class UIDs, returning generic session bean: [1.2.840.10008.5.1.4.1.1.1.1]
2022-05-15 12:08:42,444 [DefaultMessageListenerContainer-10] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Warnings occurred in processing /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1; see org.nrg.sl4fj.PrintWriterLogger(/data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1/dcmtoxnat.log)
prearchive.log:
2022-05-15 12:08:42,635 [DefaultMessageListenerContainer-10] ERROR org.nrg.xnat.helpers.prearchive.PrearcUtils - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml is empty.
turbine.log:
2022-05-15 12:08:46,505 [http-nio-8080-exec-4] ERROR org.nrg.xdat.turbine.modules.screens.SecureScreen -
org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
...
xdat.log:
2022-05-15 12:08:42,168 [DefaultMessageListenerContainer-10] ERROR org.nrg.xnat.archive.XNATSessionBuilder - Couldn't parse previous session xml /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml, no resources will be copied into the rebuilt version
org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
...
After I found this thread, I started creating a plugin. During it, reading logs carefully brought an idea to use the anonymisation as a workaround.
Setting the Modality tag in the anonymisation script for the project does the job for the time-being:
(0008,0060) := "CR"
The dcm file gets uploaded and lands into the subject's CR experiment.
But after enabling it I'm getting the "Project anonymisation script contains variables. Only single session upload is available." message in the XNAT Desktop Client. Is there any polite way to bulk upload with the project level anonymization? Currently I'm turning it off before selecting the project in the Client and later turning it on right before clicking the "Finish and Upload" button.
Another thing, when i'm trying to be more specific in the tag setting:
// Set Modality based on SOPClassUID.
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1.1" ? (0008,0060) := "DX"
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1" ? (0008,0060) := "CR"
it reverts me back to the previous state of not being able to upload the dcm at all - errors from the above are back again.
Best regards,
Ihor
tldr; - How to politely bulk upload with the project level anonymisation or how to fix the CR (DX) upload in general?
The story:
I'm looking for a way to upload a lot (~18K images) of strongly de-identified data to my private XNAT. These are Kaggle competition data (https://www.kaggle.com/competitions/vinbigdata-chest-xray-abnormalities-detection/data). The problem is that those dcm files do not have the Modality tag set, but only the SOPClassUID one (1.2.840.10008.5.1.4.1.1.1 and 1.2.840.10008.5.1.4.1.1.1.1).
First problem I've encountered was the prearchive ERROR:
Almost fresh dockerized XNAT setup, with default site-wide anonymisation turned on.
Bulk upload with autogenerated subject id and session label.
XNAT Desktop Client error log:
Type Message Date/Time Details
error Session upload failed (with status code: 500 - ""). 2022-05-15 12:57:59 FAILED: Unable to archive FN450_undefined_1: org.xml.sax.SAXParseException; systemId: file:///data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml; lineNumber: 1; columnNumber: 1; Premature end of file. You can get technical details here. Please continue your visit at our home page.
success Series uploaded 1.2.826.0.1.3680043.2.1143.5315754475745396317638352422706470748. 2022-05-15 12:57:58 undefined
application.log:
2022-05-15 10:57:59,154 [http-nio-8080-exec-8] ERROR org.nrg.session.SessionBuilder - Unable to write session XML for /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1
java.lang.NullPointerException: null
2022-05-15 10:57:59,274 [http-nio-8080-exec-8] ERROR org.nrg.session.SessionBuilder - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1 must contain exactly one session
org.nrg.session.SessionBuilder$NoUniqueSessionException: No session found
at org.nrg.session.SessionBuilder.writeSession(SessionBuilder.java:210)
at org.nrg.session.SessionBuilder.run(SessionBuilder.java:249)
at org.nrg.xnat.archive.XNATSessionBuilder.buildPetSession(XNATSessionBuilder.java:227)
at org.nrg.xnat.archive.XNATSessionBuilder.call(XNATSessionBuilder.java:174)
at org.nrg.xnat.helpers.prearchive.PrearcUtils.buildSession(PrearcUtils.java:808)
at org.nrg.xnat.helpers.prearchive.PrearcUtils.buildSession(PrearcUtils.java:771)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase$13.extSync(PrearcDatabase.java:1035)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase$13.extSync(PrearcDatabase.java:1033)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase$LockAndSync.run(PrearcDatabase.java:1596)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase.buildSession(PrearcDatabase.java:1047)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase.buildSession(PrearcDatabase.java:1015)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.postActionCommit(PrearcSessionResource.java:402)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.handlePost(PrearcSessionResource.java:267)
at org.restlet.Finder.handle(Finder.java:357)
...
dicom.log:
2022-05-15 10:57:59,059 [http-nio-8080-exec-8] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Building session for 1.2.826.0.1.3680043.2.1143.3781799682068820516421578260039507767
2022-05-15 10:57:59,086 [http-nio-8080-exec-8] WARN org.nrg.dcm.xnat.DICOMSessionBuilder - Session builder not implemented for SOP class(es) [1.2.840.10008.5.1.4.1.1.1.1] or modality(ies) [null], trying default session builder factory
2022-05-15 10:57:59,090 [http-nio-8080-exec-8] INFO org.nrg.dcm.xnat.DefaultXnatImagesessiondataBeanFactory - Did not find a DICOM session type but did find scan type "dxScanData" for SOP class UIDs, returning generic session bean: [1.2.840.10008.5.1.4.1.1.1.1]
2022-05-15 10:57:59,152 [http-nio-8080-exec-8] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Warnings occurred in processing /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1; see org.nrg.sl4fj.PrintWriterLogger(/data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1/dcmtoxnat.log)
jms.log:
2022-05-15 10:57:59,437 [DefaultMessageListenerContainer-6] ERROR org.nrg.xnat.services.messaging.prearchive.PrearchiveOperationRequestListener - An error occurred processing a request from user admin to perform Archive operation on prearchive session at: /prearchive/projects/COVID/20220515_105754243/FN450_undefined_1
java.lang.IllegalStateException: org.xml.sax.SAXParseException; systemId: file:///data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at org.nrg.xnat.helpers.prearchive.PrearcDatabase._archive(PrearcDatabase.java:949)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase.archive(PrearcDatabase.java:935)
at org.nrg.xnat.helpers.prearchive.handlers.PrearchiveArchiveHandler.commitSessionToArchive(PrearchiveArchiveHandler.java:61)
at org.nrg.xnat.helpers.prearchive.handlers.PrearchiveArchiveHandler.execute(PrearchiveArchiveHandler.java:47)
at org.nrg.xnat.services.messaging.prearchive.PrearchiveOperationRequestListener.onRequest(PrearchiveOperationRequestListener.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:180)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:112)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:104)
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:69)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:719)
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:679)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:649)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:317)
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:255)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1167)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1159)
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1056)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1238)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:326)
at javax.xml.parsers.SAXParser.parse(SAXParser.java:274)
at org.nrg.framework.services.SerializerService.parse(SerializerService.java:419)
at org.nrg.framework.services.SerializerService.parse(SerializerService.java:378)
at org.nrg.xft.schema.Wrappers.XMLWrapper.SAXReader.parse(SAXReader.java:136)
at org.nrg.xnat.turbine.utils.XNATSessionPopulater.populate(XNATSessionPopulater.java:56)
at org.nrg.xnat.archive.PrearcSessionArchiver.<init>(PrearcSessionArchiver.java:142)
at org.nrg.xnat.helpers.prearchive.PrearcDatabase._archive(PrearcDatabase.java:946)
... 21 common frames omitted
prearchive.log:
2022-05-15 10:57:59,304 [http-nio-8080-exec-8] ERROR org.nrg.xnat.helpers.prearchive.PrearcUtils - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml is empty.
received.log:
2022-05-15 10:57:58,775 - admin:/data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1/SCANS/1_2_826_0_1_3680043_2_1143_5315754475745396317638352422706470748/DICOM/1.2.826.0.1.3680043.2.1143.3781799682068820516421578260039507767-no-value-for-SeriesNumber-no-value-for-InstanceNumber-1ebynrv.dcm
restlet.log:
2022-05-15 10:57:59,870 [http-nio-8080-exec-8] ERROR org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource -
org.nrg.action.ServerException: FAILED: Unable to archive FN450_undefined_1: org.xml.sax.SAXParseException; systemId: file:///data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
at org.nrg.xnat.archive.QueueBasedImageCommit.call(QueueBasedImageCommit.java:169)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.postActionCommit(PrearcSessionResource.java:414)
at org.nrg.xnat.restlet.resources.prearchive.PrearcSessionResource.handlePost(PrearcSessionResource.java:267)
at org.restlet.Finder.handle(Finder.java:357)
...
<prearchive>/COVID/20220515_105754243/FN450_undefined_1/dcmtoxnat.log:
session attribute date has no value
session attribute time has no value
session attribute scanner/model has no value
session attribute study_id has no value
session attribute dcmPatientWeight has no value
session attribute modality has no value
session attribute dcmPatientBirthDate has no value
session attribute scanner/manufacturer has no value
session attribute dcmAccessionNumber has no value
session attribute acquisition_site has no value
session attribute fields/field has no value
session attribute operator has no value
This leads also to the "archive pending" status in the project's prearchive.
Rebuilding gives next logs:
application.log:
2022-05-15 12:08:42,446 [DefaultMessageListenerContainer-10] ERROR org.nrg.session.SessionBuilder - Unable to write session XML for /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1
java.lang.NullPointerException: null
2022-05-15 12:08:42,563 [DefaultMessageListenerContainer-10] ERROR org.nrg.session.SessionBuilder - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1 must contain exactly one session
org.nrg.session.SessionBuilder$NoUniqueSessionException: No session found
at org.nrg.session.SessionBuilder.writeSession(SessionBuilder.java:210)
at org.nrg.session.SessionBuilder.run(SessionBuilder.java:249)
...
dicom.log:
2022-05-15 12:08:42,361 [DefaultMessageListenerContainer-10] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Building session for 1.2.826.0.1.3680043.2.1143.3781799682068820516421578260039507767
2022-05-15 12:08:42,375 [DefaultMessageListenerContainer-10] WARN org.nrg.dcm.xnat.DICOMSessionBuilder - Session builder not implemented for SOP class(es) [1.2.840.10008.5.1.4.1.1.1.1] or modality(ies) [null], trying default session builder factory
2022-05-15 12:08:42,379 [DefaultMessageListenerContainer-10] INFO org.nrg.dcm.xnat.DefaultXnatImagesessiondataBeanFactory - Did not find a DICOM session type but did find scan type "dxScanData" for SOP class UIDs, returning generic session bean: [1.2.840.10008.5.1.4.1.1.1.1]
2022-05-15 12:08:42,444 [DefaultMessageListenerContainer-10] INFO org.nrg.dcm.xnat.DICOMSessionBuilder - Warnings occurred in processing /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1; see org.nrg.sl4fj.PrintWriterLogger(/data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1/dcmtoxnat.log)
prearchive.log:
2022-05-15 12:08:42,635 [DefaultMessageListenerContainer-10] ERROR org.nrg.xnat.helpers.prearchive.PrearcUtils - /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml is empty.
turbine.log:
2022-05-15 12:08:46,505 [http-nio-8080-exec-4] ERROR org.nrg.xdat.turbine.modules.screens.SecureScreen -
org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
...
xdat.log:
2022-05-15 12:08:42,168 [DefaultMessageListenerContainer-10] ERROR org.nrg.xnat.archive.XNATSessionBuilder - Couldn't parse previous session xml /data/xnat/prearchive/COVID/20220515_105754243/FN450_undefined_1.xml, no resources will be copied into the rebuilt version
org.xml.sax.SAXParseException: Premature end of file.
at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203)
...
After I found this thread, I started creating a plugin. During it, reading logs carefully brought an idea to use the anonymisation as a workaround.
Setting the Modality tag in the anonymisation script for the project does the job for the time-being:
(0008,0060) := "CR"
The dcm file gets uploaded and lands into the subject's CR experiment.
But after enabling it I'm getting the "Project anonymisation script contains variables. Only single session upload is available." message in the XNAT Desktop Client. Is there any polite way to bulk upload with the project level anonymization? Currently I'm turning it off before selecting the project in the Client and later turning it on right before clicking the "Finish and Upload" button.
Another thing, when i'm trying to be more specific in the tag setting:
// Set Modality based on SOPClassUID.
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1.1" ? (0008,0060) := "DX"
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1" ? (0008,0060) := "CR"
it reverts me back to the previous state of not being able to upload the dcm at all - errors from the above are back again.
Best regards,
Ihor
пʼятниця, 12 березня 2021 р. о 23:12:58 UTC+1 jamesd...@gmail.com пише:
Charlie Moore
May 16, 2022, 4:16:00 PM5/16/22
to xnat_discussion
Hi Ihor,
2. "Project anonymization script contains variables": this is a bug in the XNAT Desktop Client. I've filed a ticket for it, but I don't know when it would be addressed. The desktop client wants to avoid user-defined variables in anon within the bulk upload workflow because the UI would become a nightmare to support that. However, your script fragment is using any settable variables, so it shouldn't be stopping you.
// Set Modality based on SOPClassUID.
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1.1" ? (0008,0060) := "DX"
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1" ? (0008,0060) := "CR"
First me say: I appreciate the amount of detail and experimentation you've put into your post; it really helps a lot. It sounds like you're running into 3 distinct issues:
1. Missing Modality: (0008,0060) Modality is defined as type 1 within the General Series Module, so for pretty much any DICOM data you have, it's going to be noncompliant to have an empty or missing value for it. As such, XNAT doesn't really support importing data that is missing the Modality header.
2. "Project anonymization script contains variables": this is a bug in the XNAT Desktop Client. I've filed a ticket for it, but I don't know when it would be addressed. The desktop client wants to avoid user-defined variables in anon within the bulk upload workflow because the UI would become a nightmare to support that. However, your script fragment is using any settable variables, so it shouldn't be stopping you.
3. Anonymization issues: this one I can't replicate. I tried the snippet you provided to make this script:
----
----
version "6.1"
// Set Modality based on SOPClassUID.
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1.1" ? (0008,0060) := "DX"
(0008,0016) = "1.2.840.10008.5.1.4.1.1.1" ? (0008,0060) := "CR"
----
and it worked just fine for me in the Desktop client. Can you provide the full script you're using? Also to confirm: the instances in each series have exactly one SOP Class UID, right? I could see there being problems if the instances have mixed SOP classes and so you ended up with a "series" that's half DX and half CR.
Overall, you might want to just modify the data locally first with something like dcmodify from dcmtk or with DicomBrowser to correct the modality header. That would let you skip all of the things you've run into here.
Thanks,
Charlie
Reply all
Reply to author
Forward
0 new messages