Unable to delete subject/session in certain circumstance

[akluiber]

Running the current docker-compose 1.8.3 build 5, and I'm having an issue where I can delete a session, then the subject. But I can't delete the subject first (which should also delete the session). This results in an error, after which I don't seem to be able to delete either the subject or session after that.

If I re-upload the images and merge the session, I can then delete the session and, subsequently, the subject correctly.

Error is showing up in xdat.log.

2022-02-04 21:11:31,283 [http-nio-8080-exec-9] ERROR org.nrg.xdat.om.base.auto.AutoXnatExperimentdata - 

java.lang.NullPointerException: null

at org.nrg.xft.db.DBAction.RemoveItemReference(DBAction.java:2470)

at org.nrg.xft.db.DBAction.DeleteItem(DBAction.java:2951)

at org.nrg.xft.db.DBAction.DeleteItem(DBAction.java:2990)

at org.nrg.xft.db.DBAction.DeleteItem(DBAction.java:2866)

at org.nrg.xft.utils.SaveItemHelper.delete(SaveItemHelper.java:96)

at org.nrg.xft.utils.SaveItemHelper$$FastClassBySpringCGLIB$$64be15d8.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:737)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:55)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:672)

at org.nrg.xft.utils.SaveItemHelper$$EnhancerBySpringCGLIB$$dd5e042d.delete(<generated>)

at org.nrg.xft.utils.SaveItemHelper.authorizedDelete(SaveItemHelper.java:181)

at org.nrg.xdat.om.base.BaseXnatExperimentdata.delete(BaseXnatExperimentdata.java:588)

at org.nrg.xdat.om.base.BaseXnatSubjectdata.delete(BaseXnatSubjectdata.java:1304)

at org.nrg.xnat.restlet.resources.SecureResource.deleteItem(SecureResource.java:1695)

at org.nrg.xnat.restlet.resources.SubjectResource.handleDelete(SubjectResource.java:402)

at org.restlet.Finder.handle(Finder.java:361)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Router.handle(Router.java:504)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Router.handle(Router.java:504)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Filter.doHandle(Filter.java:150)

at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:130)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at com.noelios.restlet.ChainHelper.handle(ChainHelper.java:124)

at com.noelios.restlet.application.ApplicationHelper.handle(ApplicationHelper.java:112)

at org.restlet.Application.handle(Application.java:341)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Router.handle(Router.java:504)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Router.handle(Router.java:504)

at org.restlet.Filter.doHandle(Filter.java:150)

at com.noelios.restlet.StatusFilter.doHandle(StatusFilter.java:130)

at org.restlet.Filter.handle(Filter.java:195)

at org.restlet.Filter.doHandle(Filter.java:150)

at org.restlet.Filter.handle(Filter.java:195)

at com.noelios.restlet.ChainHelper.handle(ChainHelper.java:124)

at org.restlet.Component.handle(Component.java:673)

at org.restlet.Server.handle(Server.java:331)

at com.noelios.restlet.ServerHelper.handle(ServerHelper.java:68)

at com.noelios.restlet.http.HttpServerHelper.handle(HttpServerHelper.java:147)

at com.noelios.restlet.ext.servlet.ServerServlet.service(ServerServlet.java:881)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.nrg.xnat.security.XnatInitCheckFilter.doFilterInternal(XnatInitCheckFilter.java:41)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.nrg.xnat.security.XnatExpiredPasswordFilter.doFilterInternal(XnatExpiredPasswordFilter.java:111)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.nrg.xnat.security.XnatBasicAuthenticationFilter.doFilterInternal(XnatBasicAuthenticationFilter.java:115)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:155)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:157)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1735)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)

at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:750)

[Herrick, Rick]

I think the issue is that one or more scans in one or more of the sessions associated with the subject has had their XSI type changed. When that happens, XNAT tries to get a canonical version of the scan object from the database but can’t because there’s no row in the database for what XNAT thinks is the scan’s data type. For example, suppose you start with a secondary capture scan. Later, something changes the data-type mapping for that scan (which is stored in the extension column of the xnat_imagescandata table). XNAT tries to find the attributes for the scan in the xnat_mrscandata table but can’t find it because that row is in the xnat_scscandata table.

 

We just recently saw this error in another deployment but haven’t been able to figure out how the data-type mapping is getting changed after the instance is created. It’s not related to docker-compose because that deployment is running on a stand-alone server.

 

The good thing is that this is fairly easy to fix as long as you can access the database. I’ve attached an SQL script that you can run to find any broken data-type mappings. You can run that like this:

 

psql --username=dbUser --host=dbHost --no-align --tuples-only --file=misplaced_scans.sql

 

If that returns any results, you can then use the output to fix the issue in place:

 

psql --username=dbUser --host=dbHost --no-align --tuples-only --file=misplaced_scans.sql > cure.sql

psql --username=dbUser --host=dbHost --no-align --tuples-only --file=cure.sql

 

Since you’re using docker-compose, you’ll need to run this directly on the PostgreSQL container (the name will be xnat-docker-compose_xnat-db_1 or something like that but I’m just using xnat-db):

 

docker cp misplaced_scans.sql xnat-db:.

docker exec --interactive --tty xnat-db bash

 

Once there you can run the query directly with:

 

psql --username=xnat --dbname=xnat --file=misplaced_scans.sql

 

Then capture any results and run that as the cure script.

 

Please let me know if this fixes your issue and/or if you have any idea what kinds of operations people may have been using when working with your image sessions. I’d love to figure out how this is happening so we can prevent it from happening in the first place!

 

-- 

Rick Herrick

XNAT Architect/Developer

Computational Imaging Laboratory

Washington University School of Medicine

 

 

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of akluiber <al...@kluiber.net>
Date: Friday, February 4, 2022 at 3:21 PM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: [XNAT Discussion] Unable to delete subject/session in certain circumstance

* External Email - Caution *

--
You received this message because you are subscribed to the Google Groups "xnat_discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to xnat_discussi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/25f608f9-c7a6-44ee-9777-a087d1287e01n%40googlegroups.com.

 

---

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

[akluiber]

Thanks for the SQL. I can confirm that it worked and let me delete the session and subject afterwards.

After more testing it seems to be related to running a container which makes REST calls via curl to modify the "notes" and "quality" fields and for the session and scans within (xnat:experimentdata/note, xnat:ctScanData/note, and xnat:ctScanData/quality)

Basically, implements the container runs through the for entire session, evaluates adherence to an imaging protocol, and puts back some summary information at the session and scan levels.

Here are the specific commands which modify the fields mentioned:

quality:

curl -s -u $XNAT_USER:$XNAT_PASS -X PUT $XNAT_HOST/data/projects/$project/subjects/$subject/experiments/$session/scans/$id\?xnat:ctScanData/quality\="unusable"

scan notes:

encoded=$(echo "$message" | python3 -c "import urllib.parse, sys; print(urllib.parse.quote(sys.stdin.read()))")
curl -s -u $XNAT_USER:$XNAT_PASS -X PUT $XNAT_HOST/data/projects/$project/subjects/$subject/experiments/$session/scans/$id\?xnat:ctScanData/note\="$encoded"

session notes:

encoded=$(echo "$message" | python3 -c "import urllib.parse, sys; print(urllib.parse.quote(sys.stdin.read()))")
curl -s -u $XNAT_USER:$XNAT_PASS -X PUT $XNAT_HOST/data/projects/$project/subjects/$subject/experiments/$session\?xnat:experimentdata/note\="$encoded"

I've run through a few scenarios. Here are the results:

1) run the container but don't modify anything - delete success

2) modify quality, session notes and scan notes with url encoded message - delete fail

2) modify quality, session notes and scan notes with "test" - delete fail

3) modify only quality - delete fail

4) modify only scan and session notes - delete fail

5) modify only scan notes - delete fail

6) modify only session notes - delete success

So it seems to be related to the scan level fields that are being modified. Perhaps I'm just editing these fields in a way that breaks them?

Something I did find while developing this script. $message can't be multiline. For some reason if I subsequently re-run the same container, there will be unescaped newlines (\n vs \\n) will be returned in the json in my other curl queries, which seems breaks jq's ability to parse them.

i.e.: curl -s -u $XNAT_USER:$XNAT_PASS $XNAT_HOST/data/projects/$project/subjects/$subject/experiments/$session/scans/$id\?format\=json | jq -r '.items[].data_fields.UID'

The only other customizations I have going on in this xnat-docker-compose instance are:

- some plugins (container service, batch launch, ohif)

- some site-wide series import blacklist filters

- site-wide session routing rule: (0012,0050):(.*) 

I hope this is helpful.

- Alex

[Moore, Charlie]

Hi Alex,

Were all of the scans you modified originally of exactly xsitype "xnat:ctScanData"? By referencing quality and note as "xnat:ctScanData/quality" and "xnat:ctScanData/note", it's causing XNAT to change the scan's xsitype from whatever they were originally to "xnat:ctScanData". That's not really supported, which is causing the issue Rick's talking about. I've added a ticket for us to block updates that would change a scan's xsi type here (https://issues.xnat.org/browse/XNAT-7014), but in the meantime you can avoid the issue by using the scan's xsi type in the property names.

Thanks,

Charlie

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of akluiber <al...@kluiber.net>

Sent: Friday, February 4, 2022 6:24 PM
To: xnat_discussion <xnat_di...@googlegroups.com>
Subject: Re: [XNAT Discussion] Unable to delete subject/session in certain circumstance

 

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/819f6725-c89e-4853-9e38-e85d85fdf0den%40googlegroups.com.

[akluiber]

Hmmm. That may have been it. Removing a Dose Report series in my example (xsi:type="xnat:scScanData") and re-trying seems to allow me to delete the session and subject. Is there a more appropriate, type agnostic way to set that field? Otherwise I can just extract the type in my script I'm sure.

[Moore, Charlie]

As far as I'm aware, the answer is "not really". You can skip the xsi type when setting the property by just setting "?note=NOTE", but XNAT wants an xsiType somewhere​, so it would require you to submit 2 query parameters like this: "?note=NOTE&xsiType=xnat:ctScanData". That may look slightly nicer (and you can also set other scan properties in the same call without duplicating the xsi type), but all it really does it move around where you need to put the xsi type. As far as I'm aware, there's no way to get around that.

Thanks,

Charlie

---

From: xnat_di...@googlegroups.com <xnat_di...@googlegroups.com> on behalf of akluiber <al...@kluiber.net>

Sent: Monday, February 7, 2022 11:04 AM

To view this discussion on the web visit https://groups.google.com/d/msgid/xnat_discussion/93e8d04f-552b-4e46-9a43-c6bcdf7cefbcn%40googlegroups.com.