Additions to abstractDemographicData in xnat.xsd

[Santiago Timón]

Hi everyone,

We are trying to add to our subjects some demographic variables we need. We have read some approaches in the group and currently we are trying to directly add the fields to abstractDemographicData.

This is what it looks like (custom variables are at the end and in red):

<xs:complexContent>

<xs:extension base="xnat:abstractDemographicData">

<xs:sequence>

<xs:choice minOccurs="0">

<xs:element name="dob" type="xs:date"/>

<xs:element name="yob">

<xs:simpleType>

<xs:restriction base="xs:integer"/>

</xs:simpleType>

</xs:element>

<xs:element name="age" type="xs:integer"/>

</xs:choice>

<xs:element name="gender" minOccurs="0">

<xs:simpleType>

<xs:restriction base="xs:string">

<xs:enumeration value="male"/>

<xs:enumeration value="female"/>

<xs:enumeration value="other"/>

<xs:enumeration value="unknown"/>

<xs:enumeration value="M"/>

<xs:enumeration value="F"/>

</xs:restriction>

</xs:simpleType>

</xs:element>

<xs:element name="handedness" minOccurs="0">

<xs:simpleType>

<xs:restriction base="xs:string">

<xs:enumeration value="left"/>

<xs:enumeration value="right"/>

<xs:enumeration value="ambidextrous"/>

<xs:enumeration value="unknown"/>

</xs:restriction>

</xs:simpleType>

</xs:element>

<xs:element name="ses" minOccurs="0">

<xs:annotation>

<xs:documentation>Socioeconomic status</xs:documentation>

</xs:annotation>

<xs:simpleType>

<xs:restriction base="xs:integer"/>

</xs:simpleType>

</xs:element>

                    <xs:element name="employment" minOccurs="0">

                        <xs:annotation>

                            <xs:documentation>

                                Employment status:

                                0: Employed

                                1: Unemployed

                                2: Retired

                                3: Unknown or N/A

                            </xs:documentation>

                        </xs:annotation>

                        <xs:simpleType>

                            <xs:restriction base="xs:integer">

                                <xs:minInclusive value="0"/>

                                <xs:maxInclusive value="3"/>

                            </xs:restriction>

                        </xs:simpleType>

                    </xs:element>

<xs:element name="education" minOccurs="0">

<xs:simpleType>

<xs:restriction base="xs:integer">

<xs:minInclusive value="0"/>

<xs:maxInclusive value="30"/>

</xs:restriction>

</xs:simpleType>

</xs:element>

<xs:element name="educationDesc" type="xs:string" minOccurs="0"/>

<xs:element name="race" type="xs:string" minOccurs="0"/>

<xs:element name="race2" type="xs:string" minOccurs="0"/>

<xs:element name="race3" type="xs:string" minOccurs="0"/>

<xs:element name="race4" type="xs:string" minOccurs="0"/>

<xs:element name="race5" type="xs:string" minOccurs="0"/>

<xs:element name="race6" type="xs:string" minOccurs="0"/>

<xs:element name="ethnicity" type="xs:string" minOccurs="0"/>

<xs:element name="weight" minOccurs="0">

<xs:complexType>

<xs:simpleContent>

<xs:extension base="xs:float">

<xs:attribute name="units" type="xs:string"/>

</xs:extension>

</xs:simpleContent>

</xs:complexType>

</xs:element>

<xs:element name="height" minOccurs="0">

<xs:complexType>

<xs:simpleContent>

<xs:extension base="xs:float">

<xs:attribute name="units" type="xs:string"/>

</xs:extension>

</xs:simpleContent>

</xs:complexType>

</xs:element>

<xs:element name="gestational_age" type="xs:float" minOccurs="0"/>

<xs:element name="post_menstrual_age" type="xs:float" minOccurs="0"/>

<xs:element name="birth_weight" type="xs:float" minOccurs="0"/>

<!-- custom variables -->

<xs:element name="subj_group" type="xs:integer"></xs:element>

<xs:element name="nation" type="xs:integer"></xs:element>

                    <xs:element name="nation_other" type="xs:string" maxOccurs="1" minOccurs="0"></xs:element>

                    <xs:element name="language" type="xs:integer"></xs:element>

                    <xs:element name="language_other" type="xs:string" maxOccurs="1" minOccurs="0"></xs:element>

                    <xs:element name="recruit" type="xs:integer"></xs:element>

                    <xs:element name="recruit_other" type="xs:string" maxOccurs="1" minOccurs="0"></xs:element>

</xs:sequence>

</xs:extension>

</xs:complexContent>

Database rebuild and deploy doesn't seem to throw any errors, but when I try to create a new subject, we get turbine and velocity errors, this seems to be the source:

2015-05-20 01:35:56,388 [http-bio-8080-exec-2] ERROR org.apache.turbine.services.velocity.TurbineVelocityService - Class org.nrg.xdat.om.XnatSubjectdata.getFieldDefinitionGroups threw Exception

java.lang.NullPointerException

at java.util.Hashtable.hash(Hashtable.java:239)

at java.util.Hashtable.put(Hashtable.java:519)

at org.nrg.xdat.om.base.BaseXnatSubjectdata.getFieldDefinitionGroups(BaseXnatSubjectdata.java:1229)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:260)

at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:207)

at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:357)

at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:135)

at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:271)

at org.apache.velocity.runtime.directive.Parse.render(Parse.java:232)

at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:153)

at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:271)

at org.apache.velocity.runtime.directive.Parse.render(Parse.java:232)

at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:153)

at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:271)

at org.apache.velocity.Template.merge(Template.java:296)

at org.apache.velocity.app.Velocity.mergeTemplate(Velocity.java:492)

at org.apache.velocity.app.Velocity.mergeTemplate(Velocity.java:461)

at org.apache.turbine.services.velocity.TurbineVelocityService.executeRequest(TurbineVelocityService.java:415)

at org.apache.turbine.services.velocity.TurbineVelocityService.handleRequest(TurbineVelocityService.java:281)

at org.apache.turbine.services.velocity.TurbineVelocity.handleRequest(TurbineVelocity.java:71)

at org.apache.turbine.modules.screens.VelocityScreen.buildTemplate(VelocityScreen.java:119)

at org.apache.turbine.modules.screens.TemplateScreen.doBuild(TemplateScreen.java:101)

at org.apache.turbine.modules.Screen.build(Screen.java:57)

at org.apache.turbine.modules.ScreenLoader.eval(ScreenLoader.java:104)

at org.apache.turbine.modules.layouts.VelocityOnlyLayout.doBuild(VelocityOnlyLayout.java:92)

at org.apache.turbine.modules.Layout.build(Layout.java:53)

at org.apache.turbine.modules.LayoutLoader.exec(LayoutLoader.java:98)

at org.apache.turbine.modules.pages.DefaultPage.doBuild(DefaultPage.java:146)

at org.apache.turbine.modules.Page.build(Page.java:53)

at org.apache.turbine.modules.PageLoader.exec(PageLoader.java:98)

at org.apache.turbine.Turbine.doGet(Turbine.java:751)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:620)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:163)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)

at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.nrg.xnat.restlet.util.UpdateExpirationCookie.doFilter(UpdateExpirationCookie.java:38)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)

at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.nrg.xnat.security.XnatArcSpecFilter.doFilter(XnatArcSpecFilter.java:40)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.nrg.xnat.security.XnatBasicAuthenticationFilter.doFilter(XnatBasicAuthenticationFilter.java:134)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:109)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.nrg.xnat.security.XnatExpiredPasswordFilter.doFilter(XnatExpiredPasswordFilter.java:177)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109)

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)

at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Are we missing something in our additions? 

I know this demographics hacking is tricky, thanks for your help.

Best,

Santi

[Santiago Timón]

All right, looking into the files it seems that the exception is thrown when no Project is selected, because it tries to do the following:

projects.put(this.getPrimaryProject(false), "");

And the hashtable doesn't allow null values. 

However, even with a selected project, the edit screen fails to load the context, this is the velocity.log:

2015-05-20 02:12:54,110 [http-bio-8080-exec-1] ERROR velocity - RHS of #set statement is null. Context will not be modified. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 112, column 29]

2015-05-20 02:12:54,110 [http-bio-8080-exec-1] ERROR velocity - RHS of #set statement is null. Context will not be modified. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 185, column 9]

2015-05-20 02:12:54,110 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_gender) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 187, column 41]

2015-05-20 02:12:54,110 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_gender) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 188, column 25]

2015-05-20 02:12:54,110 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_gender) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 188, column 49]

2015-05-20 02:12:54,110 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_gender) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 192, column 53]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_gender) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 193, column 57]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_gender) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 195, column 55]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - RHS of #set statement is null. Context will not be modified. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 199, column 5]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 202, column 37]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 203, column 47]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 204, column 45]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 205, column 61]

2015-05-20 02:12:54,111 [http-bio-8080-exec-1] ERROR velocity - Right side ($temp_hand) of '==' operation has null value. If a reference, it may not be in the context. Operation not possible. /screens/xnat_subjectData/xnat_subjectData_edit_core_demographics.vm [line 206, column 51]

2015-05-20 02:12:54,236 [http-bio-8080-exec-1] ERROR velocity - Left side ($om.getItem().getGenericSchemaElement().getPrimaryElements().indexOf("xnat:experimentData")) of '!=' operation has null value. Operation not possible. navigations//Breadcrumb.vm [line 177, column 102]

So these kind of set operations fail: #set($temp_gender=$!item.getProperty("xnat:subjectData/demographics[@xsi:type=xnat:demographicData]/gender"))

And I don't understand, because for a new subject is logical that they are null. 

I assume it is something we have done wrong, but still don't know what it could be. 

Thanks,

Santi

[Santiago Timón]

Well, it seems that these errors doesn't actually affect the submit process and it happens even in a fresh installation without any modifications. I can create new subjects with our new demographic data.

However we are experiencing very weird errors. Subject finder screen fails to load the subject data, but only for one experiment. Specifically in line 29:

 #set($part=$om.getSubjectData())

The log entry:

2015-05-20 07:07:37,279 [http-bio-8080-exec-10] ERROR velocity - RHS of #set statement is null. Context will not be modified. /screens/SubjectFinder.vm [line 29, column 1]

What can be preventing the screen to load the om variable or the subject data for that item? I have various custom datatypes experiments, and only happens in one of them. 

Any help is greatly appreciated, it is very difficult to debug these kind of errors.

Thanks, 

Santi