problem with Notifier pipeline, no email, other pipelines marked as failed

[goo...@merctech.com]

I'm running XNAT 1.6.2.1 and seem to be having a problem with the Notifier pipeline.

Other pipelines (anonymization, DICOM-to-NiFTI) launched during the pre-archive
processing seem to work, but the notification email step fails. This causes
the parent pipeline to be marked as failed.

For example, the Application log shows this apparent failure of the DicomToNifti pipeline:

2014-02-06 15:33:45,198 [Thread-49] ERROR org.nrg.pipeline.ProcessLauncher - Couldnt launch /gpfs/lab/labsfw/external/xnat/1.6.2.1/xnat/pipeline/bin/schedule /gpfs/lab/labsfw/external/xnat/1.6.2.1/xnat/pipeline/bin/XnatPipelineLauncher -pipeline /lab/labsfw/external/xnat/1.6.2.1/xnat/pipeline/catalog/mricron/DicomToNifti.xml -id CBICA_E00023 -host http://center-pacs.example.com:8080/xnat -u a7ad3c6d-ce6d-46fb-ad80-84dd0bfe2719 -pwd 1391718814692 -dataType xnat:mrSessionData -label ASP221 -supressNotification -project "MeningiomaVol" -parameterFile /lab/labsfw/external/xnat/build/MeningiomaVol/20140206_153334/archive_trigger/20140206153334622/ASP221/ASP221_DicomToNifti_params_20140206153334622.xml -notify lab-...@example.com -notify lab-...@example.com -parameter mailhost='smtp.example.com' -parameter userfullname='A.Admin' -parameter builddir='/lab/labsfw/external/xnat/build/MeningiomaVol/20140206_153334/archive_trigger/20140206153334622' -parameter xnatserver='CBICA' -p
arameter adminemail='lab-...@example.com' -parameter useremail='lab-...@example.com' -workFlowPrimaryKey 466


When I examine the
/lab/labsfw/external/xnat/build/MeningiomaVol/20140206_153334/archive_trigger/20140206153334622/ASP221
directory, I see that the RAW and NIFTI directories exist, and the
dmc2nii step of the pipeline was successful.

Manually removing those directories & re-running the pipeline as the
'xnat' user from the command-line produces the NiFTI conversion, and
then the error:

-----------------------------------------------------------------
org.nrg.pipeline.exception.PipelineEngineException: Execution of /lab/labsfw/external/xnat/1.6.2.1/xnat/pipeline/catalog/notifications/Notifier -user a7ad3c6d-ce6d-46fb-ad80-84dd0bfe2719 -password 1391718814692 -from lab-admin\@example.com -to lab-admin\@example.com -cc lab-admin\@example.com -subject CBICA update: NIFTI files generated for ASP221 -body Dear A.Admin,\<br\> \<p\> NIFTI files have been generated for ASP221 . Details of the session are available \<a href=\"http://center-pacs.example.com:8080/xnat//app/action/DisplayItemAction/search_element/xnat:mrSessionData/search_field/xnat:mrSessionData.ID/search_value/CBICA_E00023\"\> here. \</a\> \</p\>\<br\> \</p\>\<br\>XNAT Team. -host smtp.example.com at Step[id= END-Notify:] was unsuccesfull Personnel notification email could not be sent class java.lang.ArrayStoreExceptionnull null
at org.nrg.pipeline.manager.ExecutionManager.execute(ExecutionManager.java:121)
at org.nrg.pipeline.task.StepManager.execute(StepManager.java:166)
at org.nrg.pipeline.manager.PipelineManager.launchPipeline(PipelineManager.java:213)
at org.nrg.pipeline.manager.PipelineManager.launchPipeline(PipelineManager.java:193)
at org.nrg.pipeline.client.XNATPipelineLauncher.launch(XNATPipelineLauncher.java:78)
at org.nrg.pipeline.client.XNATPipelineLauncher.run(XNATPipelineLauncher.java:290)
at org.nrg.pipeline.client.XNATPipelineLauncher.main(XNATPipelineLauncher.java:255)
org.nrg.pipeline.exception.PipelineEngineException: Unable to complete pipeline Execution of /lab/labsfw/external/xnat/1.6.2.1/xnat/pipeline/catalog/notifications/Notifier -user a7ad3c6d-ce6d-46fb-ad80-84dd0bfe2719 -password 1391718814692 -from lab-admin\@example.com -to lab-admin\@example.com -cc lab-admin\@example.com -subject CBICA update: NIFTI files generated for ASP221 -body Dear A.Admin,\<br\> \<p\> NIFTI files have been generated for ASP221 . Details of the session are available \<a href=\"http://center-pacs.example.com:8080/xnat//app/action/DisplayItemAction/search_element/xnat:mrSessionData/search_field/xnat:mrSessionData.ID/search_value/CBICA_E00023\"\> here. \</a\> \</p\>\<br\> \</p\>\<br\>XNAT Team. -host smtp.example.com at Step[id= END-Notify:] was unsuccesfull Personnel notification email could not be sent class java.lang.ArrayStoreExceptionnull null
at org.nrg.pipeline.manager.PipelineManager.launchPipeline(PipelineManager.java:226)
at org.nrg.pipeline.manager.PipelineManager.launchPipeline(PipelineManager.java:193)
at org.nrg.pipeline.client.XNATPipelineLauncher.launch(XNATPipelineLauncher.java:78)
at org.nrg.pipeline.client.XNATPipelineLauncher.run(XNATPipelineLauncher.java:290)
at org.nrg.pipeline.client.XNATPipelineLauncher.main(XNATPipelineLauncher.java:255)
Unable to launch pipeline /lab/labsfw/external/xnat/1.6.2.1/xnat/pipeline/catalog/mricron/DicomToNifti.xml
sh -c tail -n40 /lab/labsfw/external/xnat/build/MeningiomaVol/20140206_153334/archive_trigger/20140206153334622/ASP221/DicomToNifti.err
sh -c tail -n40 /lab/labsfw/external/xnat/build/MeningiomaVol/20140206_153334/archive_trigger/20140206153334622/ASP221/DicomToNifti.log
Couldnt send email msg
-----------------------------------------------------------------



Notice that the error states:
Execution of /lab/labsfw/external/xnat/1.6.2.1/xnat/pipeline/catalog/notifications/Notifier
but there is no file of that name. There is "Notifier.xml" in that directory.

When I check the installed pipelines via the web interface, I don't see a
"Notifier" pipeline.

If I attempt to add that pipeline to the repository, and supply the full path
to the Notifier.xml descriptor, I get the message "Unknown Error." in the
"ManagePipeline" browser window.

The "xdat.log" file shows the following:

-----------------------------------------------------------------
2014-02-06 16:45:30,306 [http-8080-4] ERROR org.nrg.xnat.turbine.modules.actions.ManagePipeline -
java.lang.ClassCastException: org.nrg.pipeline.xmlbeans.impl.ResourceDocumentImpl cannot be cast to org.nrg.pipeline.xmlbeans.PipelineDocument
at org.nrg.pipeline.PipelineRepositoryManager.SetInfo(PipelineRepositoryManager.java:76)
at org.nrg.xnat.turbine.modules.actions.ManagePipeline.doAdd(ManagePipeline.java:322)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at org.apache.turbine.util.velocity.VelocityActionEvent.executeEvents(VelocityActionEvent.java:136)
at org.apache.turbine.util.velocity.VelocityActionEvent.perform(VelocityActionEvent.java:78)
at org.apache.turbine.modules.actions.VelocityAction.perform(VelocityAction.java:72)
at org.apache.turbine.modules.actions.VelocitySecureAction.perform(VelocitySecureAction.java:61)
at org.apache.turbine.modules.ActionLoader.exec(ActionLoader.java:96)
at org.apache.turbine.modules.pages.DefaultPage.doBuild(DefaultPage.java:113)
at org.apache.turbine.modules.Page.build(Page.java:53)
at org.apache.turbine.modules.PageLoader.exec(PageLoader.java:98)
at org.apache.turbine.Turbine.doGet(Turbine.java:751)
at org.apache.turbine.Turbine.doPost(Turbine.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.nrg.xnat.restlet.util.UpdateExpirationCookie.doFilter(UpdateExpirationCookie.java:38)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:100)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:78)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.nrg.xnat.security.XnatArcSpecFilter.doFilter(XnatArcSpecFilter.java:40)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:35)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.nrg.xnat.security.XnatBasicAuthenticationFilter.doFilter(XnatBasicAuthenticationFilter.java:134)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:187)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:109)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.nrg.xnat.security.XnatExpiredPasswordFilter.doFilter(XnatExpiredPasswordFilter.java:79)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:701)
-----------------------------------------------------------------


Any suggestions?

Thanks,

Mark