Ontoserver 5 Availability

[Preston Lee]

Folks,

The Ontoserver 5 instance is available at https://ontoserver.hspconsortium.org/fhir, with some outstanding issues I’d love to resolve. If you run the Postman examples linked to from https://ontoserver.csiro.au, many of them seem to work. The syndication credentialing issue seems to have resolved so we’re using an HSPC-specific client_id now, and the system seems stable though low on RAM. Here are some of the issues:

1. AFAICT, the syndication URL (https://api.healthterminologies.gov.au/syndication/v1/syndication.xml) *only* has Australian content packages, so SNOMED AU is the only content loaded and indexed, at the moment. I don’t see the SCT international release, nor anything other than AU-specific entries. I’m guessing this is what’s used to generate “availableCodeSystemIndexes” via the API? (http://ontoserver.hspconsortium.org/api/availableCodeSystemIndexes) How is content syndication intended to work for non-aussies? Is it a manual combination of /api/sct/addRf2 and /api/indexCodeSystem as others have mentioned? ..or are these packages distributed upstream? 
2. snomed.default.edition doesn’t seem to do anything? When set to the international edition I’m not sure what the functional difference is.
3. There is no authentication and authorization. I’ll need some help configuring this to tie it into the HSPC ID system for managed write access. There are a number of configuration options, and I think what we want is a default “read” role for all users (both authenticated and unauthenticated), and a “write” role somehow tied to a membership scope.
4. I can’t figure out the “correct” way of getting rid of the “/fhir” path in the URL. I’ve tried setting the “ontoserver.fhir.base” property on the server as well as getting rid of it in the proxy (just Apache httpd) in front of it, but the application doesn’t seem happy with any of that and remains confused across restarts. Smells like a Tomcat issue.

Thoughts?

Preston

[prest...@prestonlee.com]

It looks like the issue with /api/sct/addRf2 is the underlying database volume size spiking considerably, and when there's an issue, the app's database connection terminates and reports the red herring "There was a problem reading an RF2 file."

I'll keep playing with it a bit to get SCT international running.

Preston

[prest...@prestonlee.com]

I have emails out to various people on a number of issues, but for search engine and posterity purposes think we should try to have these conversations online on this list, and will try to direct people here instead of emailing me directly.

On the below issue, loading of a CodeSystem index consumes a significant amount of disk space, IIRC between 10-20GiB. I had to resize the underlying volume to accommodate this, and according to Jeff Chung the index seems to match what he'd expect from other instances with AU content.

The current technical blocker is an unchecked exception thrown during the index API call corresponding to the SCT International release. AFAICT from tailing the logs and watching resource usage, it appears that the indexer is run within a single thread (likely just the web app's handler from the main pool) within the main application process that loads raw records/rows into RAM (~15GiB heap needed), does whatever it algorithmically needs to do to extract indexable content, proceeds to cram stuff into an embedded Lucene index (from a stack trace), encounters something null that shouldn't be, vomits the runtime exception that unwinds the importer thread that in turns causes the API layer to return an error to the client blocking on the API call. Below is the entire stack trace, in case anyone sees this in the future. I've emailed Michael Lawley already about it.

Preston



2017-09-20 21:41:54.471Z INFO 1 --- [nio-8080-exec-1] o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] : Initializing Spring FrameworkServlet 'dispatcherServlet'
2017-09-20 21:41:54.567Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.atom.AtomManager : Getting feed from https://api.healthterminologies.gov.au/syndication/v1/syndication.xml,https://ontoserver.csiro.au/synd/syndication.xml
2017-09-20 21:42:01.427Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer : Indexing module Module [name=SNOMED CT core module (core metadata concept), effectiveTime=2017-07-31T00:00:00.000]
2017-09-20 21:42:01.427Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer : Getting rows for module
2017-09-20 21:42:01.428Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Getting rows for module
2017-09-20 21:42:05.493Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 443237 concept rows.
2017-09-20 21:42:43.348Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 2628056 relationship rows.
2017-09-20 21:43:04.706Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 1342502 description rows.
2017-09-20 21:43:04.774Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved the following concrete domains refsets: []
2017-09-20 21:43:04.774Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 0 concrete domains rows.
2017-09-20 21:43:04.840Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved the following refset descriptors refsets: [900000000000456007]
2017-09-20 21:43:04.840Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 142 refset descriptors rows.
2017-09-20 21:43:34.710Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved the following language refsets: [900000000000509007, 900000000000508004]
2017-09-20 21:43:34.711Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 2611779 language rows.
2017-09-20 21:43:34.933Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved the following simple refsets: [723264001]
2017-09-20 21:43:34.934Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 19688 simple rows.
2017-09-20 21:43:48.465Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved the following generic refsets: [900000000000527005, 900000000000525002, 900000000000490003, 900000000000524003, 734139008, 900000000000530003, 900000000000531004, 734138000, 900000000000489007, 900000000000523009, 723561005, 723563008, 446608001, 900000000000538005, 723560006, 900000000000497000, 723562003, 900000000000526001, 900000000000528000]
2017-09-20 21:43:48.465Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Retrieved 1060449 generic rows.
2017-09-20 21:43:48.465Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.importer.snomed.rf2.RowDataFactory : Finished getting rows for module in 107037ms
2017-09-20 21:43:48.465Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer : Finished getting rows for module in 107037ms
2017-09-20 21:43:48.471Z INFO 1 --- [nio-8080-exec-1] au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer : Indexing reference sets.
2017-09-20 21:44:44.281Z ERROR 1 --- [nio-8080-exec-1] au.csiro.ontoserver.web.RestResponseEntityExceptionHandler : Exception: [347c95ec-2960-44a8-adfb-7083a2e2d2d2]: There was a problem writing the index for module Module [name=SNOMED CT core module (core metadata concept), effectiveTime=2017-07-31T00:00:00.000]


au.csiro.ontoserver.exceptions.IndexingException: [347c95ec-2960-44a8-adfb-7083a2e2d2d2]: There was a problem writing the index for module Module [name=SNOMED CT core module (core metadata concept), effectiveTime=2017-07-31T00:00:00.000]
at au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer.index(Rf2Indexer.java:484)
at au.csiro.ontoserver.OntoserverApi.indexSnomedFromSource(OntoserverApi.java:516)
at au.csiro.ontoserver.OntoserverApi.indexCodeSystem(OntoserverApi.java:257)
at au.csiro.ontoserver.web.OntoserverController.indexCodeSystem(OntoserverController.java:285)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.ebaysf.web.cors.CORSFilter.handleNonCORS(CORSFilter.java:437)
at org.ebaysf.web.cors.CORSFilter.doFilter(CORSFilter.java:172)
at au.csiro.ontoserver.security.SimpleCORSFilter.doFilter(SimpleCORSFilter.java:156)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at au.csiro.ontoserver.Application$1.doFilter(Application.java:82)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:478)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:677)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:799)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1455)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: value must not be null
at org.apache.lucene.document.Field.<init>(Field.java:246)
at org.apache.lucene.document.StringField.<init>(StringField.java:61)
at au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer.createRefsetDocument(Rf2Indexer.java:950)
at au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer.lambda$indexGenericReferenceSets$1(Rf2Indexer.java:521)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer.indexGenericReferenceSets(Rf2Indexer.java:518)
at au.csiro.ontoserver.indexer.snomed.rf2.Rf2Indexer.index(Rf2Indexer.java:324)
... 98 common frames omitted