Hapi-fhir-jpaserver 5.0 Unable to access lob stream & Failed to decompress contents

[Caleb Steele-Lane]

Hi,

I was running the hapi-fhir-jpaserver on version 5.0.0 with postgres 9.5. After entering large amounts of data, I am getting some errors when searching for some data types (SearviceRequets, Organization).

When I search for ServiceRequest I get:

{

"resourceType": "OperationOutcome",

"text": {

"status": "generated",

"div": "<div xmlns=\"http://www.w3.org/1999/xhtml\"><h1>Operation Outcome</h1><table border=\"0\"><tr><td style=\"font-weight: bold;\">ERROR</td><td>[]</td><td><pre>Unable to access lob stream; nested exception is org.hibernate.HibernateException: Unable to access lob stream</pre></td>\n\t\t\t\t\t\n\t\t\t\t\n\t\t\t</tr>\n\t\t</table>\n\t</div>"

},

"issue": [ {

"severity": "error",

"code": "processing",

"diagnostics": "Unable to access lob stream; nested exception is org.hibernate.HibernateException: Unable to access lob stream"

} ]

}

Looking at the logs for the server the full stack trace is:

2021-04-28 01:08:11.669 [https-jsse-nio-8443-exec-10] ERROR c.u.f.r.s.i.ExceptionHandlingInterceptor [ExceptionHandlingInterceptor.java:160] Failure during REST processing: org.springframework.orm.jpa.JpaSystemException: Unable to access lob stream; nested exception is org.hibernate.HibernateException: Unable to access lob stream
org.springframework.orm.jpa.JpaSystemException: Unable to access lob stream; nested exception is org.hibernate.HibernateException: Unable to access lob stream
    at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:331)
    at ca.uhn.fhir.jpa.config.HapiFhirHibernateJpaDialect.convertHibernateAccessException(HapiFhirHibernateJpaDialect.java:120)
    at ca.uhn.fhir.jpa.config.HapiFhirHibernateJpaDialect.convertHibernateAccessException(HapiFhirHibernateJpaDialect.java:63)
    at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233)
    at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551)
    at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61)
    at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:242)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
    at com.sun.proxy.$Proxy247.findByResourceIds(Unknown Source)
    at ca.uhn.fhir.jpa.search.builder.SearchBuilder.doLoadPids(SearchBuilder.java:624)
    at ca.uhn.fhir.jpa.search.builder.SearchBuilder.lambda$loadResourcesByPid$3(SearchBuilder.java:743)
    at ca.uhn.fhir.jpa.util.QueryChunker.chunk(QueryChunker.java:46)
    at ca.uhn.fhir.jpa.util.QueryChunker.chunk(QueryChunker.java:38)
    at ca.uhn.fhir.jpa.search.builder.SearchBuilder.loadResourcesByPid(SearchBuilder.java:743)
    at ca.uhn.fhir.jpa.search.PersistedJpaBundleProvider.toResourceList(PersistedJpaBundleProvider.java:377)
    at ca.uhn.fhir.jpa.search.PersistedJpaSearchFirstPageBundleProvider.lambda$getResources$0(PersistedJpaSearchFirstPageBundleProvider.java:74)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
    at ca.uhn.fhir.jpa.search.PersistedJpaSearchFirstPageBundleProvider.getResources(PersistedJpaSearchFirstPageBundleProvider.java:74)
    at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.createBundleFromBundleProvider(BaseResourceReturningMethodBinding.java:166)
    at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.doInvokeServer(BaseResourceReturningMethodBinding.java:335)
    at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.invokeServer(BaseResourceReturningMethodBinding.java:380)
    at ca.uhn.fhir.rest.server.RestfulServer.handleRequest(RestfulServer.java:1084)
    at ca.uhn.fhir.rest.server.RestfulServer.doGet(RestfulServer.java:380)
    at ca.uhn.fhir.rest.server.RestfulServer.service(RestfulServer.java:1794)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.hibernate.HibernateException: Unable to access lob stream
    at org.hibernate.type.descriptor.java.PrimitiveByteArrayTypeDescriptor.wrap(PrimitiveByteArrayTypeDescriptor.java:123)
    at org.hibernate.type.descriptor.java.PrimitiveByteArrayTypeDescriptor.wrap(PrimitiveByteArrayTypeDescriptor.java:26)
    at org.hibernate.type.descriptor.sql.BlobTypeDescriptor$1.doExtract(BlobTypeDescriptor.java:48)
    at org.hibernate.type.descriptor.sql.BasicExtractor.extract(BasicExtractor.java:47)
    at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:257)
    at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:253)
    at org.hibernate.type.AbstractStandardBasicType.nullSafeGet(AbstractStandardBasicType.java:243)
    at org.hibernate.type.AbstractStandardBasicType.hydrate(AbstractStandardBasicType.java:329)
    at org.hibernate.persister.entity.AbstractEntityPersister.hydrate(AbstractEntityPersister.java:3135)
    at org.hibernate.loader.Loader.loadFromResultSet(Loader.java:1870)
    at org.hibernate.loader.Loader.hydrateEntityState(Loader.java:1798)
    at org.hibernate.loader.Loader.instanceNotYetLoaded(Loader.java:1771)
    at org.hibernate.loader.Loader.getRow(Loader.java:1623)
    at org.hibernate.loader.Loader.getRowFromResultSet(Loader.java:740)
    at org.hibernate.loader.Loader.getRowsFromResultSet(Loader.java:1039)
    at org.hibernate.loader.Loader.processResultSet(Loader.java:990)
    at org.hibernate.loader.Loader.doQuery(Loader.java:959)
    at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
    at org.hibernate.loader.Loader.doList(Loader.java:2850)
    at org.hibernate.loader.Loader.doList(Loader.java:2832)
    at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2664)
    at org.hibernate.loader.Loader.list(Loader.java:2659)
    at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:506)
    at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:400)
    at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:219)
    at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1414)
    at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1625)
    at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1593)
    at org.hibernate.query.Query.getResultList(Query.java:165)
    at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:126)
    at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:88)
    at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:155)
    at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:143)
    at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137)
    at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121)
    at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152)
    at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:131)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:80)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
    at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
    ... 55 common frames omitted
Caused by: org.postgresql.util.PSQLException: ERROR: large object 26599 does not exist
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2505)
    at org.postgresql.core.v3.QueryExecutorImpl.receiveFastpathResult(QueryExecutorImpl.java:805)
    at org.postgresql.core.v3.QueryExecutorImpl.fastpathCall(QueryExecutorImpl.java:591)
    at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:105)
    at org.postgresql.fastpath.Fastpath.fastpath(Fastpath.java:146)
    at org.postgresql.fastpath.Fastpath.getInteger(Fastpath.java:158)
    at org.postgresql.largeobject.LargeObject.<init>(LargeObject.java:102)
    at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:252)
    at org.postgresql.largeobject.LargeObjectManager.open(LargeObjectManager.java:235)
    at org.postgresql.jdbc.AbstractBlobClob.getLo(AbstractBlobClob.java:269)
    at org.postgresql.jdbc.AbstractBlobClob.getBinaryStream(AbstractBlobClob.java:114)
    at org.hibernate.type.descriptor.java.PrimitiveByteArrayTypeDescriptor.wrap(PrimitiveByteArrayTypeDescriptor.java:120)
    ... 99 common frames omitted

Searching Organization I get:

2021-04-28 01:20:13.032 [https-jsse-nio-8443-exec-1] ERROR c.u.f.r.s.i.ExceptionHandlingInterceptor [ExceptionHandlingInterceptor.java:160] Failure during REST processing: ca.uhn.fhir.parser.DataFormatException: Failed to decompress contents
ca.uhn.fhir.parser.DataFormatException: Failed to decompress contents
    at ca.uhn.fhir.jpa.dao.GZipUtil.decompress(GZipUtil.java:41)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirDao.decodeResource(BaseHapiFhirDao.java:1578)
    at ca.uhn.fhir.jpa.dao.BaseHapiFhirDao.toResource(BaseHapiFhirDao.java:946)
    at ca.uhn.fhir.jpa.search.builder.SearchBuilder.doLoadPids(SearchBuilder.java:656)
    at ca.uhn.fhir.jpa.search.builder.SearchBuilder.lambda$loadResourcesByPid$3(SearchBuilder.java:743)
    at ca.uhn.fhir.jpa.util.QueryChunker.chunk(QueryChunker.java:46)
    at ca.uhn.fhir.jpa.util.QueryChunker.chunk(QueryChunker.java:38)
    at ca.uhn.fhir.jpa.search.builder.SearchBuilder.loadResourcesByPid(SearchBuilder.java:743)
    at ca.uhn.fhir.jpa.search.PersistedJpaBundleProvider.toResourceList(PersistedJpaBundleProvider.java:377)
    at ca.uhn.fhir.jpa.search.PersistedJpaSearchFirstPageBundleProvider.lambda$getResources$0(PersistedJpaSearchFirstPageBundleProvider.java:74)
    at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
    at ca.uhn.fhir.jpa.search.PersistedJpaSearchFirstPageBundleProvider.getResources(PersistedJpaSearchFirstPageBundleProvider.java:74)
    at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.createBundleFromBundleProvider(BaseResourceReturningMethodBinding.java:166)
    at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.doInvokeServer(BaseResourceReturningMethodBinding.java:335)
    at ca.uhn.fhir.rest.server.method.BaseResourceReturningMethodBinding.invokeServer(BaseResourceReturningMethodBinding.java:380)
    at ca.uhn.fhir.rest.server.RestfulServer.handleRequest(RestfulServer.java:1084)
    at ca.uhn.fhir.rest.server.RestfulServer.doGet(RestfulServer.java:380)
    at ca.uhn.fhir.rest.server.RestfulServer.service(RestfulServer.java:1794)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:126)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.access$000(ErrorPageFilter.java:64)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:101)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.springframework.boot.web.servlet.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.zip.ZipException: Not in GZIP format
    at java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:166)
    at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:80)
    at java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:92)
    at ca.uhn.fhir.jpa.dao.GZipUtil.decompress(GZipUtil.java:38)
    ... 50 common frames omitted

I have tried upgrading to the docker image hapiproject/hapi:v5.4.0, but these problems have persisted.

Is this a current problem that would be fixed by going to the latest codebase or are there any configurations I could try to get around this issue?

Any help is much appreciated.

Caleb

[James Agnew]

The error message "ERROR: large object 26599 does not exist" is definitely suspicious. That sounds to me like an issue with the DB itself as opposed to a HAPI issue.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/0154839e-9b14-4e7a-b8dc-27e9617015c6n%40googlegroups.com.