OutOfMemory error when reindexing

[Pete]

I just upgraded to Hapi 8.2 and went to reindex with the body:

{
  "resourceType": "Parameters",
  "parameter": [ {
    "name": "reindexSearchParameters",
    "valueString": "NONE"
  },{
    "name": "batchSize",
    "valueDecimal": 100
  },{
    "name": "optimizeStorage",
    "valueString": "ALL_VERSIONS"
  }]
}

Unfortunately, I keep getting OutOfMemory errors even when lowering my batchSize and increasing my java memory settings. It did work when I removed optimizeStorage, but I'd really like to free up some disk space. Any ideas on what I can try?

[James Agnew]

Hmm, that's interesting.

Do you get a stack trace that sheds any light on where we're running out of memory? Are you in a position to take a heap dump and does that give any indications?

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/06f5f049-ad90-45ad-b229-92fdbcfea43dn%40googlegroups.com.

[Pete]

I get this error stack

[1426.948s][warning][gc,alloc] batch2-work-notification-7: Retried waiting for GCLocker too often allocating 131072 words

ERROR c.uhn.fhir.log.batch_troubleshooting [StepExecutor.java:108] Unexpected failure executing job REINDEX step reindex-start
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:228)
at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:740)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:233)
at org.apache.commons.io.output.StringBuilderWriter.write(StringBuilderWriter.java:155)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1611)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1589)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1384)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1157)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:3153)
at org.apache.commons.io.IOUtils.toString(IOUtils.java:3179)
at ca.uhn.fhir.jpa.dao.GZipUtil.decompress(GZipUtil.java:38)
at ca.uhn.fhir.jpa.dao.BaseHapiFhirDao.decodeResource(BaseHapiFhirDao.java:1750)
at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.reindexOptimizeStorageHistoryEntity(BaseHapiFhirResourceDao.java:1779)
at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.reindexOptimizeStorage(BaseHapiFhirResourceDao.java:1749)
at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.reindex(BaseHapiFhirResourceDao.java:1696)
at jdk.internal.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:138)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720)
at ca.uhn.fhir.jpa.dao.JpaResourceDaoBundle$$SpringCGLIB$$0.reindex(<generated>)
at ca.uhn.fhir.batch2.jobs.reindex.v2.ReindexTaskV2.doInTransaction(ReindexTaskV2.java:195)
at ca.uhn.fhir.batch2.jobs.reindex.v2.ReindexTaskV2.doInTransaction(ReindexTaskV2.java:48)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at ca.uhn.fhir.jpa.dao.tx.HapiTransactionService.doExecuteCallback(HapiTransactionService.java:458)

Bunch of
[hapi-fhir-jpa-scheduler-clustered-3] ERROR c.uhn.fhir.log.batch_troubleshooting [JobInstanceProcessor.java:337] Job Instance f7ecc3af-f49c-461d-ab73-b846cca6a0f6 failed to transition work chunk with id 6ce7451d-8bbe-4a52-8a17-60b4c994e2c1 from READY to QUEUED; found 0, expected 1; skipping work chunk.

[James Agnew]

Wow, that's strange. That is just expanding the compressed resource body, so unless you have some crazy large resources it really shouldn't be occupying too much memory.

What is your -Xmx value set to?

To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/f41b11f3-ffb4-40f5-a060-7d3f996f804bn%40googlegroups.com.

[Pete]

It's possibly a large resource. Is there a way to narrow down which one killed it?

I have XmX set to 4g

[James Agnew]

The following query should give you the ID of any resources that failed to reindex:

> select res_type,fhir_id from hfj_resource where sp_index_status = 2

This does assume that the OOM didn't destabilize the process so badly that it couldn't update the status. Does this query yield any interesting results? Note that this will give you the ID of the resource, but in the point of the code where that OOM occurs we're looping across all versions of it, so it might be an older version that is causing the issue.

Cheers,

James

To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/33ea0cb7-0626-4af1-a636-a7afb29d2ed1n%40googlegroups.com.

[Pete]

I get 12 records (probably the number of attempts I've made). When I looked at the records in hfj_res_ver, there are a few duplicate entries. The length(res_text_vc) on the biggest records is about 79,300 characters.

Is there anything in particular I should be looking for?

[Pete]

So, I was able to reindex all resourceType's except Bundle when including the url params in my reindex requests. Unfortunately, reindexing the rest of the resources didn't shrink the DB at all. Is there a way to get the Bundle's, or is it ok just to leave them as is?