Reindex failing

[Pete]

Hello, 

I'm trying to reindex my FHIR instance (hfj_spidx_token is 23GB, so a lot of stuff). Running $reindex is taking hours, which I'd expect, but it keeps failing and rolling back. In the batch table, I just see the error "Transaction silently rolled back because it has been marked as rollback-only". Can someone help me understand what could cause the reindex to fail?

I'm running it with the following body in the request

{

"resourceType": "Parameters",

"parameter": [ {

"name": "everything",

"valueBoolean": "true"

}, {

"name": "batchSize",

"valueDecimal": 1000

}]

}

[Pete]

Also, will reindexing shrink hfj_spidx_token? I retired a number of search parameters, but they're still showing up in this table after reindexing the specific resource type.

[James Agnew]

You would need to check the server logs to see why the transaction is rolling back. Typically this is caused by some kind of error when reindexing a particular resource, but there are all kinds of reasons that can happen. Try looking in the logs for the first instance of "Transaction silently rolled back", then work backwards from there - the error that caused it should be in the logs shortly before.

The reindex process should absolutely be deleting rows in the index tables related to Search Parameters that are no longer active. If you see a row in that table you expect should no longer be there after the reindex, you could try doing a single $reindex-dryrun (link is to an example on the public hapi server) on it to see if the system believes it should actually be deleting index rows.

Also - if you are on an old version of HAPI, consider upgrading. Reindexing and batch jobs in general have had a lot of optimization and performance work over the last few releases.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/2e71ce47-aa0c-4deb-b945-e6f7abd8e832n%40googlegroups.com.

[Pete]

Thanks James,

I tried rerunning the job and see this error in the bulk job table "Following up here, after retrying the job I get the error "Too many errors: 6. Last error msg was possible nonthreadsafe access to session".

[Pete]

Sorry, sent the reply too soon. Looking through our logs, it looks like it was indeed a particular resource (see error at the end). Is there a way to set it so that this is done as single threaded? We're running this on AWS with 3 instances running.

I'm also planning on upgrading to 8.0.0, but for some reason our validation isn't working as it was in 6.4. I have a separate thread on that, so won't distract from that thread here.

Failure reindexing Observation/496899: HAPI-0826: The operation has failed with a version constraint failure. This generally means that two clients/threads were trying to update the same resource at the same time, and this request was chosen as the failing request.