Reindexing failed

[Shivam Bhaskar]

Hi team,

we upgraded our HAPI FHIR application from v6 to v7.2.0. during which the reindexing operation failed on some of the resources with error

"HAPI-0550: Newer version [2] of entity [[ca.uhn.fhir.jpa.model.entity.ResourceTable#57154164]] found in database"

total ETA for reindexing was 4 days and it failed after 3 days twice with the same error. the application was up and running we could not afford a 4 day downtime.

1. does the reindex operation start from the scratch every time we execute it? is there a way to complete reindexing without a 4 day downtime?

2. The QuestionnaireResponse resource reindexing failed with error 

Unable to perform afterTransactionCompletion callback: HSEARCH800024: Unable to index documents for indexing after transaction completion: HSEARCH700124: Indexing failure: this IndexWriter is closed.
The following entities may not have been updated correctly in the index: [ResourceTable#19334132, ResourceTable#19296230, ResourceTable#19334626, ResourceTable#19334721, ResourceTable#19270768, ResourceTable#19277594, ResourceTable#19334727].

3. To reindex we sent a POST request through postman on the fhir server url
with the following payload

{ "resourceType": "Parameters", "parameter": [ { "name": "optimizeStorage", "valueCode": "ALL_VERSIONS" }, { "name": "reindexSearchParameters", "valueCode": "ALL" }, { "name": "url", "valueString": "ClinicalImpression?" }, { "name": "url", "valueString": "DocumentReference?" }, { "name": "url", "valueString": "ClaimResponse?" }, { "name": "url", "valueString": "Appointment?" }, { "name": "url", "valueString": "Claim?" }, { "name": "url", "valueString": "Account?" }, { "name": "url", "valueString": "ActivityDefinition?" }, { "name": "url", "valueString": "AppointmentResponse?" }, { "name": "url", "valueString": "AllergyIntolerance?" }, { "name": "url", "valueString": "CareTeam?" }, { "name": "url", "valueString": "CodeSystem?" }, { "name": "url", "valueString": "Communication?" }, { "name": "url", "valueString": "Composition?" }, { "name": "url", "valueString": "ConceptMap?" }, { "name": "url", "valueString": "Condition?" }, { "name": "url", "valueString": "Coverage?" }, { "name": "url", "valueString": "CoverageEligibilityRequest?" }, { "name": "url", "valueString": "CoverageEligibilityResponse?" }, { "name": "url", "valueString": "Device?" }, { "name": "url", "valueString": "DeviceUseStatement?" }, { "name": "url", "valueString": "DiagnosticReport?" }, { "name": "url", "valueString": "Encounter?" }, { "name": "url", "valueString": "EpisodeOfCare?" }, { "name": "url", "valueString": "Flag?" }, { "name": "url", "valueString": "List?" }, { "name": "url", "valueString": "Location?" }, { "name": "url", "valueString": "Media?" }, { "name": "url", "valueString": "Medication?" }, { "name": "url", "valueString": "MedicationAdministration?" }, { "name": "url", "valueString": "MedicationRequest?" }, { "name": "url", "valueString": "MedicationStatement?" }, { "name": "url", "valueString": "Organization?" }, { "name": "url", "valueString": "Patient?" }, { "name": "url", "valueString": "PlanDefinition?" }, { "name": "url", "valueString": "Practitioner?" }, { "name": "url", "valueString": "PractitionerRole?" }, { "name": "url", "valueString": "Procedure?" }, { "name": "url", "valueString": "Provenance?" }, { "name": "url", "valueString": "Questionnaire?" }, { "name": "url", "valueString": "QuestionnaireResponse?" }, { "name": "url", "valueString": "RequestGroup?" }, { "name": "url", "valueString": "Schedule?" }, { "name": "url", "valueString": "SearchParameter?" }, { "name": "url", "valueString": "ServiceRequest?" }, { "name": "url", "valueString": "Slot?" }, { "name": "url", "valueString": "StructureDefinition?" }, { "name": "url", "valueString": "Task?" }, { "name": "url", "valueString": "ValueSet?" } ] }

4. We are planning to stay up to date with the HAPI FHIR versions. to upgrade from 7.2.0 to 7.8.0 we are afraid the previous incomplete reindexing could cause an issue. is there any workaround to get it done?

[James Agnew]

Hi Shivam,

> does the reindex operation start from the scratch every time we execute it? is there a way to complete reindexing without a 4 day downtime?

It does. It looks from your start URL though that you are running a reindex for a large number of resource types. You can do these types individually one after the other, which would certainly let you better segment the job and restart only the portion that failed. Also, no downtime is needed during reindex, the reindex job is designed to be used online.

> Unable to perform afterTransactionCompletion callback: HSEARCH800024: Unable to index documents for indexing after transaction completion: HSEARCH700124: Indexing failure: this IndexWriter is closed.

This is a Lucene error. I would guess you have some kind of Lucene index corruption causing these issues (I've seen this in the past if a disk fills up, or HAPI+Lucene isn't shut down cleanly). Do you need Lucene enabled? If not, I might suggest disabling it and these errors should go away. If so, it may be worth considering moving to ElasticSearch, which is much more resiliant.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/c9c94105-d3da-490f-91b4-f7d36165e5b8n%40googlegroups.com.