Bulk Import Queue

[Edward Miller]

Hello -

I'm a new user here and relatively inexperienced with HAPI. I am currently using the jpa-starter project and attempting to use /$import for a relatively large dataset. However, when I run the import, I end up getting a 500 error on the polling result with the following error message:

"Job is in FAILED state with 38 error count. Last error: Too many errors: 4. Last error msg was HAPI-0550: HAPI-0823: The operation has failed with a version constraint failure. This generally means that two clients/threads were trying to update the same resource at the same time, and this request was chosen as the failing request."

Further, after this failure, it appears that the request does not stop processing and causes any pending requests to be blocked.

My questions are:

1. has anyone encountered this before and if so, what was your solution?

2. Are there any configurations that need to be adjusted?

Thanks,

Ed

[Keith Boone]

If you are importing a large data set and it contains duplicate resources, you could run into this problem.

 

               Keith

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/545815c5-3116-43f2-90ee-f8c07f4614d1n%40googlegroups.com.

No PHI in Email: PointClickCare and Collective Medical, A PointClickCare Company, policies prohibit sending protected health information (PHI) by email, which may violate regulatory requirements. If sending PHI is necessary, please contact the sender for secure delivery instructions.

Confidentiality Notice: This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

[Sean McIlvenna]

Hey Keith,

That's note the case here.

The import itself does not contain duplicate IDs, but some of the resources in the import already exist on the HAPI FHIR server.

It seems to be trying to create a duplicate entry in the `hfj_forced_id` table...

Here's the error message I'm seeing that makes me think that:

--------

2023-05-03 22:08:50.588 [batch2-work-notification-47] ERROR o.h.e.j.batch.internal.BatchingBatch [BatchingBatch.java:134] HHH000315: Exception executing batch [java.sql.BatchUpdateException: Batch entry 2 insert into hfj_forced_id (partition_date, partition_id, forced_id, resource_pid, resource_type, pid) values (NULL, NULL, 'direct-trust-20230501-4', 2755619, 'Organization', 2755319) was aborted: ERROR: duplicate key value violates unique constraint "idx_forcedid_type_fid"

Detail: Key (resource_type, forced_id)=(Organization, direct-trust-20230501-4) already exists. Call getNextException to see other errors in the batch.], SQL: insert into hfj_forced_id (partition_date, partition_id, forced_id, resource_pid, resource_type, pid) values (?, ?, ?, ?, ?, ?)

2023-05-03 22:08:50.588 [batch2-work-notification-47] WARN o.h.e.jdbc.spi.SqlExceptionHelper [SqlExceptionHelper.java:137] SQL Error: 0, SQLState: 235

--------

In this case, Organization/direct-trust-20230501-4 is not duplicated in the import ndjson file, but it already exists on the FHIR server.

The above error, for the record, only shows up in server logs, not in the response from the $import request or $status.

Any ideas?

[Kevin Dougan]

Have you tried using the ifNoneExist parameter on the Resources in your Bundle?

https://smilecdr.com/docs/fhir_standard/transactions.html#conditional-create

[Sean McIlvenna]

Hi Kevin,

It's not a Bundle we're dealing with... It's an .ndjson file that is used by Bulk Import.

So, there's no place, that I know, in the .ndjson format to specify ifNoneExist.