Understanding FHIR incremental export and import

[Philip Roberts]

Hi all,

I am trying to implement a backup strategy for our FHIR data stores. As I understand it the only option for this is to use export/import. From a cost and performance perspective it would make sense to use incremental exports to only export data that has changed since the last export.

However, the docs on how to actually handle this are a little light, wondered if anyone can answer any/all of the below?

1. How should importing incremental exports be done? The docs say you can just import all files from a directory with wildcards - however, from my initial testing it doesn't seem like the import process resolves versions in any way if different versions of the same resource appear across exports - and it seems like the last imported value wins, even if it's an older version? Given that the export filenames are not ordered and are just based on the operation id, this makes importing from incremental exports difficult. Am I missing something here?

2. Relatedly - am i correct in understanding that a) exports only export the latest version of a resource; and imports don't maintain version history (related to above I guess) meaning that a full export and import of a store will lose all version history.

3. Is there any guidance of how to handle the `since` parameter for incremental exports? Is there a returned value that i should use from one export for the next to ensure i don't miss data? Or should i just maintain my own value and ensure it's older than the previous backup to avoid missing data?

Thanks in advance!

[Truc Le]

Hi Philip Roberts,

Thanks for reaching out!

Our FHIR store comes equipped with a built-in internal backup method for data protection. Additionally, we offer a rollback feature that allows you to restore resources to a specified point in time.

For more comprehensive backup strategies, we also have exportHistory and importHistory APIs available in public preview. These can be valuable tools for your needs.

Please let us know if you have any further questions.

Best

Truc

[Philip Roberts]

Hi Truc Le,

Thanks for the response!

We are ultimately trying to ensure we have sufficient disaster recovery in place for our application. For context we use a data set per customer in a single data store to back a SaaS application.

Ultimately I am trying to figure out:

1. If say the GCP healthcare api has a major issue, what protections are in place.

2. If an entire data set for one of our customers customer was deleted either maliciously or accidentally in production, could we restore the data set

3. If a customer's data got messed up in some other way can we recover back to a point in time.

> Our FHIR store comes equipped with a built-in internal backup method for data protection. 

This sounds like it would cover situation (1)? Would it help us restore data if we deleted a dataset for a customer (2)? Are these backups available to us/to be selectively restored, or are these just about overall system stability/protection? Can you talk more to this?

> Additionally, we offer a rollback feature that allows you to restore resources to a specified point in time.

I have taken a look at this. This looks like it would help with (3) but unless I am mistaken it doesn't look like it would help us with (2) recovering an entire store if deleted?

> For more comprehensive backup strategies, we also have exportHistory and importHistory APIs available in public preview. These can be valuable tools for your needs.

Thanks I'll take a deeper look at these? Will the importHistory endpoint better merge histories for the same resource if split across multiple incremental exports? Or is it still last-import-wins like the import api? Also, i tried the exportHistory api on a tiny store (less than 10 resources total) and it took ~10mins vs ~4s for `export` - is that a limitation of the beta? Some of our production stores are much much larger (100k+ resources).

Thanks for anything you can provide to help me out!

Philip

[Truc Le]

Thanks for providing the helpful context.

> Would it help us restore data if we deleted a dataset for a customer (2)? Are these backups available to us/to be selectively restored, or are these just about overall system stability/protection? Can you talk more to this?

> I have taken a look at this. This looks like it would help with (3) but unless I am mistaken it doesn't look like it would help us with (2) recovering an entire store if deleted?

Deleted dataset (or FHIR store) needs a separate procedure to recover. In case this happens, please file an urgent support ticket for our product team because deleted dataset or store can only be recovered within 48 hours since the deletion time.

For other cases, the rollback API would revert the store (or a subset of the store based on the request) to a specific point-in-time.

> Will the importHistory endpoint better merge histories for the same resource if split across multiple incremental exports? Or is it still last-import-wins like the import api?

ExportHistory outputs history bundle for each resource. If you export multiple snapshots of the store from different periods, it's fine to use ImportHistory (not the regular import) to merge all of them into the store.

> The exportHistory api on a tiny store (less than 10 resources total) and it took ~10mins vs ~4s for `export` - is that a limitation of the beta? Some of our production stores are much much larger (100k+ resources).

The ExportHistory scales better with large store and hence it doesn't perform well on small store. On the large export, it scales similarly or even better than the regular export.

To summarize your original questions:

> If say the GCP healthcare api has a major issue, what protections are in place.

We have our own database backup and we would be able to restore production data for catastrophic events. That's being said, customers could have their own backup if there is no cost concern.

> If an entire data set for one of our customers customer was deleted either maliciously or accidentally in production, could we restore the data set

We have 48-hour window to fully recover deleted dataset or datastore. Please file support ticket urgently in this case.

> If a customer's data got messed up in some other way can we recover back to a point in time.

Looks like this is a good use case of the rollback API.

Hope that helps!

Truc

[Philip Roberts]

All very helpful, thanks Truc Le,

One question re: importHistory. It looks from error messages I am receiving like `enable_history_modifications` (or `enableHistoryModifications` different places refer to it differently) must be enabled on the store for importHistory to work. However, I cannot seem to enable this setting. I have tried both creating a new store with this setting enabled; or enabling it manually by editing the store configuration in the console UI, but no matter what I do the setting remains disabled. Any ideas what might be happening here?

Thanks in advance,

Philip

[Truc Le]

Could you share your full API request and the returned error? The feature is only available in public preview (a.k.a. v1beta1) so likely you need to change the endpoint. Here is an example curl command to update "enable_history_modifications" on an existing FhirStore (please replace with your own Project, Location, Dataset, and FhirStore):

curl -S -H "Authorization: Bearer $(gcloud auth print-access-token)" -H "Content-Type: application/json" -X PATCH -d '{"enable_history_modifications":true}' "https://healthcare.googleapis.com/v1beta1/projects/${PROJECT?}/locations/${LOCATION?}/datasets/${DATASET?}/fhirStores/${FHIR_STORE?}?update_mask=enable_history_modifications"

Please let me know if you still run into issues.

[Philip Roberts]

Ah hah, yes okay that's the issue and that works. I don't get any errors if I use the v1 url it just doesn't update the setting, but if i do my own requests like that it does indeed work.

The UI though I guess just isn't handling this properly and even though it shows the setting because it's not using the v1beta1 url internally I guess it's not able to update the setting? So I guess that's just a UI bug in the console. Either way I'm unblocked and importHistory looks to be doing what I hoped it would so that's great, thanks for your help.

--
You received this message because you are subscribed to a topic in the Google Groups "GCP Healthcare Discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gcp-healthcare-discuss/xewssJI8R-Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gcp-healthcare-di...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gcp-healthcare-discuss/34220531-ce93-422e-ae04-3f86033be951n%40googlegroups.com.

[Truc Le]

Thanks a lot for letting us know the bug! We're going to fix the UI.