Resource content compression

[Lars Kristian Roland]

Hi, and thanks for a great product.

I'm storing some quite large Observation resource objects and would like to compress the storage of these in the database. 

I can see there's an earlier topic on compression in the the hfj_res_ver res_text column, but all my data seem to be in the res_text_vc (and it's in cleartext). 

It seems RES_ENCODING sort of manages this, but I don't quite understand how to 'force' storage to be JSONC, or if it's possible.  

Can it be set per resource, for example in the REST POST HTTP header?

If compression is possible, will it be possible to change on an existing database so that the old (uncompressed) data and the new (compressed) resources both still work?

Best regards,

Lars

[Lars Kristian Roland]

Hi,

We're getting to a stage where we need to compress resources. Does HAPI-FHIR support this? Or do I need to implement something myself or have large resources stored outside? 

Best regards,

Lars

[James Agnew]

Hi Lars,

We have deprecated and are hoping to eventually remove the RES_TEXT column on the HFJ_RES_VER table, and as a result we don't provide any way of forcing new data to use it. The issue is that this column uses the pg_largeobject table for storage on Postgres, and this is very problematic - The postgres documentation generally recommends against using it, it can't store more than 2^32 entries in total, it is slower than other ways of accessing data, etc.

We're currently planning on implementing a mode which offloads the resource bodies entirely into external storage (e.g. S3 or Azure Blob Storage), although this isn't yet complete. There is also Externalized Binary Storage which could be used if your resources are large because of binary attachments or Binary resources. Otherwise though, I think looking at lower level compression options (e.g. whole-disk compression) is probably the only feasible option at this point.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/cb197e15-9e9e-4bbc-ab23-6414800cbc0an%40googlegroups.com.

[Lars Kristian Roland]

Ok, thanks. I will create a workaround in the meantime and eagerly look forward to an S3 storage for (all, some or archived) resources will become available. I might implement some sort of redirect to S3 for the largest archived observations in pur frontend. Our largest observations are ECG which are several megabytes, but we also have many tiny ones. 

Lars 

Lars Kristian Roland

Email: la...@roland.bz

Phone: +47 90733036