PageSize for FHIR Queries (Meds, Labs, etc)

[Cameron Hurst]

Am I doing something wrong? I'm passing "_Count=100" (or any number) and I never get more than 60 record back for my queries.

For example, my test patient has 99 Medication Statements but if always get 60, then next page of 39.

It would make my App WAY MORE performant if I didn't have to page.

Same appears true for other FHIR bundles.

Thanks in advance!!

[Max Philips (Cerner)]

Hi Cameron,

For paged resources in Cerner's sandbox, upper limits on requested page sizes are enforced behind the scenes.  We have these limits in place for performance reasons so that our server isn't tied up attempting to return e.g. thousands of records in a single page.  On MedicationStatement, you are running into this situation - the upper limit for page size is 60, and requesting higher page sizes (be sure to use "_count" instead of "_Count") will return at most 60 results per page.

Our server behaves this way based on HL7's definition for _count (http://hl7.org/fhir/DSTU2/search.html#count) - "a hint to the server regarding how many resources should be returned in a single page. Servers SHALL NOT return more resources ... but are allowed to return less than the client requested."  Each of our paged resources has a tuned behind the scenes cap to try to hit a sweet spot between returning as much information per page as possible while remaining performant.  Note that these caps may change over time as we monitor performance and adjust accordingly.

Thanks,

Max (Cerner)

[Jenni Syed (Cerner)]

I will add that the max limits may be adjusted behind the scenes based on performance testing and additional data we get over time. They're set resource by resource based on initial testing (memory limits, processing limits, etc).

~ Jenni

[Cameron Hurst]

That's very helpful - thank you both!

Can you advise limits per resource type? These are test patients with tons of records so it's probably not a big issue for us in the wild ...

[Mark Gidman]

Page size is generally documented on fhir.cerner.com. You'll find it in the details of the _count parameter.  Here's how it's written for Observation: 

The maximum number of results to return per page. Defaults to 50. Capped at 100.

Most of the caps are at 100, some smaller. Generally the cap is larger than the default so it's a good idea to send the _count parameter.

Paging is a huge deal in the wild. Even our friend Tim Peters is lightweight when it comes to real patient data. There's nothing in the sandbox that comes close to a real patient with a multiyear history and inpatient hospital stays. That said, Cerner has done a lot of work on the performance side and there are ways to tune your queries to get the data you need in a reasonable timeframe.

First things first - on the queries that allow you to filter on status - do that.  There's usually no point in fetching entered-in-error or draft status items.

In the wild, your biggest paging risks are Observations and MedOrders. Observations is like 10x+ everything else.  If you are gathering observations for social history, then using the category filter is your best bet from a performance perspective.  For vitals and labs, it's faster to get them together and sort them out yourselves.  If you are lucky enough to have a short list of LOINC codes you need, filtering by those is the absolute fastest.

On our Timeline product we are currently only fetching 3 years of lab data, the full patient record for everything else.  Here's some of the record counts for a sample patient with one hospital stay in that time period.

Encounters (192) Meds (1116) Labs (1741) Vitals (3587) Procedures (11) Diagnoses (132) Diagnostic Reports (57)

Observations represent both labs and vitals in our counts above - so thats over 5000 records - being loaded in groups of 100 - it's a lot of pages. Since pages are loaded sequentially, only getting the link to page 2 after you get page 1, this represents 50+ pages. Even at 1 second each, that's almost a full minute of load time, for only 3 years of records. Often, observation queries are a bit longer than 1 second with data transfer time.  

If you need your data faster than that, you'll need to use the date filtering query params to break your range into smaller chunks and multi-thread the paging.  For us, we take the last 3 years and break them up into 36 separate months and ask for them all at the same time. The non-hospitalization periods will be relatively light and will generally not trigger paging. The months that had a hospitalization in them will trigger paging...sometimes 20-30 pages deep. Still, you have the rest of the data already so the user is not stuck waiting for something useful, especially if they don't need the data in an old hospitalization period.

- Mark

[Jenni Syed (Cerner)]

Another item to consider is how the application is displaying or using the data. If the application is analyzing data or graphing it, more information can be consumed in a single view. But if you're displaying it as a list or individually for a user, the amounts of data mentioned above are far beyond what is typically useful for a user to see on a single page.

~ Jenni