Paging the $everything Operation

[Michael Litcher]

Is there a reason "offset" paging is not allowed on the /Patient/{id}/$everything operation using the JPA starter? Am I able to override this?

i.e. I want to be able to do this:

/Patient/{id}/$everything?_count=100&_offset=100

[Anton Kurylovich]

Hey there, just to clarify, what do you mean by 'not allowed' are you seeing an error or something? Or everything/nothing being returned regardless of the query params? Because I see the query param available in the swagger of the test server: https://hapi.fhir.org/baseR4/swagger-ui/?page=Patient#/Patient/post_Patient__id___everything so it should be allowed I think.

Does it work for other searches you are performing? I wonder if you need to set a paging provider? https://hapifhir.io/hapi-fhir/docs/server_plain/paging.html 

I see the starter project has this search provider out of the box: https://github.com/hapifhir/hapi-fhir-jpaserver-starter/blob/1f7d25c5e57b1901fb8290151dc41a3d238933bf/src/main/java/ca/uhn/fhir/jpa/starter/common/StarterJpaConfig.java#L121 

Do you have the same in your JPA starter?

Unless I am missing something special about the $everything operator of course, but if it returns a bundle, I imagine paging should work?

[Michael Litcher]

I'm seeing a "not supported" response when you include the _offset parameter in the $everything operation. All other paging works as expected.

You can see example here: https://hapi.fhir.org/baseR4/Patient/123/$everything?_count=10&_offset=10

Source here: https://github.com/hapifhir/hapi-fhir/blob/rel_6_8/hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/dao/JpaResourceDaoPatient.java#L76

Some context, I'm trying to use "offset" paging in my implementation exclusively, because the database platform I'm trying to use does a really good job of caching, so I wanted to try avoiding the database writes for searches.

[Anton Kurylovich]

Interesting, thanks for the additional info. It looks like this is issue where HAPI started to reject the param in the everything operation: https://github.com/hapifhir/hapi-fhir/pull/2579, so if we wanted to try to re-introduce it, that's what we would have to look out for. So I guess the answer of the original question of why it is unsupported can be traced to this comment: https://github.com/hapifhir/hapi-fhir/pull/2579#discussion_r622896945 

[Michael Litcher]

Hmm, ok. I'm not trying to support "type/$everything", just "instance/$everything". Do you think there is any reasonable way to provide an override to this while still using the JPA server?

[Anton Kurylovich]

I don't think so. Since the error is thrown pretty deep in the codebase, at the DAO level, I doubt we have many options in terms of overriding. For example none of the interceptor pointcuts can go that deep: https://hapifhir.io/hapi-fhir/docs/interceptors/server_pointcuts.html 

Only ideas I have are: 

1) Branch HAPI, and try to make a change to not throw an exception and point your starter project dependency to your version of HAPI (just for testing purpose, and can submit a pull request if it works) 

2) If you want to get real hacky you can create your own dao class in your project and make sure it has the exact same name + path as the existing one so that maven should use your DAO when it compiles the project. This is a super iffy and hacky approach, I'm not sure it would even work

Hope that made some sense

[Michael Litcher]

That makes sense. Number 2 does work, but like you said, not ideal. However, for my use case, might be ok for a tactical approach while I look into number 1.

It was mentioned in the github PR comments, but the Patient is repeated on each page. I think I need to test offset paging a bit more in general. When using includes/revincludes, the synchronous search operates a little different. What I would have expected is: the search generates a list of resource ids, including whatever is brought in by the include, and then paginates that entire set. But, when includes/revincludes are involved, it loads all those at once, or up to the defined max include per page, and even though a "next" link might be shown, doesn't actually paginate the whole result set.

[Michael Litcher]

I re-read the spec on the _count parameter, and I understand now that it doesn't include the resources brought in by _include/_revinclude. The separate max_includes_per_page setting makes sense now.

I'm still looking into what other issues are present (if any) with offset paging on the $everything operation.

[Anton Kurylovich]

Good to know!