HAPI FHIR Server can not return all resources of some type?

[Andrew Guselnikov]

Hello, 

I stuck with one problem that I don't know how to resolve.  

The case is the following:

1. There are a lot of resources (~10M) of one type on FHIR server. Let's say Claim.

2. I need to get them all to update one field because of erroneous previous upload

3. I wrote simple app that performs [base]/Claim REST call (to get resources page by page) and all calls to the next page 

=> 

On one happy moment processing stops - as a result I got only ~350K resources. 

I found that there is an option SearchPreFetchThresholds  that regulates if I'm not mistaken the number of items that HAPI tries to get as a prefetch

search_pre_fetch_thresholds=500,2000,10000,50000,100000,300000,600000,1000000,1500000,2000000,-1

I suspect that if there are a lot of items,  this mechanism tries to get some big number as a prefetch and is not able to do so

I was able to get more results with changing this option max_millis_to_wait_for_remote_results

But not all anyway. 

if I just write here something like 

search_pre_fetch_threshold=500, in order to prefetch to be constant  - I can see that I get only 500 items as result, but don't understand why...

Could you please help me?