ValueSet/$expand requires Lucene or Elasticsearch?

[Xiaocheng Luan]

Hi,

Does ValueSet/$expand require  hibernate.search.enabled=true, that is, either Lucene or Elasticsearch?

We recently set up an instance (HAPI 7.2) using JPAserver starter with hibernate.search.enabled=false (No Lucene nor Elasticsearch), and the ValueSet/$expand gives an error that says

    "Failed to call access method: org.hibernate.search.util.common.SearchException: HSEARCH800001: Hibernate Search was not initialized"

Our other instance with hibernate.search.enabled=true with Elasticsearch works fine. But the HAPI FHIR documentation does not seem to indicate such a dependency. 

Can anyone help confirm this, or am I missing something?

As a separate, lesser issue, the instance with Elasticsearch throws a confusing error when "count" parameter is used, even if count=1 (but works without the count parameter):

"HAPI-0831: Expansion of ValueSet produced too many codes (maximum 1) - Operation aborted!"

The N in "maximum N" in the message is whatever you give as the count parameter.

 Thanks!

Xiaocheng

[James Agnew]

HAPI can expand basic valuesets without HSearch enabled, but there are lots of VS features it can't handle without the added index. Most filters you can put into a VS would be examples of those features.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/CAEsMcbvmjg12UviYhTxLh%3DhKxTX9uek_eEfJWGpOsac6qU8Q7Q%40mail.gmail.com.

[Xiaocheng Luan]

Thanks for the clarification, James!

Xiaocheng

[Xiaocheng Luan]

Hi James,

"HAPI can expand basic valuesets without HSearch enabled..."

The query "/ValueSet/$expand?url=http://hl7.org/fhir/ValueSet/yesnodontknow&_format=json", which seems to be a basic expansion query, also triggers the error "Hibernate Search was not initialized".

The specific ValueSet resource has expanded format, that is, its expansion.contains already has the 3 values.

Any ideas? 

Also, what's the configuration for hapi.fhir.org, that is, does it use Lucene or Elasticsearch?

Thanks,

Xiaocheng

On Wed, Nov 13, 2024 at 1:47 PM James Agnew <james...@gmail.com> wrote:

[James Agnew]

Hmm.. I would have expected that to work. Not entirely sure why it doesn't. Will need to dig into why..

The hapi.fhir.org public server is running Lucene currently, but I'm increasingly convinced that this is the reason for a lot of the instability over the last year on that server. I made a config change about 2 weeks ago to stop indexing or supporting FullText/Resource indexing (_text, _content, $lastn etc) on that server, and there have been no further crashes since.

I know this isn't what you're asking, but just since I'm already talking about it: I'm pretty seriously considering changing HAPI's stance that Lucene is a recommended option at this point. It seems to have pretty big memory management challenges, at least in our case. The instability on the server always seems to start with an OOM, and as soon as we reduced our use of Lucene those OOMs went away. I suppose maybe part of the problem is that Lucene probably doesn't assume it's running in a JVM where it's only a tiny part of the stack.

I don't know - I certainly don't want this to sound like an attack on Lucene. I think it's a really cool piece of technology, and obviously it's also the core of ElasticSearch. I just wonder if it's not intended to be used in applications like this one. At some point I'm definitely going to try provisioning a full dedicated (and separate infrastructure) ElasticSearch cluster and getting our public server pointed at that.

Cheers,

James

To view this discussion visit https://groups.google.com/d/msgid/hapi-fhir/CAEsMcbsbavKEZCW5c8cVR8CFoi9_j0s9Ny%2Ba_1KBqCnWHCTfUw%40mail.gmail.com.

[Xiaocheng Luan]

Thanks for sharing the details and thoughts, James!

In addition to competing for resources with HAPI, Luene is a library, it takes a lot more effort to build a high-performing (indexing and search), scalable, and stable search layer on top of it, and that's what Elasticsearch and/or Solr do. So I think you are heading in the right direction. 

There is one major challenge, though, with Elasticsearch, or with any other search strategy that's not the database itself, is what I call the "split memory" problem, or "join" problem. For example, when a query has multiple clauses, some to be fulfilled by the database and others by search. You can fetch 10,000 results from search but that may not be enough to produce a single result after "join", and that may cause performance problems or incomplete results set problems, or both. 

Our overall experience with using Elasticsearch with HAPI is that it generally works well but issues pop up once in a while, which is why we are experimenting with the Postgres-only configuration, and this value set expansion issue was noticed during this experiment. With sufficient attention (from the HAPI team), which I suspect is not yet the case, I think the Elasticsearch option is a viable solution.

Thanks,

Xiaocheng