How to disable Lucene/Hibernate Search?

[karl....@cms.hhs.gov]

Howdy all,

In a recent thread [1], James mentioned that it would be relatively simple to disable Hibernate Search/Lucene in HAPI:

If you don't need fulltext or terminology you could get away with disabling hibernate search entirely.

Some questions:

1. First, an uninformed one: what exactly are the terminology services that James is referring to here? What specific requests/resources/whatever would their absence break in HAPI?
2. Has anyone done this -- has anyone disabled Hibernate Search in HAPI? If so, how?

I took a very quick stab at this tonight, myself. Starting with the example JPA server [2] and some Googling (since I knew nothing about Hibernate Search before tonight), I ended up with this:

private Properties jpaProperties() {
  Properties extraProperties = new Properties();
  extraProperties.put("hibernate.format_sql", "true");
  extraProperties.put("hibernate.show_sql", "false");
  extraProperties.put("hibernate.hbm2ddl.auto", "update");
  extraProperties.put("hibernate.jdbc.batch_size", "20");
  extraProperties.put("hibernate.cache.use_query_cache", "false");
  extraProperties.put("hibernate.cache.use_second_level_cache", "false");
  extraProperties.put("hibernate.cache.use_structured_entries", "false");
  extraProperties.put("hibernate.cache.use_minimal_puts", "false");
  extraProperties.put("hibernate.search.default.directory_provider", "filesystem");
  extraProperties.put("hibernate.search.default.indexBase", "target/lucenefiles");
  extraProperties.put("hibernate.search.lucene_version", "LUCENE_CURRENT");
 
  // My brief Googling led me to believe that these two additional properties would disable Hibernate Search:
  extraProperties.put("hibernate.search.autoregister_listeners", "false");
  extraProperties.put("hibernate.search.indexing_strategy", "manual");
 
  return extraProperties;
}

Which yielded the following error when trying to deploy the resulting WAR:

12:11:04,099 INFO  [stdout] (default task-4) at org.hibernate.search.hcore.impl.SearchFactoryReference.getSearchIntegrator(SearchFactoryReference.java:35) ~[hibernate-search-orm-5.5.1.Final.jar:5.5.1.Final]
12:11:04,099 INFO  [stdout] (default task-4) at org.hibernate.search.hcore.util.impl.ContextHelper.getSearchintegratorBySFI(ContextHelper.java:44) ~[hibernate-search-orm-5.5.1.Final.jar:5.5.1.Final]
12:11:04,099 INFO  [stdout] (default task-4) at org.hibernate.search.hcore.util.impl.ContextHelper.getSearchintegratorBySessionImplementor(ContextHelper.java:37) ~[hibernate-search-orm-5.5.1.Final.jar:5.5.1.Final]
12:11:04,099 INFO  [stdout] (default task-4) at org.hibernate.search.hcore.util.impl.ContextHelper.getSearchintegrator(ContextHelper.java:33) ~[hibernate-search-orm-5.5.1.Final.jar:5.5.1.Final]
12:11:04,099 INFO  [stdout] (default task-4) at org.hibernate.search.impl.FullTextSessionImpl.getSearchIntegrator(FullTextSessionImpl.java:179) ~[hibernate-search-orm-5.5.1.Final.jar:5.5.1.Final]
12:11:04,099 INFO  [stdout] (default task-4) at org.hibernate.search.impl.FullTextSessionImpl.getSearchFactory(FullTextSessionImpl.java:172) ~[hibernate-search-orm-5.5.1.Final.jar:5.5.1.Final]
12:11:04,099 INFO  [stdout] (default task-4) at org.hibernate.search.jpa.impl.FullTextEntityManagerImpl.getSearchFactory(FullTextEntityManagerImpl.java:106) ~[hibernate-search-orm-5.5.1.Final.jar:5.5.1.Final]
12:11:04,099 INFO  [stdout] (default task-4) at ca.uhn.fhir.jpa.dao.FhirSearchDao.doSearch(FhirSearchDao.java:140) ~[hapi-fhir-jpaserver-base-1.4-SNAPSHOT.jar:na]
12:11:04,100 INFO  [stdout] (default task-4) at ca.uhn.fhir.jpa.dao.FhirSearchDao.search(FhirSearchDao.java:210) ~[hapi-fhir-jpaserver-base-1.4-SNAPSHOT.jar:na]
12:11:04,100 INFO  [stdout] (default task-4) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_31]
12:11:04,100 INFO  [stdout] (default task-4) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_31]
12:11:04,100 INFO  [stdout] (default task-4) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_31]
12:11:04,100 INFO  [stdout] (default task-4) at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_31]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,100 INFO  [stdout] (default task-4) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,101 INFO  [stdout] (default task-4) at com.sun.proxy.$Proxy102.search(Unknown Source) ~[na:na]
12:11:04,101 INFO  [stdout] (default task-4) at ca.uhn.fhir.jpa.dao.SearchBuilder.search(SearchBuilder.java:1574) ~[hapi-fhir-jpaserver-base-1.4-SNAPSHOT.jar:na]
12:11:04,101 INFO  [stdout] (default task-4) at ca.uhn.fhir.jpa.dao.BaseHapiFhirResourceDao.search(BaseHapiFhirResourceDao.java:912) ~[hapi-fhir-jpaserver-base-1.4-SNAPSHOT.jar:na]
12:11:04,101 INFO  [stdout] (default task-4) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_31]
12:11:04,101 INFO  [stdout] (default task-4) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_31]
12:11:04,101 INFO  [stdout] (default task-4) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_31]
12:11:04,101 INFO  [stdout] (default task-4) at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_31]
12:11:04,101 INFO  [stdout] (default task-4) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,101 INFO  [stdout] (default task-4) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,101 INFO  [stdout] (default task-4) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,101 INFO  [stdout] (default task-4) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,101 INFO  [stdout] (default task-4) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,101 INFO  [stdout] (default task-4) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) ~[spring-tx-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,102 INFO  [stdout] (default task-4) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,102 INFO  [stdout] (default task-4) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) ~[spring-aop-4.2.4.RELEASE.jar:4.2.4.RELEASE]
12:11:04,102 INFO  [stdout] (default task-4) at com.sun.proxy.$Proxy95.search(Unknown Source) ~[na:na]
12:11:04,102 INFO  [stdout] (default task-4) at ca.uhn.fhir.jpa.rp.dstu21.OrganizationResourceProvider.search(OrganizationResourceProvider.java:156) ~[hapi-fhir-jpaserver-base-1.4-SNAPSHOT.jar:na]
12:11:04,102 INFO  [stdout] (default task-4) ... 45 common frames omitted

Looking through the code, I see a fair number of references to Hibernate Search APIs in there. I'm guessing I have to produce a custom version of the JPA server module that avoids these APIs (and the HAPI functions that rely on them)? That would be a fairly beefy undertaking, I'm thinking, so any advice or corrections that folks can offer would be appreciated!

Thanks kindly,

Karl

[1] https://groups.google.com/forum/#!topic/hapi-fhir/dqIA-BAPJbk

[2] Frustratingly, I'm still stuck on a random HAPI 1.4 snapshot, though that should be changing this sprint. It'll be great to finally catch up.

[James Agnew]

Hi Karl,

1. First, an uninformed one: what exactly are the terminology services that James is referring to here? What specific requests/resources/whatever would their absence break in HAPI?

Terminology service in this context is stuff like the ability to upload CodeSystems and ValueSets and then validate codes against them, expand valuesets, validate resources which have bindings to those valuesets, etc.

In the absence of Lucene (and therefore the term services) you could still upload/search/etc CodeSystem and ValueSet resources, but th fancy terminology stuff wouldn't work.

1. Has anyone done this -- has anyone disabled Hibernate Search in HAPI? If so, how?

FWIW what you did looks correct to me. One thing I notice in your stack trace, it shows you using HAPI FHIR 1.4-SNAPSHOT, which is a somewhat old version.. Not ancient by any means (it was released Feb 2016) but there have been a bunch of enhancements and fixes since then.. I believe (although I'm failing to find the correct commit on my phone) that the ability to disable Lucene was only added in 1.6 or possibly 2.0. Are you in a position to try upgrading?

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+unsubscribe@googlegroups.com.
To post to this group, send email to hapi...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/91f7f01c-e0cf-4644-a608-f09d6dae239b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[karl....@cms.hhs.gov]

James, all,

Terminology service in this context is stuff like the ability to upload CodeSystems and ValueSets and then validate codes against them, expand valuesets, validate resources which have bindings to those valuesets, etc.

Interesting. As far as I know, that's not something we'll be doing, but I think it's a neat feature!

One thing I notice in your stack trace, it shows you using HAPI FHIR 1.4-SNAPSHOT, which is a somewhat old version.. Not ancient by any means (it was released Feb 2016) but there have been a bunch of enhancements and fixes since then.. I believe (although I'm failing to find the correct commit on my phone) that the ability to disable Lucene was only added in 1.6 or possibly 2.0. Are you in a position to try upgrading?

It was buried at the bottom of my post, but yeah: we're working this sprint on catching up to 2.0 and the STU3 structures. We've spent the last several months putting together a bunch of ETL code, and it was just easier to keep things stable until that was mostly done. We've got another dev working on the upgrade right now, and I may help him out some to get it done this week. I'm eager to see where the performance is at with 2.0!

All that aside, I spent a bit more time futzing with this, and if I removed the hibernate.search.autoregister_listeners=false setting (but kept the hibernate.search.indexing_strategy=manual change in), it looks like it might be doing what I want it to. Mostly. I can see the index directories getting created still, but they seem to stay empty, and my ETL just about doubles in speed -- a rather shocking speedup!

Do you have any info on how to disable Hibernate Search in 2.0? I took a look through the site, but didn't find it. I'd much rather use any supported mechanisms you've put in place than my own hacky attempts here.

Thanks very kindly,

Karl M. Davis

[James Agnew]

Hi Karl,

The way you've done it is correct. Specifically the "autoregister_listeners" bit. There's nothing more fancy than that. :)

Cheers,

James

--

You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+unsubscribe@googlegroups.com.
To post to this group, send email to hapi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/4946005d-03af-401e-8add-443f4005a8de%40googlegroups.com.

[karl....@cms.hhs.gov]

Okay, thanks! Glad to hear that error from my original post should go away. I'll give that a try just as soon as we have things running in 2.0.

On Sunday, October 16, 2016 at 11:27:49 PM UTC-4, karl....@cms.hhs.gov wrote:

[Kevin Mayfield]

So if I understand this correctly, by adding this line

extraProperties.put("hibernate.search.indexing_strategy", "manual");

 We disable hibernate search and this would be a recommended option when doing an ETL process. 

This implies we would need to enable indexing after the ETL to ensure optimal READ performance. 

Does this sound correct?

p.s. I am getting a similar improvement while running a load.

[karl....@cms.hhs.gov]

Kevin,

That does work for me in a pre-1.4 release snapshot of HAPI. However, James said that in HAPI 1.6+, the supported mechanism is to instead set hibernate.search.autoregister_listeners=false.

And I don't believe the Lucene indexes speed anything up, per-se, it's that certain operations will completely fail without them.

Thanks very much for your confirmation on the indexes' effects on write speed performance. What version of HAPI are you seeing that with?

Best regards,

Karl

[karl....@cms.hhs.gov]

James,

Some further benchmarking indicates that, while Lucene data might not be persisted when hibernate.search.autoregister_listeners=false, some sort of Lucene-related processing is still occurring that slows things down. Put another way: if that's all that I set, I don't see any performance improvements from disabling Lucene.

Adding hibernate.search.indexing_strategy=manual as well does yield the performance improvements that I reported earlier, though.

Best regards,

Karl M. Davis

On Sunday, October 16, 2016 at 11:27:49 PM UTC-4, karl....@cms.hhs.gov wrote:

[James Agnew]

Fascinating. That seems to go against what the hibernate search documentation claims, but it doesn't totally surprise me that it's not right :)

-James

--

You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+unsubscribe@googlegroups.com.
To post to this group, send email to hapi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/9ee47432-8dd0-46c6-9ba5-52d54d5750e4%40googlegroups.com.