Hapi extremely slow for first call

[Johanna Häusler]

Hello all,

I am running hapi in a container (version 6.10.1) with an oracle database that contains ~40k resources. I have the issue that the initial call to hapi (first read and first write after startup) is so slow that no client ever gets an answer. Creating a simple patient resource lasts ~4 minutes and searching by identifier lasts ~1 minute. Once these initial calls were done by hapi (when I see the actual execution in the logs) hapi is working in its normal and totally fine speed.

It is not as bad with a postgres database with the same amount of resources stored, but still the inital calls take way longer.

Has anyone experienced the same or can give an advice what to do about that?

Thanks in advance,

Johanna

[James Agnew]

I haven't seen this before, but FWIW a suggested first step would be to take a few thread dumps of the server while it's processing its first request and see what it's actually doing at that time. I am not aware of any bugs fixed that sound like this, but it's also always worth verifying whether you can replicate the issue on the current release of HAPI.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/2b32a19c-5ff7-4008-8b74-f11434363dfbn%40googlegroups.com.

[Tim Harsch]

Hi Johanna,

It's been a long time since I've worked with Oracle, but it might be worth going to the Enterprise Manager console and looking at the query logs to see if there is a query happening at that time that is taking a long time to resolve.  Assuming you have administrator privileges.

You could also switch the container to talk to a postgres or MySQL container to determine if the database is the culprit.

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/CAKowd8mfJAsyCSzr5GX3BnEkn%3DZp_BEOLTppaeniB%3D6-evyeWA%40mail.gmail.com.

[James Agnew]

Wow, that's fascinating - So the cause appears to be a very slow set of calls to "JarInputStream.getNextJarEntry".

Just googling that phrase with keywords like slow tomcat startup yields quite a few hits, this seems to be a thing lots of people have encountered. Maybe this would be helpful? https://medium.com/emlakjet/spring-boot-embedded-tomcat-getnextjarentry-f16df0c8032a

Might also be worth trying using Jetty instead of Tomcat to see if that helps?

Cheers,

James

On Tue, Sep 10, 2024 at 1:49 AM Johanna Häusler <jona...@gmail.com> wrote:

Thanks a lot for your input :)

I unfortunately don't have access to the oracle database, but other systems are connecting to the same database and they do not have any issues with it. I setup a minimal example on my localhost with postgres and attached a VisualVM to it.

Even though I am not close to the duration on the other system, I can reproduce the behavior:

Setup:

- Run Hapi in a container with the starter-project (version 6.10.1)

- Custom @Interceptor classes (docker-compose.yml: hapi.fhir.custom-interceptor-classes)

-Custom spring package (application.yaml: custom-bean-packages)

First run:

- Use an example Slot resource that is sent via POST

- First invocation lasts ~9 seconds

- 8.4 seconds are spent here: ca.uhn.fhir.jpa.dao.BaseHapiFhirDao.createHistoryEntry()

- the time is split in two major parts ending up in the same method for ~4 seconds: java.util.jar.JarInputStream.getNextJarEntry()

Second run:

- Use the same example Slot resource and send it again via POST

- Second invocations lasts ~300ms 

- 99.2ms are spent here: ca.uhn.fhir.jpa.dao.BaseHapiFhirDao.createHistoryEntry()

I can share the above information in screenshots - I beg your pardon that I can not share the entire snapshot. 

Maybe I am using the custom classes in a wrong way.

Kind regards,

Johanna