CannotCreateTransactionException: Could not open JPA EntityManager for transaction

[Lars Kristian Roland]

Hi

I wonder if anyone has seen this issue before and can help me. I've had the HAPI-FHIR server up and running for a good while now, but suddenly I'm getting this exception on searches (getting single resources is working. Not sure if this is an issue with the database infrastructure or somehow that the database is corrupt. Any help would be appreciated!

(Also sorry for posting this to the group before I'm done debugging. But this is running in production so I'm a bit desperate... apologies for that... )

2024-08-20 09:49:07.790 [hapi-fhir-jpa-scheduler-clustered-2] ERROR org.quartz.core.JobRunShell [JobRunShell.java:211] Job HAPI.ca.uhn.fhir.jpa.subscription.async.AsyncResourceModifiedProcessingSchedulerSvc threw an unhandled Exception:
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:466)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:531)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:405)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:137)
        at ca.uhn.fhir.jpa.dao.tx.HapiTransactionService.doExecuteCallback(HapiTransactionService.java:435)
        at ca.uhn.fhir.jpa.dao.tx.HapiTransactionService.doExecuteInTransaction(HapiTransactionService.java:320)
        at ca.uhn.fhir.jpa.dao.tx.HapiTransactionService.doExecute(HapiTransactionService.java:271)
        at ca.uhn.fhir.jpa.dao.tx.HapiTransactionService$ExecutionBuilder.execute(HapiTransactionService.java:519)
        at ca.uhn.fhir.jpa.dao.tx.HapiTransactionService$ExecutionBuilder.execute(HapiTransactionService.java:512)
        at ca.uhn.fhir.jpa.subscription.ResourceModifiedMessagePersistenceSvcImpl.findAllOrderedByCreatedTime(ResourceModifiedMessagePersistenceSvcImpl.java:86)
        at ca.uhn.fhir.jpa.subscription.async.AsyncResourceModifiedSubmitterSvc.runDeliveryPass(AsyncResourceModifiedSubmitterSvc.java:52)
        at ca.uhn.fhir.jpa.subscription.async.AsyncResourceModifiedProcessingSchedulerSvc$Job.execute(AsyncResourceModifiedProcessingSchedulerSvc.java:63)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection [HikariPool-1 - Connection is not available, request timed out after 30001ms.] [n/a]
        at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:51)
        at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:58)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:108)
        at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:94)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:116)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:143)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:273)
        at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:281)
        at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:232)
        at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
        at org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:176)
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:420)

--

Lars Kristian Roland

Email: la...@roland.bz

Phone: +47 90733036

[James Agnew]

For what it's worth, that exception isn't a search, that's a scheduled job running. Your failing search would be a different stack trace. That issue looks to me though like your connection pool is exhausted. No idea why and there are plenty of possible causes, but if I were in your shoes the my next step would be to grab a thread dump for analysis and then restart the server to see if that brings things back up. The thread dump may have clues about why the pool has no more connections available.

Cheers,

James

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/CANjCRy8n-Q6AZALY7qp9Ciiu9CdB59YbAjMzeFCp4f-ybF6s7Q%40mail.gmail.com.

[Lars Kristian Roland]

OK, thanks. It definitely seems like a database issue rather than a HAPI-FHIR issue, so I'll concentrate the efforts there. 

When debugging further it seems like I can read from the database (including getting exact resources with url such as /fhir/Patient/<resourceid>), but I cannot search (/fhir/Patient) or write. I think perhaps the search does some writing to the db? When testing with another application, it can run selects, but not updates. So I'll focus on getting the db up and running again... 

Thanks for the help, and for the great server software. 

Best regards, 

Lars

[James Agnew]

FWIW Search does write by default - it creates an entry for each search in the HFJ_SEARCH table. There's a bunch of documentation on how this works here. This is Smile documentation, but it's describing how it also works in HAPI FHIR.

One thing you could try is adding a header to your search like the following:

Cache-Control: no-cache, no-store, max-results=50

That header disables any writes during the search - If that works then you have a writing problem in the DB I suppose.

Cheers,

James