Is it okay to point multiple JPA Servers at a single PostgreSQL instance?

[as...@yahoo.com]

I suspect the answer is yes but wanted to confirm.  I will be deploying two separate Tomcat servers, each hosting a JPA Server, and would like to use the same PostgreSQL instance.

[Seth Rylan Gainey]

What was the outcome? We are also considering multiple servers with a single schema to execute an API upgrade.

    - Seth

[Xiaocheng Luan]

The database side might be fine but the Lucene search side will be out of sync?

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/a98e45dd-02b4-4dac-9521-932b8792dfec%40googlegroups.com.

[Jens Villadsen]

Lucene can be shared using in ElasticSearch. Besides that, are there then any known gotcha's?

On Tuesday, July 30, 2019 at 6:45:42 PM UTC+2, Xiaocheng Luan wrote:

The database side might be fine but the Lucene search side will be out of sync?

On Jul 30, 2019, at 11:12 AM, Seth Rylan Gainey <seth...@gmail.com> wrote:

What was the outcome? We are also considering multiple servers with a single schema to execute an API upgrade.

    - Seth

On Tuesday, November 21, 2017 at 10:29:54 PM UTC-5, as...@yahoo.com wrote:

I suspect the answer is yes but wanted to confirm.  I will be deploying two separate Tomcat servers, each hosting a JPA Server, and would like to use the same PostgreSQL instance.

--
You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hapi...@googlegroups.com.

[Seth Gainey]

> any known gotcha's?

We researched that question in the 3.8 hapi codebase. The reindexing operation (implemented in ResourceReindexingSvcImpl) is only non-scalable operation because its threading model assumes a single process reading and writing the HFJ_RES_REINDEX_JOB table.

   - Seth

[Sajith Jamal]

So does this mean we should not point multiple JPAs to the same DB for HAPI?

[Patrick Palacin]

Is there any solution for this?

Spawning multiple JPA servers, especially in a kubernetes cluster is key for high availability.

The db (psql) is already clustered and not part of k8s, so only multiple pods of the application server would be required.

I would say it's key for every kind of production deployment.

[Panayiotis Savva]

I am also very interested in this as I'm also running my Hapi Fhir JPA Server on a Kubernetes cluster.  To obtain high availability, I'm planning to run 3 instances of the JPA Server...

--

You received this message because you are subscribed to the Google Groups "HAPI FHIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hapi-fhir+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/2bd3b336-e166-4a91-b329-582f358f83f4n%40googlegroups.com.

[James Agnew]

This is absolutely possible and is often done. The server does not rely on any internal (in memory) state that would create database issues, with one exception: There are many scheduled maintenance jobs that must be executed once and only once per scheduling period. The scheduler must be disabled on all nodes except one.

Cheers,

James

To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/CADbgrcqvv-Xo-p_PccM1vh9EdCUwH9oaEvaLZC5NrfHxOdGWsA%40mail.gmail.com.

[Keerthivasan Ramanathan]

I'm really interested in this conversation. Thanks James for letting us know about the schedulers. Can you exactly help us find the config to disable it? Can you provide more details on schedulers like its functionality and the relevant class files. It would be really helpful. Thanks for you time and effort. we really appreciate it. 

Have a good day!

[James Agnew]

There is a setting on DaoConfig here: https://github.com/jamesagnew/hapi-fhir/blob/master/hapi-fhir-jpaserver-api/src/main/java/ca/uhn/fhir/jpa/api/config/DaoConfig.java#L1405

Cheers,

James

To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/b9c4db6a-fe6e-4ae3-94e4-0ab47a31faafn%40googlegroups.com.

[Brad C]

James,

We are clustering FHIR servers in our Production environment and wanted to get some more detail on your comment in this thread

>> There are many scheduled maintenance jobs that must be executed once and only once per scheduling period. The scheduler must be disabled on all nodes except one.

I couldn't find anything discussed in the docs regarding these scheduled maintenance jobs. I searched the codebase only came back with these two classes using the DaoConfig.isSchedulingDisabled() method:

https://github.com/hapifhir/hapi-fhir/blob/master/hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/search/reindex/ResourceReindexingSvcImpl.java

https://github.com/hapifhir/hapi-fhir/blob/master/hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/search/StaleSearchDeletingSvcImpl.java

I see how the ResourceReindexingSvcImpl.java is something that should only be run once. But I also discovered a separate implementation of isSchedulingDisabled() that doesn't use the DaoConfig. It uses a hardcoded boolean and an environment variable to determine the value in this class:

https://github.com/hapifhir/hapi-fhir/blob/master/hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/sched/BaseSchedulerServiceImpl.java

Is this separate implementation intentional? BaseSchedulerServiceImpl.java can be overridden, but I don't know enough to decide if we need to do this.

Thanks,

Brad

[Xiaocheng Luan]

Hi James,

In the case of deploying HAPI FHIR with the JPA server starter, I assume it (multiple instances pointing to one DB/cluster) won't work when Lucene is involved. Do you think it would work to have multiple instances pointing to the same DB instance/cluster and the same Elastic instance/cluster?

Thanks,

Xiaocheng

To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/CAKowd8krMGYsR8vVPPkCpz2-99edAOw4xorv9_40Wi4F1m15cQ%40mail.gmail.com.

[James Agnew]

Yes, ElasticSearch is the safer option in a cluster.

[James Agnew]

The one in BaseSchedulerService disables all task scheduling including tasks that need to run locally even on a cluster - It's intended only for the unit tests (where having stray scheduled jobs running can cause inconsistent results). The DaoConfig one is the one that should be disabled.

Cheers,

James

To view this discussion on the web visit https://groups.google.com/d/msgid/hapi-fhir/658d1839-2509-43f8-ac1d-ba6f46c10fb9n%40googlegroups.com.

[Xiaocheng Luan]

Thanks James. I'm trying to think through the potential issues before moving forward with the solution - so, again, in the specific setup of JPA server starter, database, and elasticsearch, does HAPI implement transactions between/across the database and the elasticsearch? In other words, could there be a brief moment that the database and elasticsearch are out of sync during update, which could cause some issues for the clustering? My outdated memory on elasticsearch was it only supports optimistic locking internally.

It looks like things would work well for applications that only need to interact with the database. Could you help clarify on the situation when Elasticsearch is involved? Or at a less ideal but still practical level, should such inconsistencies occur, e.g., would HAPI tolerate the issues and return less perfect results, or will it throw up?

Thanks,

Xiaocheng

[Jens Villadsen]

@james - AFAIK, the spring-batch jobs keep there state in the database - so it should be safe to have multiple app servers running in regards to the internal jobs as the app servers themselves are able to inspect whether a job has been run - either by the app server itself or one of its sibling app servers.

[pvnmur...@gmail.com]

Hi James,

We are using 5.3.0 HAPI FHIR servers in a clustered configuration. We are actually using multiple HAPI FHIR servers pointing to a single Database schema. Please confirm even with the latest FHIR server 5.3.0, do we need to enable scheduler only in one Fhir server node and disable scheduler on all other Fhir server nodes?  

We would prefer to leave the scheduler on on all nodes for maintenance reasons.

thanks & regards,

Narasimha Murthy

[Jens Villadsen]

@Narasimha - if you would like to have the scheduler turned on on all instances it would require the scheduling job to have its state persisted and an algorithm to make sure that two or more instances never starts the job if there is already one running. For such case I would recommend you to have a look at eg. https://github.com/lukas-krecan/ShedLock

[Maor Izenberg]

@Narasimha - were you able to run multiple replicas of JPA server and enabling/disabling the scheduling jobs of maintenance?

We would like to know where exactly inside Hapi server source code need to make the relevant modifications for example we found hapi-fhir/hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/reindex/job/ReindexJobConfig.java . We run over PostgreSQL DB.

Are there other places to change in code?

Did you use https://github.com/lukas-krecan/ShedLock solution or did you solve it in a different way?

[pa.f....@gmail.com]

Hi all,

I am evaluating HAPI FHIR JPA Starter in a clustered setup and would like to confirm that I understand the scheduling behavior correctly as the behaviour and code changed a lot since this thread was active.

Goal
I want to run HAPI in a cluster and control which nodes execute scheduled/background tasks.

What I found in the code (8.4.3)
From my current reading of the codebase, JpaStorageSettings#setSchedulingDisabled(boolean) (mapped to hapi.fhir.jpa.scheduling-disabled in the jpastarter project) does not appear to be a global scheduler switch.

What looks like the global scheduler disable mechanism seems to be the separate scheduling_disabled property/environment setting used in BaseSchedulerServiceImpl.

Also, JpaStorageSettings#isSchedulingDisabled() appears to be checked only in a limited number of task implementations. In the current code, I mainly found it in:

• StaleSearchDeletingSvcImpl
• ResourceReindexingSvcImpl

Many other scheduled/background jobs seem to be controlled independently, for example via their own feature flags such as isEnableTask..., and do not appear to depend on JpaStorageSettings.schedulingDisabled.

My current understanding of the runtime behavior
Based on the code, my current assumption is:

• If I call myStorageSettings.setSchedulingDisabled(true) at runtime, only those jobs that explicitly check this flag will effectively become no-op.
• If I set scheduling_disabled=true at startup, scheduler creation/bootstrapping is affected more globally.
• Changing scheduling_disabled at runtime in an already running JPA Starter process does not appear to be a reliable global on/off switch for all schedulers/jobs without restarting the node.

Questions regarding clustered production setups
For production cluster deployments, what is the recommended approach to control scheduler execution per node?

More specifically:

1. what should be set: is myStorageSettings.setSchedulingDisabled(true) already enough, or do i need to set scheduling_disabled=true as well/instead?
2. Is scheduling_disabled intended to be a startup-time setting only?
3. If I want one node to serve regular REST traffic but not execute scheduled/background jobs, what is the recommended approach?

Thanks in advance for any clarification.

Patrick

[pa.f....@gmail.com]

After reading this thread again i think i can answer this myself: JpaStorageSettings#isSchedulingDisabled() would be sufficient to set to false, as the other local jobs should be fine to run, no need do activate scheduling_disabled=true