Wildfly 26 -> 31 DB down prevents deployment

[D E]

We are moving from Wildfly 26.1.3.Final to Wildfly 31.0.1.Final.

Sometimes we need to start/deploy our software while the DB is down and then when the DB comes up later, our app just starts working.

On Wildfly 26.1.3.Final that was possible.  However, on Wildfly 31.0.1.Final, our deployments fail so when the DB comes up some manual action must be taken.

Obviously we have moved from the javax to the jakarta namespace during this conversion.

And both javax and jakarta versions of my app work fine when the DB is up and running.

Here are the specifics for 26.1.3.Final:

$ cat wildfly-26.1.3.Final/modules/com/oracle/jdbc/main/module.xml

<?xml version='1.0' encoding='UTF-8'?>
<module name='com.oracle.jdbc' xmlns='urn:jboss:module:1.9'>
  <resources>
    <resource-root path='ojdbc8-19.21.0.0.jar' />
  </resources>
  <dependencies>
    <module name='java.se' />
    <module name='javax.transaction.api' />
  </dependencies>
</module>

$ cat META-INF/persistence.xml

<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
  <persistence-unit name="oracledb">
    <jta-data-source>java:jboss/datasources/OracleDS</jta-data-source>
    <properties>
      <property name="hibernate.dialect" value="org.hibernate.dialect.Oracle12cDialect"/>
      <property name="jboss.entity.manager.factory.jndi.name" value="java:/OracleEntityManagerFactory"/>
    </properties>
  </persistence-unit>
</persistence>


Here are the specifics for 31.0.1.Final:

$ cat wildfly-31.0.1.Final/modules/com/oracle/jdbc/main/module.xml

<?xml version='1.0' encoding='UTF-8'?>
<module name='com.oracle.jdbc' xmlns='urn:jboss:module:1.9'>
  <resources>
    <resource-root path='ojdbc8-19.21.0.0.jar' />
  </resources>
  <dependencies>
    <module name='java.se' />
    <module name='jakarta.transaction.api' />
  </dependencies>
</module>

$ cat META-INF/persistence.xml
<?xml version="1.0" encoding="UTF-8"?>
<persistence xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd">
  <persistence-unit name="oracledb">
    <jta-data-source>java:jboss/datasources/OracleDS</jta-data-source>
    <properties>
      <property name="jboss.entity.manager.factory.jndi.name" value="java:/OracleEntityManagerFactory"/>
    </properties>
  </persistence-unit>
</persistence>


This "cli" update is common to both 26.1.3.Final and 31.0.1.Final:

embed-server --server-config=standalone.xml

/subsystem=datasources/jdbc-driver=oracle:add( \
  driver-module-name=com.oracle.jdbc, \
  driver-name=oracle \
)

data-source add \
  --name=OracleDS \
  --jndi-name=java:jboss/datasources/OracleDS \
  --driver-name=oracle \
  --connection-url=jdbc:oracle:thin:@xxxx:1521/xxxx \
  --url-delimiter=| \
  --user-name="xxxx" \
  --password="yyyy" \
  --exception-sorter-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter \
  --stale-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleStaleConnectionChecker \
  --valid-connection-checker-class-name=org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker \
  --max-pool-size=128 \
  --min-pool-size=24 \
  --blocking-timeout-wait-millis=5000 \
  --idle-timeout-minutes=5 \
  --query-timeout=20 \
  --enabled=true

stop-embedded-server

And here is the result of the deployments for each version (26.1.3.Final=OK, 31.0.1.Final=FAILED):

$ ./wildfly-26.1.3.Final/bin/jboss-cli.sh -c deployment-info
NAME                         RUNTIME-NAME                 PERSISTENT ENABLED STATUS
sso-xxxx-dashboard-javax.war sso-xxxx-dashboard-javax.war false      true    OK    

$ ./wildfly-31.0.1.Final/bin/jboss-cli.sh -c deployment-info
NAME                           RUNTIME-NAME                   PERSISTENT ENABLED STATUS
sso-xxxx-dashboard-jakarta.war sso-xxxx-dashboard-jakarta.war false      true    FAILED 

[Scott Marlow]

Each persistence unit defined in the persistence.xml has an (WildFly) internal PersistenceUnitService that depends on other internal services like a DataSource service that is used to obtain database connections.

Are you seeing any deployment failures in the server.log when deploying the application with the DB down?  Could you please show us the deployment failures that you see with WildFly 26 + 31 (if any).  From the jboss-cli.sh command output below I don't expect to see deployment failures for WildFly 26 but would be good to know for sure.

Scott

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/0970f2f1-ae62-4353-9795-71939d2448cen%40googlegroups.com.

[Everly, David W]

There are no deployment failures for 26, but there are for 31 (as you see from the cli status provided:

$ ./wildfly-31.0.1.Final/bin/jboss-cli.sh -c deployment-info
NAME                           RUNTIME-NAME                   PERSISTENT ENABLED STATUS
sso-xxxx-dashboard-jakarta.war sso-xxxx-dashboard-jakarta.war false      true    FAILED

I think this is the relevant sever.log entry from 31 that you are requesting (of course it cannot determine the dialect because it cannot connect to the DB):

2024-04-24 09:25:15,856 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 45) WFLYSRV0010: Deployed "sso-xxxx-dashboard-jakarta.war" (runtime-name : "sso-xxxx-dashboard-jakarta.war")
2024-04-24 09:25:15,889 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186:   Services which failed to start:      service jboss.persistenceunit."sso-xxxx-dashboard-jakarta.war#oracledb".__FIRST_PHASE__: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to determine Dialect without JDBC metadata (please set 'jakarta.persistence.jdbc.url' for common cases or 'hibernate.dialect' when a custom Dialect implementation must be provided)
WFLYCTL0448: 34 additional services are down due to their dependencies being missing or failed

[Scott Marlow]

On 4/24/24 12:31 PM, Everly, David W wrote:

There are no deployment failures for 26, but there are for 31 (as you see from the cli status provided:

$ ./wildfly-31.0.1.Final/bin/jboss-cli.sh -c deployment-info
NAME                           RUNTIME-NAME                   PERSISTENT ENABLED STATUS
sso-xxxx-dashboard-jakarta.war sso-xxxx-dashboard-jakarta.war false      true    FAILED

I think this is the relevant sever.log entry from 31 that you are requesting (of course it cannot determine the dialect because it cannot connect to the DB):

2024-04-24 09:25:15,856 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 45) WFLYSRV0010: Deployed "sso-xxxx-dashboard-jakarta.war" (runtime-name : "sso-xxxx-dashboard-jakarta.war")
2024-04-24 09:25:15,889 INFO  [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report
WFLYCTL0186:   Services which failed to start:      service jboss.persistenceunit."sso-xxxx-dashboard-jakarta.war#oracledb".__FIRST_PHASE__: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to determine Dialect without JDBC metadata (please set 'jakarta.persistence.jdbc.url' for common cases or 'hibernate.dialect' when a custom Dialect implementation must be provided)
WFLYCTL0448: 34 additional services are down due to their dependencies being missing or failed

The "Unable to determine Dialect without JDBC metadata" failure is Hibernate ORM attempting to get a connection to the database to obtain information about the database server so that it can determine which Hibernate database dialect to use.  So as per https://docs.jboss.org/hibernate/orm/6.4/dialect/dialect.html you should try adding:

<property name="hibernate.dialect" value="OracleDialect"/>

I think "OracleDialect" is a shorter form of the org.hibernate.dialect.OracleDialect name.  For more information about the older Hibernate Dialects (like org.hibernate.dialect.Oracle12cDialect), see https://in.relation.to/2023/02/15/hibernate-orm-62-db-version-support/

Scott

[Everly, David W]

This change caused the following error:

11:48:56,363 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "sso-xxxx-dashboard-jakarta.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.persistenceunit.\"sso-xxxx-dashboard-jakarta.war#oracledb\".__FIRST_PHASE__" => "org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.boot.registry.selector.spi.StrategySelectionException: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.boot.registry.classloading.spi.ClassLoadingException: Unable to load class [OracleDialect]
    Caused by: java.lang.ClassNotFoundException: Could not load requested class : OracleDialect
    Caused by: java.lang.Throwable"}}

So I tried this instead:

<property name="hibernate.dialect" value="org.hibernate.dialect.OracleDialect"/>

This gave the following warning, which made us assume it shouldn't be used:

11:55:24,467 WARN  [org.hibernate.orm.deprecation] (ServerService Thread Pool -- 78) HHH90000025: OracleDialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)

However, the deployment succeeded:

$ ./wildfly-31.0.1.Final/bin/jboss-cli.sh -c deployment-info
NAME                           RUNTIME-NAME                   PERSISTENT ENABLED STATUS

sso-xxxx-dashboard-jakarta.war sso-xxxx-dashboard-jakarta.war false      true    OK    

So presumably we should go back to that setting and ignore the warning? 

[Scott Marlow]

On 4/24/24 1:58 PM, Everly, David W wrote:

This change caused the following error:

11:48:56,363 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "sso-xxxx-dashboard-jakarta.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.persistenceunit.\"sso-xxxx-dashboard-jakarta.war#oracledb\".__FIRST_PHASE__" => "org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.boot.registry.selector.spi.StrategySelectionException: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.boot.registry.classloading.spi.ClassLoadingException: Unable to load class [OracleDialect]
    Caused by: java.lang.ClassNotFoundException: Could not load requested class : OracleDialect
    Caused by: java.lang.Throwable"}}

So I tried this instead:

<property name="hibernate.dialect" value="org.hibernate.dialect.OracleDialect"/>

This gave the following warning, which made us assume it shouldn't be used:

11:55:24,467 WARN  [org.hibernate.orm.deprecation] (ServerService Thread Pool -- 78) HHH90000025: OracleDialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)

However, the deployment succeeded:

$ ./wildfly-31.0.1.Final/bin/jboss-cli.sh -c deployment-info
NAME                           RUNTIME-NAME                   PERSISTENT ENABLED STATUS
sso-xxxx-dashboard-jakarta.war sso-xxxx-dashboard-jakarta.war false      true    OK    

So presumably we should go back to that setting and ignore the warning?

Yes, please do ignore the warning.  I think the warning is wrong when the database server is not running as clearly that case will not work.  The warning was added via the https://github.com/hibernate/hibernate-orm/commit/e5545492cc7#diff-0bcfaa7e5bdafe5e7631620b4e8596f4bbdb7cc5a4385c43e01850686034982dR225 change.  If you have ideas for improving the warning, please consider creating a tracking https://hibernate.atlassian.net/browse/HHH issue and optionally a pull request.

[Everly, David W]

Thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/6be8b2ba-cc39-4a1b-802c-a0d3df14f65d%40redhat.com.

[Scott Marlow]

I thought about this some more and think we should definitely create a HHH jira.  Do you plan to do that or should I?  See more input inline below.

On Wed, Apr 24, 2024 at 3:07 PM Everly, David W <david....@verizon.com> wrote:

Thanks!

On Wed, Apr 24, 2024 at 12:30 PM Scott Marlow <sma...@redhat.com> wrote:

On 4/24/24 1:58 PM, Everly, David W wrote:

This change caused the following error:

11:48:56,363 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("deploy") failed - address: ([("deployment" => "sso-xxxx-dashboard-jakarta.war")]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.persistenceunit.\"sso-xxxx-dashboard-jakarta.war#oracledb\".__FIRST_PHASE__" => "org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment] due to: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.boot.registry.selector.spi.StrategySelectionException: Unable to resolve name [OracleDialect] as strategy [org.hibernate.dialect.Dialect]
    Caused by: org.hibernate.boot.registry.classloading.spi.ClassLoadingException: Unable to load class [OracleDialect]

I am thinking that others that are migrating from earlier WildFly versions will also see a similar "ClassNotFoundException: org.hibernate.dialect.Oracle12cDialect" (or whatever dialect they were previously using) and they will also remove the "hibernate.dialect" property (or update it).  For those that remove that property, they will likely experience the same problem that they start the WildFly app server with the database server not running and get the deployment failure that you reported.  Once they do specify a "hibernate.dialect" that works, they will see the confusing warning that looks like "HHH90000025: OracleDialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)".

I'm thinking that the warning should at least be changed or removed.  

So, are you interested in reporting this as a https://hibernate.atlassian.net/browse/HHH issue or should I?

Thanks,

Scott

[Everly, David W]

Please report it if you think it is worthy, thanks!

[Everly, David W]

By the way, my personal upgrade journey was this:

1. org.hibernate.dialect.Oracle12cDialect not found
   
2. in response to #1 I researched and changed it to org.hibernate.dialect.OracleDialect
   

1. HHH90000025: OracleDialect does not need to be specified explicitly using 'hibernate.dialect' (remove the property setting and it will be selected by default)
   

1. in response to #3 I just removed it to find no errors or warnings
2. sometime later the corner case regarding starts when DB is down was noticed

[Scott Marlow]

Just created https://hibernate.atlassian.net/browse/HHH-18076 issue, please add comments or feedback on there.  Thanks for reporting this!