[WildFly 18.0.1] Could not lookup datasource java:jboss/datasources/JGroups

[Victor Mendonca]

Hi all,

One of our environments failed to start due to not being able to read 'jgroups-ds.xml'. While a restart fixed the problem, we are not sure why this happened in the first place (seeing how this configuration is deployed across multiple environments and no changes were made to the file).

Any ideas on why this could happen?

18:21:57,691 WARN [org.jboss.as.connector] (MSC service thread 1-7) WFLYJCA0091: -ds.xml file deployments are deprecated. Support may be removed in a future version.
18:21:57,957 INFO [org.jboss.weld.deployer] (MSC service thread 1-3) WFLYWELD0003: Processing weld deployment jgroups-jdbc-manager-web.war
18:21:58,458 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 62) MSC000001: Failed to start service org.wildfly.clustering.jgroups.channel.ee: org.jboss.msc.service.StartException in service
org.wildfly.clustering.jgroups.channel.ee: java.lang.IllegalStateException: java.lang.IllegalArgumentException: Could not lookup datasource java:jboss/datasources/JGroups
at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70)
at org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.IllegalStateException: java.lang.IllegalArgumentException: Could not lookup datasource java:jboss/datasources/JGroups

Thanks,

Victor.

[Victor Mendonca]

Hi all,

Replying back to this... this intermittent issue doesn't seem to want to go away. It happens sporadically and a second attempt to start WildFly is usually successful.

I've done more research and I've found a couple of posts advising to refer to the datasource by name and not by the JNDI name. Would that cause intermittent issues with the startup?

Our vendor currently provides us the following configuration:

jgroups-ds.xml

<?xml version="1.0" encoding="UTF-8"?>
<datasources xmlns="http://www.jboss.org/ironjacamar/schema"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.jboss.org/ironjacamar/schema http://docs.jboss.org/ironjacamar/schema/datasources_1_0.xsd">
    <datasource jndi-name="java:jboss/datasources/JGroups" pool-name="jgroups-connection-pool">
        <connection-url>jdbc:sqlserver://[DB URL];DatabaseName=[DB Name]</connection-url>

vendor-ha.xml

<server xmlns="urn:jboss:domain:10.0">

    [ truncated ]

    <system-properties>

    [ truncated ]

        <property name="com.vendor.fss.jgroups.datasource" value="java:jboss/datasources/JGroups"/>

    </system-properties>

    [ truncated ]

        <subsystem xmlns="urn:jboss:domain:jgroups:7.0">

            <channels default="ee">

                <channel name="ee" stack="tcp" cluster="ejb"/>

            </channels>

            <stacks>

            [ truncated ]

                <stack name="tcp">

                    <transport type="TCP" socket-binding="jgroups-tcp"/>

                    <protocol type="org.jgroups.protocols.JDBC_PING">

                        <property name="datasource_jndi_name">${com.vendor.fss.jgroups.datasource}</property>

                    </protocol>

                    <protocol type="MERGE3"/>

                    <socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                    <protocol type="FD_ALL"/>

                    <protocol type="VERIFY_SUSPECT"/>

                    <protocol type="pbcast.NAKACK2"/>

                    <protocol type="UNICAST3"/>

                    <protocol type="pbcast.STABLE"/>

                    <protocol type="pbcast.GMS"/>

                    <protocol type="MFC"/>

                    <protocol type="FRAG3"/>

                </stack>

            </stacks>

        </subsystem>

        [ truncated ]

    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

    [ truncated ]

        <socket-binding name="jgroups-tcp-fd" interface="external" port="57600"/>

Should the recommended config be as:

                    <protocol type="JDBC_PING" data-source="jgroups-connection-pool"/>

Thanks for any help!

Victor.

[Emmanuel Hugonnet]

Hello,
I think that the issue is that jgroups subsystem doesn't have a clear dependency on the datasource so it might not be ready when jgroups
tries to use it. But I'm not a clustering/jgroups expert.
Emmanuel

Le 27/04/2021 à 02:36, Victor Mendonca a écrit :
> Hi all,
>
> Replying back to this... this intermittent issue doesn't seem to want to go away. It happens sporadically and a second attempt to start
> WildFly is usually successful.
>
> I've done more research and I've found a couple of posts advising to refer to the datasource by name and not by the JNDI name. Would that
> cause intermittent issues with the startup?
>
> Our vendor currently provides us the following configuration:
>

> *jgroups-ds.xml*

>
> <?xml version="1.0" encoding="UTF-8"?>
> <datasources xmlns="http://www.jboss.org/ironjacamar/schema"
>         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>         xsi:schemaLocation="http://www.jboss.org/ironjacamar/schema http://docs.jboss.org/ironjacamar/schema/datasources_1_0.xsd">
>     <datasource jndi-name="java:jboss/datasources/JGroups" pool-name="jgroups-connection-pool">
>         <connection-url>jdbc:sqlserver://[DB URL];DatabaseName=[DB Name]</connection-url>
>
>

> *vendor-ha.xml*
>
> //

> org.wildfly.clustering.jgroups.channel.ee <http://org.wildfly.clustering.jgroups.channel.ee>: org.jboss.msc.service.StartException in
> service
> org.wildfly.clustering.jgroups.channel.ee <http://org.wildfly.clustering.jgroups.channel.ee>: java.lang.IllegalStateException:

> java.lang.IllegalArgumentException: Could not lookup datasource java:jboss/datasources/JGroups
> at org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70)
> at org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117)
> at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
> at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
> at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
> at java.lang.Thread.run(Thread.java:748)
> at org.jboss.threads.JBossThread.run(JBossThread.java:485)
> Caused by: java.lang.IllegalStateException: java.lang.IllegalArgumentException: Could not lookup datasource java:jboss/datasources/JGroups
>
>
> Thanks,
> Victor.
>

> --
> You received this message because you are subscribed to the Google Groups "WildFly" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com
> <mailto:wildfly+u...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/wildfly/f317194e-7bf6-40ed-95ca-4bac444fa117n%40googlegroups.com
> <https://groups.google.com/d/msgid/wildfly/f317194e-7bf6-40ed-95ca-4bac444fa117n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[dt pham]

Let use the pool name of the datasource as data-source for the jdbc protocol to see if it helps.

<jdbc-protocol type="JDBC_PING" data-source=" jgroups-connection-pool  "/>

[Victor Mendonca]

Thank you all. I will contact the vendor and also test the changes. I'll reply back here with the outcome.

Cheers,

Victor

[dt pham]

your stack may look like this if my previous msg not clear enough:

<stack name="tcp>

<transport type="TCP" socket-binding="jgroups-tcp"/>

<jdbc-protocol type="JDBC_PING" data-source="jgroups-connection-pool"/>

<protocol type="MERGE3"/>

<socket-protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

<protocol type="FD_ALL"/>

<protocol type="VERIFY_SUSPECT"/>

<protocol type="pbcast.NAKACK2"/>

<protocol type="UNICAST3"/>

<protocol type="pbcast.STABLE"/>

<protocol type="pbcast.GMS"/>

<protocol type="UFC"/>

<protocol type="FRAG3"/>

</stack>

[Victor Mendonca]

Hello everyone,

I finally got a reply back from the vendor and approval to test this, and after making the changes I'm getting the following error:

2021-07-05 17:32:26,911 ERROR [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0362: Capabilities required by resource '/subsystem=jgroups/stack=tcp/protocol=JDBC_PING' are not available:
    org.wildfly.data-source.jgroups-connection-pool; Possible registration points for this capability:
        /subsystem=datasources/data-source=*
        /subsystem=datasources/xa-data-source=*

2021-07-05 17:32:26,936 FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
2021-07-05 17:32:26,964 INFO  [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: WildFly Full 18.0.1.Final (WildFly Core 10.0.3.Final) stopped in 21ms

I have tried the two configs below, and both produce the same result:

Config1:

<stack name="tcp">
    <transport type="TCP" socket-binding="jgroups-tcp"/>

    <jdbc-protocol type="JDBC_PING" data-source="jgroups-connection-pool" />

    <protocol type="MERGE3"/>

Config2:

<stack name="tcp">
    <transport type="TCP" socket-binding="jgroups-tcp">

        <property name="datasource_jndi_name">${com.vendor.fss.jgroups.datasource}</property>

    </transport>
    <jdbc-protocol type="org.jgroups.protocols.JDBC_PING" data-source="jgroups-connection-pool">
    <protocol type="MERGE3"/>

Any help is again appreciated.

Thanks,

Victor.

[Victor Mendonca]

I'm having fun and learning new things.

From looking at the configuration the vendor is using 'sqljdbc' as a module and has a dependency defined in 'jboss-deployment-structure.xml' inside WEB-INF. The datasource is configured in a separate 'jgroups-ds.xml' file, which I think it's part of the problem. Searching online it looks like the use of '*-ds.xml' files is deprecated.

When I add the datasource inside 'vendor-ha.xml' and remove 'jgroups-ds.xml' I'm able to start WildFly, but one of their custom deployments fails (I'm assuming because of how they are calling the datasource, which I modified):

2021-07-07 15:50:17,326 ERROR [com.vendor.fss.jgroupsjdbc.web.startup.DuplicateEntryRemover] (pool-14-thread-1) Error removing duplicate JGroups entries.: org.jboss.weld.exceptions.WeldException: WELD-000049: Unable to invoke public void
com.vendor.fss.jgroupsjdbc.web.service.helper.JGroupsManagerFactoryImpl.init() on com.vendor.fss.jgroupsjdbc.web.service.helper.JGroupsManagerFactoryImpl@23eb74c2

I'm gonna go back to the vendor and share my findings.

Let me know if anyone happens to see something I may have missed.

Thanks!

Victor.

[Paul Ferraro]

It is inadvisable to create server configuration that depends on a deployment resource.

Server resources are intended to be shared by multiple deployments.

Instead, you'll want to define your jgroups-connection-pool resource via the datasoures subsystem and reference it in your JDBC_PING by its name (like you did in Config 1).