Remote RMI authentication failed: all available authentication mechanisms failed

[Nguyễn Toàn]

Hi all,
I followed this example https://github.com/wildfly-security-incubator/elytron-examples/tree/main/jaas-realm to configure a custom realm, and it works find with http authentication.
But it does not work when I tried to use that custom realm for Remote RMI authentication. This is my configuration:
        <subsystem xmlns="urn:wildfly:elytron:18.0" final-providers="combined-providers" disallowed-providers="OracleUcrypto">
            <security-domains>
                <security-domain name="mySD" default-realm="myRealm" permission-mapper="default-permission-mapper">
                    <realm name="myRealm"/>
                </security-domain>
            </security-domains>
            <security-realms>
                <jaas-realm name="myRealm" entry="test" module="lm" callback-handler="loginmodules.CustomCallbackHandler">
                    <file path="d:/tools/elytron-examples/jaas-realm/custom-login-modules/JAAS-login-modules.conf"/>
                </jaas-realm>
            </security-realms>
            <http>
                <http-authentication-factory name="example-loginconfig-http-auth" security-domain="mySD" http-server-mechanism-factory="global">
                    <mechanism-configuration>
                        <mechanism mechanism-name="BASIC">
                            <mechanism-realm realm-name="FSRealmUsers"/>
                        </mechanism>
                    </mechanism-configuration>
                </http-authentication-factory>
                <provider-http-server-mechanism-factory name="global"/>
            </http>
            <sasl>
                <sasl-authentication-factory name="my-sasl-authentication" sasl-server-factory="configured" security-domain="mySD">
                    <mechanism-configuration>
                        <mechanism mechanism-name="DIGEST-MD5">
                            <mechanism-realm realm-name="myRealm"/>
                        </mechanism>
                    </mechanism-configuration>
                </sasl-authentication-factory>
                <configurable-sasl-server-factory name="configured" sasl-server-factory="elytron">
                    <properties>
                        <property name="wildfly.sasl.local-user.default-user" value="$local"/>
                        <property name="wildfly.sasl.local-user.challenge-path" value="${jboss.server.temp.dir}/auth"/>
                    </properties>
                </configurable-sasl-server-factory>
                <provider-sasl-server-factory name="global"/>
            </sasl>
        </subsystem>

        <subsystem xmlns="urn:jboss:domain:remoting:6.0">
            <endpoint worker="remoting"/>
            <connector name="remoting-connector" socket-binding="remoting" sasl-authentication-factory="my-sasl-authentication">
                <properties>
                    <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="30000" />
                </properties>
            </connector>
        </subsystem>

My client trys to connect to remote RMI with below example infomation:
{server=192.168.1.211, port=4447, java.naming.security.principal=user1,
java.naming.factory.initial=org.wildfly.naming.client.WildFlyInitialContextFactory,
 java.naming.provider.url=remote://192.168.1.211:4447, java.naming.security.credentials=passwordUser1}
And it throw the error as below:
Caused by: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: javax.security.sasl.SaslException: DIGEST-MD5: Server rejected authentication
at org.jboss.remoting3.remote.ClientConnectionOpenListener.allMechanismsFailed(ClientConnectionOpenListener.java:109) ~[wildfly-client-all.jar:31.0.1.Final]
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:268) ~[wildfly-client-all.jar:31.0.1.Final]
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:244) ~[wildfly-client-all.jar:31.0.1.Final]
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) ~[wildfly-client-all.jar:31.0.1.Final]
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66) ~[wildfly-client-all.jar:31.0.1.Final]
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89) ~[wildfly-client-all.jar:31.0.1.Final]
at org.xnio.nio.WorkerThread.run(WorkerThread.java:603) ~[wildfly-client-all.jar:31.0.1.Final]
 
On the server, it does not reach the class login module: CustomLoginModule1 or CustomLoginModule2, and it does not have any error log on the server.
What configuration should I change to make the authentication work wirh remote RMI?

Thanks,
Toan.