I made it work by doing the below, please let me know whether this is the right way to do it or not?
in the standalone.xml:
under wildfly elytron subsystem:
added new security domain:
<security-domain name="mySD" default-realm="myRealm" permission-mapper="default-permission-mapper">
<realm name="myRealm"/>
</security-domain>
added new jaas-realm:
<jaas-realm name="myRealm" entry="test" module="com.broadcom.itpam.customauth.custom-auth" callback-handler="com.broadcom.itpam.custommodule.CustomCallbackHandler">
<file path="C:\ITPAM_WildFly_POC\customAuth\JAAS-login-modules.conf"/>
</jaas-realm>
Added new http-authentication-factory & corresponding service-loader-http-server-mechanism-factory:
<http-authentication-factory name="custom-mechanism" security-domain="mySD" http-server-mechanism-factory="custom-factory">
<mechanism-configuration>
<mechanism mechanism-name="CUSTOM_MECHANISM">
<mechanism-realm realm-name="myRealm"/>
</mechanism>
</mechanism-configuration>
</http-authentication-factory>
<service-loader-http-server-mechanism-factory name="custom-factory" module="com.broadcom.itpam.customauth.custom-http-mechanism"/>
in the undertow subsystem:
replaced
<application-security-domains>
<application-security-domain name="other" security-domain="ApplicationDomain" />
</application-security-domains>
With
<application-security-domains>
<application-security-domain name="other" http-authentication-factory="custom-mechanism" override-deployment-config="true"/>
</application-security-domains>
And implemented two separate modules one for the JAAS-login-modules other for the custom-http-mechanism,
In a nutshell I combined both custom `HttpServerAuthenticationMechanism` and a custom `LoginModule` by following below two: