Problem with IBM HTTP Server with WebSphere Liberty and proposed solution

[Bryan Dixon]

We were having sporadic issues when fronting our WebSphere Liberty environment with the IBM HTTP Server.  The error that we would sometimes receive is:
waffle.servlet.NegotiateSecurityFilter - error logging in user: The token supplied to the function is invalid

When we issued our http requests directly to WebSphere Liberty Waffle always worked great and we never had this issue occur - it was only a problem when we fronted Liberty with IBM HTTP Server (our standard test and production architecture).

What I noticed is that when we received this error the connection id Waffle was using different during the negotiation process.  Notice port 50005 and then port 50009 from these log entries:

2014-07-07 16:08:03.394 DEBUG Default Executor-thread-7 waffle.servlet.spi.NegotiateSecurityFilterProvider - security package: NTLM, connection id: HO-DIXONB4.simon.com:50005
2014-07-07 16:08:03.394 DEBUG Default Executor-thread-7 waffle.servlet.spi.NegotiateSecurityFilterProvider - token buffer: 55 byte(s)
2014-07-07 16:08:03.402 DEBUG Default Executor-thread-7 waffle.servlet.spi.NegotiateSecurityFilterProvider - continue token: TlRMTVNTUAACAAAACgAKADgAAAAFwomiMVpSUBCjh6XQYP8BAAAAAI4AjgBCAAAABgGxHQAAAA9TAEkATQBPAE4AAgAKAFMASQBNAE8ATgABABQASABPAC0ARABJAFgATwBOAEIANAAEABIAcwBpAG0AbwBuAC4AYwBvAG0AAwAoAEgATwAtAEQASQBYAE8ATgBCADQALgBzAGkAbQBvAG4ALgBjAG8AbQAFABIAcwBpAG0AbwBuAC4AYwBvAG0ABwAIAEuCqygfms8BAAAAAA==
2014-07-07 16:08:03.404 DEBUG Default Executor-thread-7 waffle.servlet.spi.NegotiateSecurityFilterProvider - continue required: true
2014-07-07 16:08:03.413 DEBUG Default Executor-thread-7 waffle.servlet.NegotiateSecurityFilter - GET /ppm/app/connectlink, contentlength: -1
2014-07-07 16:08:03.414 DEBUG Default Executor-thread-7 waffle.servlet.spi.NegotiateSecurityFilterProvider - security package: NTLM, connection id: HO-DIXONB4.simon.com:50009
2014-07-07 16:08:03.415 DEBUG Default Executor-thread-7 waffle.servlet.spi.NegotiateSecurityFilterProvider - token buffer: 88 byte(s)
2014-07-07 16:08:03.439 WARN Default Executor-thread-7 waffle.servlet.NegotiateSecurityFilter - error logging in user: The token supplied to the function is invalid

Whenever the connection id was the same during the process, we did not receive this error.  I checked the Waffle source code that generated the connection id (class waffle.util.NtlmServletRequest) and created a local patch to try and return a unique connection id for the client (browser request) because I found the IBM plugin was occasionally using a different socket connection.   Anyway I changed the following method that generates the connection id to use the id of the HttpSession for the request and appears to have solved the issue:

public static String getConnectionId(HttpServletRequest request) {
        return request.getSession().getId();
}

I was curious if this was a good solution or if I might run into an issue down the road with this change.

Thanks

[Daniel Doubrovkine]

Imagine you have two negotiations happening in parallel because two HTTP requests were made simultaneously from the same machine (easy to do in JavaScript). This would immediate break, right?

If your application doesn't do this, maybe you can get away with it. But a better fix would be for the server to not reopen another connection and keep it alive.

--
You received this message because you are subscribed to the Google Groups "waffle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to waffle-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

dB. | Moscow - Geneva - Seattle - New York
code.dblock.org - @dblockdotorg - artsy.net - github/dblock

[Bryan Dixon]

Thanks for the reply.   We opened a ticket with IBM to make sure we had HTTP Keep alive configured correctly with their products (IBM HTTP Server and WebSphere Liberty) and they said we did and it was working properly - even though we were getting different ports occasionally.  So unfortunately we can't do anything to change the server to not use a different connection.  If it were up to me we wouldn't be using any IBM products but I digress.

I understand your scenario about simultaneaous HTTP requests causing an issue when using the HTTP Session Id.  I don't think its an issue with our app (currently - but could be in the future).  I think I might look at doing a patch/solution like Jespa seems to deal with situations like this with certain web server/app server environments.    They have created their own Request Header named 'Jespa-Connection-Id' that can be used instead of the remote host/addr and port headers.  Their doc states you can set that custom header in the apache config like this:

Location /jespa/>
ProxyPass ajp://localhost:8009/jespa/
ProxyPassReverse ajp://localhost:8009/jespa/
RewriteEngine On
RewriteRule .* - [E=INFO_REMOTE_ADDR:%{REMOTE_ADDR},NE]
RewriteRule .* - [E=INFO_REMOTE_PORT:%{REMOTE_PORT},NE]
RequestHeader set Jespa-Connection-Id "%{INFO_REMOTE_ADDR}e:%{INFO_REMOTE_PORT}e"
</Location>

Is this something (custom header) that might work for the Waffle product?  The NtlmServletRequest could be changed to return the custom header value if it exists - if not, it just does what it is currently doing.

Thanks
Bryan

[Daniel Doubrovkine]

I hope that server behind the /jespa endpoint isn't on the live internet by itself, cause I'll just send it some fun Jespa-Connection-Id values :)

Other than that something like this could be useful for waffle. I'd be happy to see a pull request. Generally I think the piece that generates a unique key for the session could be abstracted and configured externally, with Spring you could just load a different strategy, for example.

--
You received this message because you are subscribed to the Google Groups "waffle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to waffle-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[PALASH KAR]

HI Daniel,

I have used maven plugin for waffle 1.6 as below and in web.xml as below, it works in local windows jetty  and when we try to deploy in WebSphere as EAR in AIX (UNIX) machine then i get below error, Any Solution would help.

<filter>

        <filter-name>SecurityFilter</filter-name>

        <filter-class>waffle.servlet.NegotiateSecurityFilter</filter-class>

    </filter>

    <filter-mapping>

        <filter-name>SecurityFilter</filter-name>

        <url-pattern>/*</url-pattern>

    </filter-mapping>

<dependency>

          <groupId>com.github.dblock.waffle</groupId>

          <artifactId>waffle-jna</artifactId>

          <version>1.6</version>

      </dependency>

[8/6/14 11:52:51:021 PDT] 0000065d webapp        E com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[accumsuinw]: java.lang.NoClassDefFoundError: com.sun.jna.platform.win32.Sspi$CredHandle (initialization failure)
	at java.lang.J9VMInternals.initialize(J9VMInternals.java:176)
	at waffle.windows.auth.impl.WindowsCredentialsHandleImpl.initialize(WindowsCredentialsHandleImpl.java:75)
	at waffle.windows.auth.impl.WindowsAuthProviderImpl.acceptSecurityToken(WindowsAuthProviderImpl.java:82)
	at waffle.servlet.spi.NegotiateSecurityFilterProvider.doFilter(NegotiateSecurityFilterProvider.java:102)
	at waffle.servlet.spi.SecurityFilterProviderCollection.doFilter(SecurityFilterProviderCollection.java:116)
	at waffle.servlet.NegotiateSecurityFilter.doFilter(NegotiateSecurityFilter.java:99)
	at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
	at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
	at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:928)
	at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1025)
	at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3761)
	at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
	at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:975)
	at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
	at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:195)
	at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
	at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
	at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
	at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:87)
	at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
	at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
	at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
	at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
	at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
	at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
	at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
	at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1815)
Caused by: java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/aix-ppc64/libjnidispatch.a) not found in resource path (/apps/WebSphere/AppServer-8.5/profiles/ManagedNW_DIT/properties:/apps/WebSphere/AppServer-8.5/properties:/apps/WebSphere/AppServer-8.5/lib/startup.jar:/apps/WebSphere/AppServer-8.5/lib/bootstrap.jar:/apps/WebSphere/AppServer-8.5/lib/jsf-nls.jar:/apps/WebSphere/AppServer-8.5/lib/lmproxy.jar:/apps/WebSphere/AppServer-8.5/lib/urlprotocols.jar:/apps/WebSphere/AppServer-8.5/deploytool/itp/batchboot.jar:/apps/WebSphere/AppServer-8.5/deploytool/itp/batch2.jar:/apps/WebSphere/AppServer-8.5/java/lib/tools.jar)
	at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:776)
	at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736)
	at com.sun.jna.Native.(Native.java:131)
	at java.lang.J9VMInternals.initializeImpl(Native Method)
	at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
	at com.sun.jna.Pointer.(Pointer.java:42)
	at java.lang.J9VMInternals.initializeImpl(Native Method)
	at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
	at java.lang.J9VMInternals.initialize(J9VMInternals.java:203)
	at com.sun.jna.Structure.(Structure.java:1933)
	at java.lang.J9VMInternals.initializeImpl(Native Method)
	at java.lang.J9VMInternals.initialize(J9VMInternals.java:236)
	at java.lang.J9VMInternals.initialize(J9VMInternals.java:203)
	at java.lang.J9VMInternals.initialize(J9VMInternals.java:203)
	... 26 more

Thanks,

Palash

[Daniel Doubrovkine]

This looks like you don't have JNA's jna.jar and platform.jar in the right place where the server can load it from.

[Daniel Doubrovkine]

Oh wait, this is an AIX server? It's not going to work. Waffle only runs on Windows, see https://github.com/dblock/waffle/blob/master/Docs/faq/DoesWaffleWorkOnLinux.md

[PALASH KAR]

Thank you Daniel.

I guess the same but thought of asking.

Regards,

Palash

[Daniel Doubrovkine]

Let us know how you fixed it, could be an FAQ. Generally I'd like instructions of how to set this up the same as what we have for other web servers - consider contributing!