Hello,
We have been experiencing intermittent lockups recently. I wonder if anyone had similar problem. Below please find stack traces for couple of stuck threads.
We are using November 2005 jLDAP build against Netware 6.0 LDAP
If I remember correctly we share the same connection to read LDAP data from multiple threads. I recall it was specified to be thread sahe
I would appreciate any suggestions
Thanks
Alex Roytman
"TP-Processor33" daemon prio=1 tid=0x086cc720 nid=0x1b7f in Object.wait() [0x67cfe000..0x67cff6f0]
at java.lang.Object.wait(Native Method)
- waiting on <0x71f8efb0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:474)
at com.novell.ldap.Connection.acquireWriteSemaphore(Unknown Source)
- locked <0x71f8efb0> (a java.lang.Object)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Message.sendMessage(Unknown Source)
at com.novell.ldap.MessageAgent.sendMessage(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.search(LdapPrincipalBuilder.java:176)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.findPrincipal(LdapPrincipalBuilder.java:100)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.provide(LdapPrincipalBuilder.java:51)
at com.peacetech.tomcat.auth.PrincipalBuilderRealm.provide(PrincipalBuilderRealm.java:113)
at com.peacetech.tomcat.auth.PrincipalBuilderRealm.authenticate(PrincipalBuilderRealm.java:36)
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:256)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:416)
at com.peacetech.tomcat.auth.RSANextTokenValve.invoke(RSANextTokenValve.java:185)
at com.peacetech.tomcat.auth.SessionExpiredWarning.invoke(SessionExpiredWarning.java:109)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor236" daemon prio=1 tid=0x641444b8 nid=0x2702 in Object.wait() [0x671e8000..0x671e9770]
at java.lang.Object.wait(Native Method)
- waiting on <0x71f8efb0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:474)
at com.novell.ldap.Connection.acquireWriteSemaphore(Unknown Source)
- locked <0x71f8efb0> (a java.lang.Object)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Message.sendMessage(Unknown Source)
at com.novell.ldap.MessageAgent.sendMessage(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.search(LdapPrincipalBuilder.java:176)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.findPrincipal(LdapPrincipalBuilder.java:100)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.provide(LdapPrincipalBuilder.java:51)
at com.peacetech.tomcat.auth.PrincipalBuilderRealm.provide(PrincipalBuilderRealm.java:113)
at com.peacetech.tomcat.auth.PrincipalBuilderRealm.authenticate(PrincipalBuilderRealm.java:36)
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:256)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:416)
at com.peacetech.tomcat.auth.RSANextTokenValve.invoke(RSANextTokenValve.java:185)
at com.peacetech.tomcat.auth.SessionExpiredWarning.invoke(SessionExpiredWarning.java:109)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
"TP-Processor232" daemon prio=1 tid=0x64172318 nid=0x26e0 in Object.wait() [0x68ce4000..0x68ce5570]
at java.lang.Object.wait(Native Method)
- waiting on <0x71f8efb0> (a java.lang.Object)
at java.lang.Object.wait(Object.java:474)
at com.novell.ldap.Connection.acquireWriteSemaphore(Unknown Source)
- locked <0x71f8efb0> (a java.lang.Object)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Connection.writeMessage(Unknown Source)
at com.novell.ldap.Message.sendMessage(Unknown Source)
at com.novell.ldap.MessageAgent.sendMessage(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.novell.ldap.LDAPConnection.search(Unknown Source)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.search(LdapPrincipalBuilder.java:176)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.findPrincipal(LdapPrincipalBuilder.java:100)
at com.peacetech.tomcat.auth.LdapPrincipalBuilder.provide(LdapPrincipalBuilder.java:51)
at com.peacetech.tomcat.auth.PrincipalBuilderRealm.provide(PrincipalBuilderRealm.java:113)
at com.peacetech.tomcat.auth.PrincipalBuilderRealm.authenticate(PrincipalBuilderRealm.java:36)
at org.apache.catalina.authenticator.FormAuthenticator.authenticate(FormAuthenticator.java:256)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:416)
at com.peacetech.tomcat.auth.RSANextTokenValve.invoke(RSANextTokenValve.java:185)
at com.peacetech.tomcat.auth.SessionExpiredWarning.invoke(SessionExpiredWarning.java:109)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:744)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:674)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Regards,
Moises Morales
Novell Developer Support
Thank you for your suggestion. Now I know precisely WHEN (but not why)
it happens. It happens if I unload LDAP.NLM on Netware server. All
connection hung and NEVER time out. I tried everything to make them
timeout - set LDAPConstraint.timeLimit no effect, pass my own
SSLSocketFactory which decorates JSSE factory and set soTimeout on each
socket - no effect. Every time I do bind or search on one of such
connections it hangs and never ever timeout
I found a work around - call LDAPConnection.isConnectionAlive before
every LDAP call - that one does not hang on bad connection and return
false as expected. It solves immediate problem but it makes my code
EXTREMELY dirty with all this "alive" checks. Not to say it is an extra
network roundtrip for each search or bind (even afraid to think what
might happen if connection goes dead in the middle of fetch - will stuck
I guess).
I have not tested what would happen if entire NetWare server (or TCP/IP
stack) got restarted not just LDAP. I suspect it might timeout then. I
have a suspicion that NLDAP itself somehow does not close sockets they
linger somewhere in TCP/IP stack and that's why client can not detect
broken connection. But I am not very familiar with internals of TCP/IP
protocol and NLDAP implementation so it is just a guess
I hope you would have a solution for us - I would hate having to
retrofit all our LDAP code with "Alive" checks
Thank you for your assistance
Alex
"Roytman, Alex" <Royt...@peacetech.com> wrote in message news:mailman.206.113823...@forge.novell.com...
Susan,
Thank you very much for your suggestion. This does not seem to be a threading issue I can reproduce this issue with a single thread:
- Open connection, bind, search
- Unload LDAP.NLM
- Search
This sequence cause second search to hang in exactly the same place as my stack trace and never timeout regardless o what set in LDAPConstraints and even regardless ot Socket.setSoTimeout() I do in my SSLSocketFactory decorator for each created socket.
I wonder if cloning dead connection will give me a good one (one which will not block) I will try it bit I doubt it will help
Thank you
Alex
From: jldap-de...@forge.novell.com
[mailto:jldap-de...@forge.novell.com] On
Behalf Of Susan Perrin
Sent: Friday, January 27, 2006
3:26 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
In addition to what Moises wrote, when you write "If I remember correctly we share the same connection to read LDAP data from multiple threads. I recall it was specified to be thread safe", that's only when you use clone().
Thank you
Susan
Susan,
Thank you very much I will give it a try.
Couple more questions:
- When using LDAPConnection.clone() to share a physical connection by multiple thread, should each thread close its clone immediately when it is done with it?
- Is cloned connection any different from the primary one? Will clone.close() close underling connection when all clones a closed or it needs to be closed explicitly via the primary connection?
- Should I just use connection pool rather then cloning?
Thanks again for your assistance
Alex
From:
jldap-de...@forge.novell.com [mailto:jldap-de...@forge.novell.com] On Behalf Of Susan Perrin
Sent: Monday, January 30, 2006
5:29 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
Hi
In the specific case you mention, the jldap libraries send notice to registered LDAPUnsolicitedNotificationListener's. So the following will exit for me:
import com.novell.ldap.*;
import com.novell.ldap.util.Base64;
import java.util.Enumeration;
import java.util.Iterator;
import java.io.UnsupportedEncodingException;
import java.io.IOException;
import java.io.BufferedReader;
import java.io.InputStreamReader;
public class Search
{
public class theListener implements
LDAPUnsolicitedNotificationListener
{
public theListener()
{
}
// The required method that has to be
implemented by all listeners. This
// method is called on unsolicited notifications.
public void messageReceived(LDAPExtendedResponse msg)
{
String myMsg = ((LDAPExtendedResponse)msg).getID();
if(myMsg == LDAPConnection.SERVER_SHUTDOWN_OID)
{
System.out.println("Received notification of
unsolicited notification " + myMsg);
System.out.println("got shutdown");
System.exit(-1);
}
}
}
/**
* @param args
*/
public static void main(String[] args)
{
if (args.length != 5)
{
System.out.println("Usage: java Search
<host name> <login dn>"
+ " <password> <search base>\n"
+
" <search
filter>");
System.out.println("Example: java Search Acme.com"
+ " \"cn=admin,o=Acme\""
+ " secret \"ou=sales,o=Acme\"\n"
+ "
\"(objectclass=*)\"");
System.exit(0);
}
new Search(args[0], args[1], args[2], args[3], args[4]);
}
public Search(String ldapHost, String loginDN, String
password, String searchBase, String searchFilter)
{
int ldapPort = LDAPConnection.DEFAULT_PORT;
int ldapVersion = LDAPConnection.LDAP_V3;
LDAPConnection lc = new LDAPConnection();
LDAPSearchConstraints cons = new LDAPSearchConstraints();
theListener myListener = new theListener();
try
{
lc.addUnsolicitedNotificationListener(myListener);
lc.setConstraints(cons);
// connect to the server
lc.connect( ldapHost, ldapPort );
// bind to the server
lc.bind( ldapVersion, loginDN,
password.getBytes("UTF8") );
while(true)
doSearch(lc, searchBase, searchFilter);
// disconnect with the server
//lc.disconnect();
}
catch( Exception e )
{
e.printStackTrace();
}
System.exit(0);
}
public void doSearch(LDAPConnection lc, String
searchBase, String searchFilter)
{
int searchScope = LDAPConnection.SCOPE_ONE;
LDAPEntry nextEntry;
try
{
LDAPSearchResults searchResults =
lc.search(searchBase,
searchScope,
searchFilter,
null,
// return all attributes
false); //
return attrs and values
while ( searchResults.hasMore())
{
nextEntry = searchResults.next();
System.out.println("\n" + nextEntry.getDN());
System.out.println(" Attributes: ");
LDAPAttributeSet attributeSet =
nextEntry.getAttributeSet();
Iterator allAttributes = attributeSet.iterator();
while(allAttributes.hasNext())
{
LDAPAttribute attribute =
(LDAPAttribute)allAttributes.next();
String attributeName = attribute.getName();
System.out.println("
" + attributeName);
Enumeration allValues = attribute.getStringValues();
if( allValues != null)
{
while(allValues.hasMoreElements())
{
String Value = (String)
allValues.nextElement();
if (Base64.isLDIFSafe(Value))
{
// is printable
System.out.println("
" + Value);
}
else
{
// base64 encode and then print
out
Value =
Base64.encode(Value.getBytes());
System.out.println("
" + Value);
}
}
}
}
}
}
catch( LDAPException e )
{
e.printStackTrace();
System.exit(-1);
}
}
}
Thank you very much Susan. I have implemented my own simplistic pool based on your suggestion to use SERVER_SHUTDOWN message, making sure dead connections are removed and never given to pool user I will look into your sample for more details.
Does clone clones constraints from master connection?
Thanks again
Alex
From:
jldap-de...@forge.novell.com [mailto:jldap-de...@forge.novell.com] On Behalf Of Susan Perrin
Sent: Tuesday, January 31, 2006
2:23 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
Hi
Susan,
This is exactly my problem – LDAP goes down all client connections hang. As I mentioned in my first email, even if I supply a custom Socket Factory which sets soTimeout every time Socket is created client will
wait indefinitely. Can’t wait for the fix J
Thanks again for your kind assistance
Alex
From:
jldap-de...@forge.novell.com [mailto:jldap-de...@forge.novell.com] On Behalf Of Susan Perrin
Sent: Tuesday, January 31, 2006
4:03 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
Hi
It also helps to interrupt the thread so it will return and then close the
connection. The interrupt happens immediately on Solaris as it supports
interruptable IO but no on Windows.
BTW, is this more of a socket problem with java? I seem to hear about other
socket operations in java that have problems similar to this. Especially
when the machine disappears (as in the case when you pull the network cord).
Mark
----Original Message Follows----
From: "Susan Perrin" <dev...@novell.com>
Reply-To: jlda...@forge.novell.com
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
Date: Tue, 31 Jan 2006 21:09:44 GMT
If the LDAP service becomes un-responsive, then the connection fails and
the TCP session never drops.
When we spoke of this before, the solution was to create our own
"SocketFactory" setting a timeout on the socket and then use
LDAPConnection.setSocketFactory(myTLSFactory).
What we were hoping for was an exposed method so we could access the
methods on the base socket class to set the parameter for the socket.
The methods for the socket class is java.net.socket.
As I remember the methods that were of interest to our team at the time
were:
setSoTimeout(int timeout) Enable/disable SO_TIMEOUT with the specified
timeout, in milliseconds.
Hopefully, this setSoTimeout would help the "hang" problem that has been
experianced.
and
setTcpNoDelay(boolean on) - Enable/disable TCP_NODELAY (disable/enable
Nagle's algorithm).
What we have found with Nagle' algorithm, which tries to maximize the
TCP payload, that the typical LDAP client send such small TCP packets
that disabling Nagle should/would improve performance.
It would be really nice if Nagle could be disabled on the LDAP server
socket (SUN's server does expose this form the server) as many of the
response packets from the server contain only 0 or 1 as a response; but
that is another story.
-jim
Did setting soTimeout in custom socket factory work for you? Does not
seem to work for me
Thank you
Alex
-----Original Message-----
From: jldap-de...@forge.novell.com
[mailto:jldap-de...@forge.novell.com] On Behalf Of Jim Willeke
Sent: Tuesday, January 31, 2006 11:17 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
Yes, your solution seems along the lines (only even more straightforward
than) the example at
http://developer.novell.com/ndk/doc/samplecode/jldap_sample/ConnectTimeOut.java.html.
Here they're just thinking about the connect possibility, but it gets a bit
more complex with additional functionality.
Your point about the underlying socket is right, I think. It was easy
enough to use blocking reads but what do you do when the connection is
broken without any shutdown information? This seems like a good article:
http://www.javacoffeebreak.com/articles/network_timeouts/ The problem with
the socket options is that they require jdk 1.4 or higher, but I believe
(please post if you disagree) that this is an acceptable requirement these
days.
Thanks
Susan
Sounds like a reasonable bug/enhancement request to me. I'll submit it to
the eDir team. thanks!
Susan
JDK 1.4 actually 1.5 is acceptable and desirable for us.
But as I said my setting soTimeout on underlying socket in socket
factory does not seem to help on LDAP unload (I will test on network
connection broken). Why do you think is that? May be unloading LDAP does
not close open sockets on netware?
Thank you very much
Alex
-----Original Message-----
From: jldap-de...@forge.novell.com
[mailto:jldap-de...@forge.novell.com] On Behalf Of Susan Perrin
Sent: Wednesday, February 01, 2006 1:53 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
I tested socket timeout by unplugging my firewall from internet and it
worked! So socket does timeout as it suppose to if I use my custom
socket factory and set soTimeout. Now back to this peculiar issue of
socket not timing out on LDAP.NLM unload. Any idea why? It is not
critical any more since we implemented SERVER_SHUTDOWN handling but I
still would like to understand why it is happening.
Thank you
I think it's where you are... I'm seeing a problem if I connect/bind/unload
nldap.nlm then search. It's waiting indefinitly on a write semaphore but
the io exception is not processed. This looks like a thread synch issue.
I'll look into it as possible bug.
Thanks
Susan
I think it's a thread timing thing. I'm looking into it. Without the
unsolicited handler I'm getting stuck in a semaphore wait.
Thanks for pointing this out.
Susan
Alex
-----Original Message-----
From: jldap-de...@forge.novell.com
[mailto:jldap-de...@forge.novell.com] On Behalf Of Susan Perrin
Sent: Wednesday, February 01, 2006 6:22 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
Well actually it should be
if (LDAPConnection.SERVER_SHUTDOWN_OID.eqals(myMsg)) {
}
To guard agains NullPointerException right? J
From:
jldap-de...@forge.novell.com [mailto:jldap-de...@forge.novell.com] On Behalf Of Susan Perrin
Sent: Thursday, February 02, 2006
4:02 PM
To: jlda...@forge.novell.com
Subject: Re: LDAP Hangs
correction,