errors using weblogic sql driver: "No JDBC connection can be made because the transaction state is marked rollback"
lixin
We checked our log files. It seems that the error happens when
to obtain a jdbc connection. Have anyone seen similar problems
and knows how to fix it? thanks in advance.
We are using weblogic server 6.1sp2, and weblogic sql type 4 driver.
The functions that invoke the jdbc calls are stateless session bean
methods with their transaction attributes marked as Required.
There is no nested calls of these methods.
A partial stack trace we obtained is as following:
java.sql.SQLException: No JDBC connection can be made
because the transaction state is
Marked Rollback
at weblogic.jdbc.jts.Connection.getOrCreateConnection(Connection.java:586)
at weblogic.jdbc.jts.Connection.prepareStatement(Connection.java:115)
at weblogic.jdbc.rmi.internal.ConnectionImpl.prepareStatement(ConnectionImpl.java:135)
at weblogic.jdbc.rmi.SerialConnection.prepareStatement(SerialConnection.java:76)
....
lixin
Joseph Weinstein
lixin wrote:
> One of our customers starts to encounter this error message recently.
> We checked our log files. It seems that the error happens when
> to obtain a jdbc connection. Have anyone seen similar problems
> and knows how to fix it? thanks in advance.
>
> We are using weblogic server 6.1sp2, and weblogic sql type 4 driver.
> The functions that invoke the jdbc calls are stateless session bean
> methods with their transaction attributes marked as Required.
> There is no nested calls of these methods.
>
> A partial stack trace we obtained is as following:
>
> java.sql.SQLException: No JDBC connection can be made
> because the transaction state is
> Marked Rollback
> at weblogic.jdbc.jts.Connection.getOrCreateConnection(Connection.java:586)
Hi. This sounds like a JVM thread starvation issue, and/or a server load issue. What is
happening is that the transaction is started, and times out before the SSB even gets to
the first JDBC work. I would first verify that the customer is using the very latest JVM
available for the machine.
Joe Weinstein
YuanHui Liu
Joe,
We got the exact same error message. The error came after we got the JDBC connection,
and trying to create statement off it.
It occurs intermitently when we are running another standalone JAVA app to do
some end of day work, which results in the DB Server being very busy(90+% CPU
usage) for about 5 minutes. We see a surge of requests to the WLS JDBC Connection
pool. This would sometimes result in all our subsequent DB requests to fail and
lead to a crash.
We are using WLS6.0SP1. I do not think there's a 30 seconds wait leading to a
connection timeout that caused this(rather it is the end effect).
Can you give us a more detailed explanation? Is there a miscommunication between
our DB(Sybase12) and WLS?
Thanks.
-YuanHui Liu
YuanHui Liu
Joe,
We got the exact same error message. The error came after we got the JDBC connection,
and trying to create statement off it.
It occurs intermitently when we are running another standalone JAVA app to do
some end of day work, which results in the DB Server being very busy(90+% CPU
usage) for about 5 minutes. We see a surge of requests to the WLS JDBC Connection
pool. This would sometimes result in all our subsequent DB requests to fail and
lead to a crash.
We are using WLS6.0SP1. I do not think there's a 30 seconds wait leading to a
connection timeout that caused this(rather it is the end effect).
Can you give us a more detailed explanation? Is there a miscommunication between
our DB(Sybase12) and WLS?
Thanks.
-YuanHui Liu
Joseph Weinstein
YuanHui Liu wrote:
> Joe,
>
> We got the exact same error message. The error came after we got the JDBC connection,
> and trying to create statement off it.
>
> It occurs intermitently when we are running another standalone JAVA app to do
> some end of day work, which results in the DB Server being very busy(90+% CPU
> usage) for about 5 minutes. We see a surge of requests to the WLS JDBC Connection
> pool. This would sometimes result in all our subsequent DB requests to fail and
> lead to a crash.
>
> We are using WLS6.0SP1. I do not think there's a 30 seconds wait leading to a
> connection timeout that caused this(rather it is the end effect).
>
> Can you give us a more detailed explanation? Is there a miscommunication between
> our DB(Sybase12) and WLS?
Hi. It looks to you like it's after you get the connection, but really it's when the server is
gettng the pool connection. For performance/synchronization reasons we do a clever
delay: When your code asks for a pool connection we quickly give you the pool wrapper,
but we delay actually reserving the real underlying DBMS connection until your first
real need for a connection, at your first JDBC call, such as createStatement() etc.
It is while waiting for a pool connection long enough for the transaction coordinator
to have timed you out before you ever get a chance. It's nothing to do with the
DBMS or even JDBC, I believe. I think the weblogic server either has too few execute-threads
and/or too few CPU cycles to do the work load.
Yuanhui Liu
Okay, so there's a lazy initialization of the connection.
From reading our log I believe our failur is immediate rather
than waiting for 30+ seconds(the default setting) from the DB,
the timeout occurred later as a result. At the time either because the DB Server
is very busy.
Since we are running WLS6.0 we have only one connection pool,
we have defined a max of 150 threads in the pool. While this
is happening the DB Server is being pinned by an overnight job,
but the WLS Server is not busy at all. The DB and WLS resides
on different physical boxes.
We also have a thread dump from the WLS console when we rebooted the server, it
showed that we are hanging on to the thread & jdbc
connections after these exceptions has occurred instead of releasing them, note
"16083"(~4.5 hours) seconds has passed:
142 116222 Retry rollback request for tx: 'transaction=(IdHash=2963855,Name =
[EJB UserManagerBeanImpl.signalICUserServletHeartBeat()],Xid=30643:8f3838f3709bf53d,Status=Rolling
Back. [Reason = Unknown],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since
begin=16083,seconds left=10,ServerResourceInfo[weblogic.jdbc.jts.Connection]=(state=started,assigned=server),SCInfo[server]=(state=active),properties=({weblogic.jdbc=t3://159.55.158.25:8005,
weblogic.transaction.name=[EJB UserManagerBeanImpl.signalICUserServletHeartBeat()]}))'
Scheduled Trigger
So I would argue this problem actually chewed up resources on the WLS server.
-Yuanhui Liu
Joseph Weinstein
Yuanhui Liu wrote:
> Okay, so there's a lazy initialization of the connection.
> From reading our log I believe our failur is immediate rather
> than waiting for 30+ seconds(the default setting) from the DB,
> the timeout occurred later as a result. At the time either because the DB Server
> is very busy.
Ok. Do you have testConnsOnReserve set true? If so, when we reserve a pool
connection we run a query. If this query hangs a long time, it may cause the
problem.
> Since we are running WLS6.0 we have only one connection pool,
> we have defined a max of 150 threads in the pool. While this
> is happening the DB Server is being pinned by an overnight job,
> but the WLS Server is not busy at all. The DB and WLS resides
> on different physical boxes.
Ok. The pool doesn't contain threads, it contains JDBC connections, which
are reserved and used by execute-threads. If your server is running 35
execute-threads, then normally you would only need 35 connections in the
pool. More might actually degrade performance. If your typical user invoke
involves only one pool connection, and if you have refresh turned off, then
the connection-count=execute-thread-count holds. I recommend pool initial
capacity = pool max capacity = execute-thread-count, and turning off refresh.
TestConnsOnReserve is all the checking you need. However, if the connection
test query is too much for the DBMS to handle, it's a bad thing. I might consider
running without testConns at all, just for a test.
>
>
> We also have a thread dump from the WLS console when we rebooted the server, it
> showed that we are hanging on to the thread & jdbc
> connections after these exceptions has occurred instead of releasing them, note
> "16083"(~4.5 hours) seconds has passed:
>
> 142 116222 Retry rollback request for tx: 'transaction=(IdHash=2963855,Name =
> [EJB UserManagerBeanImpl.signalICUserServletHeartBeat()],Xid=30643:8f3838f3709bf53d,Status=Rolling
> Back. [Reason = Unknown],numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since
> begin=16083,seconds left=10,ServerResourceInfo[weblogic.jdbc.jts.Connection]=(state=started,assigned=server),SCInfo[server]=(state=active),properties=({weblogic.jdbc=t3://159.55.158.25:8005,
> weblogic.transaction.name=[EJB UserManagerBeanImpl.signalICUserServletHeartBeat()]}))'
> Scheduled Trigger
>
> So I would argue this problem actually chewed up resources on the WLS server.
Yes, perhaps. This execute-thread certainly didn't run for a long time.
Yuanhui Liu
Joseph Weinstein <j...@bea.com.remove.this> wrote:
>
>
>Yuanhui Liu wrote:
>
>> Okay, so there's a lazy initialization of the connection.
>> From reading our log I believe our failur is immediate rather
>> than waiting for 30+ seconds(the default setting) from the DB,
>> the timeout occurred later as a result. At the time either because
>the DB Server
>> is very busy.
>
>Ok. Do you have testConnsOnReserve set true? If so, when we reserve a
>pool
>connection we run a query. If this query hangs a long time, it may cause
>the
>problem.
>
>> Since we are running WLS6.0 we have only one connection pool,
>> we have defined a max of 150 threads in the pool. While this
>> is happening the DB Server is being pinned by an overnight job,
>> but the WLS Server is not busy at all. The DB and WLS resides
>> on different physical boxes.
>
>Ok. The pool doesn't contain threads, it contains JDBC connections, which
Sorry my mistake, our threadpool max is 150,
the JDBC connection pool in question has a max of 60, initial
value of 2. Both TestOnReserve & TestOnRelease is set to TRUE,
and RefreshPeriod is set to 10.
From the sniffer log we do see the query defined for testing is regularly being
executed, and have its transaction canceled right after.
At the onset of the problem, I believe neither threads nor JDBC connections were
anywhere close to the max, but then WLS crashed we probably maxed out on the connections
-- it is as if we did get the connections into the pool, they are stuck in some
retry-rollback loop.
If you can email me directly to continue this discussion, that would be great.
-YL
[-- snip --]
Joseph Weinstein
Ok. Unless you're running on some huge 16-processor machine,
this is probably way too many threads for the server. Note that the
JVM is only one OS process, and there is overhead in just checking
whether any of the 150 threads have some work to do. I'll bet this
has something to do with it. Back it off to 50 execute threads or even
35, and you may get better performance.
> the JDBC connection pool in question has a max of 60, initial
> value of 2. Both TestOnReserve & TestOnRelease is set to TRUE,
> and RefreshPeriod is set to 10.
This means you're asking your application invokes to suffer the lengthy
delay while the pool makes a new DBMS connection at runtime. TestOnRelease
is a waste of time (we shouldn't have ever implemented it IMO), and
refresh is redundant if you have testConnsOnReserve. It's only useful
for keeping idle connections busy enough that firewalls don't close them.
Please try:
Execute-threads=35,testOnReserve=true, testOnRelease=false, refresh period
=9999999.