JZ0EM: End of data Error With ASA 9.0.2 And BEA Weblogic 8.1 SP6
Anthony Mangione
I'm currently using iAnywhere Adaptive Server Anywhere Network Server
Version 9.0.2.2451. I have a web application created through BEA Weblogic
Workshop.
We've recently upgraded our BEA environment from Weblogic Server 8.1 SP4 to
Weblogic Server 8.1 SP6. We are using the latest version of the JVM
supplied by BEA, jRockit.
I am using the jConnect 5.5 driver (that came bundled in the BEA product,
which is Build (25152)) to connect from my application servers to the
database server. I have my BEA environment setup to test connections made
to the database server and I am encountering problems with my connections
pools becoming unhealthy. I've been working with BEA support on this issue
and they would like me to send them some database error logs that might give
us all some more information on possible connection problems, database locks
or any other thing that would be helpful in terms of figuring out why
"dbping" wouldn't be working from the application server. I've set the -o
switch on my QA/test environment, but it doesn't really seem to be giving me
any detailed information...
Any suggestions?
In addition, if anyone has any idea as to why I would be encountering these
issues, any information would be helpful? Is there any known conflict with
BEA WLS 8.1 SP6 and iAnywhere 9.0.2? Should I apply EBF's to ASA? Is my
driver version too out-of-date?
Here are the errors in my BEA error log when BEA connection pools become
unhealthy for no apparent reason:
-------------------
####<Oct 13, 2007 3:10:57 PM EDT> <Error> <JDBC> <[MACHINE]> <[SERVERNAME]>
<ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_4>.ioThreadPool'>
<<WLS Kernel>> <> <BEA-001112> <Test "SELECT 1" set up for pool
"cgJMSPool-nonXA" failed with exception: "java.sql.SQLException: JZ006:
Caught IOException: java.io.IOException: JZ0EM: End of data.".>
####<Oct 13, 2007 3:10:57 PM EDT> <Info> <JDBC> <[MACHINE]> <[SERVERNAME]>
<ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_4>.ioThreadPool'>
<<WLS Kernel>> <> <BEA-001128> <Connection for pool "cgJMSPool-nonXA"
closed.>
####<Oct 13, 2007 3:10:57 PM EDT> <Warning> <JDBC> <[MACHINE]>
<[SERVERNAME]> <ExecuteThread: '0' for queue:
'JMSStore<cgJMSStore_auto_4>.ioThreadPool'> <<WLS Kernel>> <> <BEA-001129>
<Received exception while creating connection for pool "cgJMSPool-nonXA":
JZ00L: Login failed. Examine the SQLWarnings chained to this exception for
the reason(s).>
####<Oct 13, 2007 3:10:57 PM EDT> <Info> <JDBC> <[MACHINE]> <[SERVERNAME]>
<ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_4>.ioThreadPool'>
<<WLS Kernel>> <> <BEA-001156> <Stack trace associated with message 001129
follows:
java.sql.SQLException: JZ00L: Login failed. Examine the SQLWarnings chained
to this exception for the reason(s).
at com.sybase.jdbc2.jdbc.ErrorMessage.raiseError(ErrorMessage.java:506)
at com.sybase.jdbc2.tds.Tds.processLoginAckToken(Tds.java:3224)
at com.sybase.jdbc2.tds.Tds.doLogin(Tds.java:483)
at com.sybase.jdbc2.tds.Tds.login(Tds.java:405)
at com.sybase.jdbc2.jdbc.SybConnection.tryLogin(SybConnection.java:218)
at
com.sybase.jdbc2.jdbc.SybConnection.regularConnect(SybConnection.java:195)
at com.sybase.jdbc2.jdbc.SybConnection.<init>(SybConnection.java:174)
at com.sybase.jdbc2.jdbc.SybConnection.<init>(SybConnection.java:126)
at com.sybase.jdbc2.jdbc.SybDriver.connect(SybDriver.java:179)
at
weblogic.jdbc.common.internal.ConnectionEnvFactory.makeConnection(Connection
EnvFactory.java:251)
at
weblogic.jdbc.common.internal.ConnectionEnvFactory.refreshResource(Connectio
nEnvFactory.java:330)
at
weblogic.common.resourcepool.ResourcePoolImpl.refreshResource(ResourcePoolIm
pl.java:1694)
at
weblogic.common.resourcepool.ResourcePoolImpl.checkResource(ResourcePoolImpl
.java:1590)
at
weblogic.common.resourcepool.ResourcePoolImpl.checkAndReturnResource(Resourc
ePoolImpl.java:1480)
at
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolIm
pl.java:327)
at
weblogic.common.resourcepool.ResourcePoolImpl.reserveResource(ResourcePoolIm
pl.java:293)
at
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:475
)
at
weblogic.jdbc.common.internal.ConnectionPool.reserve(ConnectionPool.java:394
)
at
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolMa
nager.java:81)
at
weblogic.jdbc.common.internal.ConnectionPoolManager.reserve(ConnectionPoolMa
nager.java:88)
at weblogic.jdbc.pool.Driver.connect(Driver.java:148)
at weblogic.jms.store.JDBCIOStream.opendb(JDBCIOStream.java:2204)
at weblogic.jms.store.JDBCIOStream.ping(JDBCIOStream.java:1286)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:357)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:493)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
>
####<Oct 13, 2007 3:10:57 PM EDT> <Info> <JDBC> <[MACHINE]> <[SERVERNAME]>
<ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_4>.ioThreadPool'>
<<WLS Kernel>> <> <BEA-001128> <Connection for pool "cgJMSPool-nonXA"
closed.>
####<Oct 13, 2007 3:10:57 PM EDT> <Alert> <JMS> <[MACHINE]> <[SERVERNAME]>
<ExecuteThread: '0' for queue: 'JMSStore<cgJMSStore_auto_4>.ioThreadPool'>
<<WLS Kernel>> <> <BEA-040095> <JMSServer "cgJMSServer_auto_4". Store I/O
failure, java.io.IOException: JMS JDBC store, connection pool =
<cgJMSPool-nonXA>, prefix = <dia_4>: ping failed, database not responding
java.io.IOException: JMS JDBC store, connection pool = <cgJMSPool-nonXA>,
prefix = <dia_4>: SQL exception
weblogic.jdbc.extensions.ConnectionDeadSQLException:
weblogic.common.resourcepool.ResourceDeadException: Could not create pool
connection. The DBMS driver exception was: JZ00L: Login failed. Examine the
SQLWarnings chained to this exception for the reason(s).
at
weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUti
l.java:203)
at weblogic.jdbc.pool.Driver.connect(Driver.java:161)
at weblogic.jms.store.JDBCIOStream.opendb(JDBCIOStream.java:2204)
at weblogic.jms.store.JDBCIOStream.ping(JDBCIOStream.java:1286)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:357)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:493)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
at weblogic.jms.store.JDBCIOStream.throwIOException(JDBCIOStream.java:498)
at weblogic.jms.store.JDBCIOStream.opendb(JDBCIOStream.java:2287)
at weblogic.jms.store.JDBCIOStream.ping(JDBCIOStream.java:1286)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:357)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:493)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
.
java.io.IOException: JMS JDBC store, connection pool = <cgJMSPool-nonXA>,
prefix = <dia_4>: ping failed, database not responding
java.io.IOException: JMS JDBC store, connection pool = <cgJMSPool-nonXA>,
prefix = <dia_4>: SQL exception
weblogic.jdbc.extensions.ConnectionDeadSQLException:
weblogic.common.resourcepool.ResourceDeadException: Could not create pool
connection. The DBMS driver exception was: JZ00L: Login failed. Examine the
SQLWarnings chained to this exception for the reason(s).
at
weblogic.jdbc.common.internal.JDBCUtil.wrapAndThrowResourceException(JDBCUti
l.java:203)
at weblogic.jdbc.pool.Driver.connect(Driver.java:161)
at weblogic.jms.store.JDBCIOStream.opendb(JDBCIOStream.java:2204)
at weblogic.jms.store.JDBCIOStream.ping(JDBCIOStream.java:1286)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:357)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:493)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
at weblogic.jms.store.JDBCIOStream.throwIOException(JDBCIOStream.java:498)
at weblogic.jms.store.JDBCIOStream.opendb(JDBCIOStream.java:2287)
at weblogic.jms.store.JDBCIOStream.ping(JDBCIOStream.java:1286)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:357)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:493)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
at weblogic.jms.store.JDBCIOStream.throwIOException(JDBCIOStream.java:498)
at weblogic.jms.store.JDBCIOStream.ping(JDBCIOStream.java:1318)
at weblogic.jms.store.StoreRequest.doTheIO(StoreRequest.java:357)
at weblogic.jms.store.JMSStore.execute(JMSStore.java:493)
at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
-------------------
Thanks in advance.
Sincerely,
Anthony Mangione
JAT Software
aman...@jatsoftware.com
Anthony Mangione
Anthony Mangione
Just hoping to see if anyone might have a suggestion for me on this... In
desperate need of some help :(
Thanks in advance...
Anthony Mangione
JAT Software
aman...@jatsoftware.com
"Anthony Mangione" <aman...@jatsoftware.com> wrote in message
news:47151786@forums-1-dub...
Unknown
[the difference is important to your approach here
as well as to who you really need to be speaking to]
I was not aware that JConnect was bundled with BEA Weblogic.
Given that you/they may need to work directly with our support
organization. There is also a JConnect newsgroup that has more
JDBC experts on it.
The fact that there is some kind of a connection failure
should be pretty trivial to fix if this fails on every connection
attempt (and as such is probably not a pooling thing at all).
You can add a -z -o to the server to capture more info there
if this is a solid failure and a basic connectivity problem.
If this is just basic connectivity you should probably show
the URL used and spell out your server information. It is
entirely possible that no connections are ever being seen by the
database server.
If this is an intermittent kind of problem it may take a lot
of logging to see it occuring and there may be not a
lot of information as to why. If it is intermittent **working
either directly with technical support** or trying this out
on the **JConnect newsgroups** may be a better approach.
The one thing I see that explains why no one else has
jumped on this is the fact that Weblogic is not chaining
through all of the JDBC SQLExceptions as recommend
by the text provided
Examine the SQLWarnings chained
that would be standard for JDBC exception programming
and something that would need to be enabled from the BEA
side of the equation. With connection issues, it is often
the 2nd exception in the chain that tells you a little more
about about how it is failing.
"Anthony Mangione" <aman...@jatsoftware.com> wrote in message
news:47177c48$1@forums-1-dub...
Anthony Mangione
Thank you for your response...
This is an intermittent problem... For example, my entire production
environment has been running just fine since early AM without any issues.
This issue just seems to pop up on it's own for no apparent reason, in fact,
at times when there is low user load.
If you could throw me the name of the jConnect newsgroup, in case I can't
find it, that would be great :)
I will implement the -z and -o switches to my server parameters so that the
DBMS will log (as requested by BEA), so thank you very much for that :) I
don't think this is a basic connectivity problem, as the application is
running just fine now, fetching data from the DB, etc...
I wanted to hit the newsgroups before I hit Sybase Technical Support,
however, it looks like ultimately I may be contacting them...
Also, thank you about the elaboration of the chaining of the JDBC
SQLExceptions. I will ask BEA why this is the case.
Any additional information by anyone would be very helpful... Someone on the
BEA newsgroups thinks we are doing to much "connection testing" and
"connection shrinking", which are options through our BEA config. However,
our issue is that we were doing it on their previous version without any
issues... This is a tough one...
Thanks again for everything.
Anthony Mangione
JAT Software
"Nick Elson" <@@@nick@@@.@@@elson@sybase@@@.@@@com@@@> wrote in message
news:47178d27@forums-1-dub...
Shuchit
> environment has been running just fine since early AM without any
> issues. This issue just seems to pop up on it's own for no apparent
> reason, in fact, at times when there is low user load.
>
> If you could throw me the name of the jConnect newsgroup, in case I
> can't find it, that would be great :)
>
Anthony Mangione
Thank you Shuchit for the URL link...
A couple of things I forgot to mention, and if anyone has any
comments/suggestions, that would be great:
First, our prior environment had iAnywhere 9.0.2 running on a Windows 2000
Server machine with only 2GB of RAM. I was allocating about a GB of that to
the engine's cache in the startup parameters (e.g., -c 1024M).
Now, the current environment has iAnywhere 9.0.2 running on a Windows 2003
Server machine with 4GB of RAM. When I initially was determining the amount
of memory to allocate to the cache I wasn't aware of the limit, which I
believe is something along the lines of 1.8GB. So, right now, I have it
running as "-c 1792M".
Is this value too high and/or too close to the limit where I wouldn't notice
a problem at startup but maybe afterwards? Is there any issues with Windows
2003 Server at this version of level of ASA? I remember doing some research
and finding that 9.0.2 was OK on Win2003.
In addition, I have not applied any EBF's whatsoever.
Anthony Mangione
JAT Software
"Shuchit" <m...@privacy.net> wrote in message
news:Xns99CE6B20D19Fs...@127.0.0.1...
Jeff Albion (iAnywhere Solutions)
It seems like you're hitting a general connectivity issue, based on what
I'm seeing from the logs.
- How are you connecting with jConnect? Do you have the URL string handy?
- How are you starting your server? (Service / command-line?)
- How is the server running? What switches are in place?
- Can you connect with another client outside of BEA at times when the
connections are dropped?
- How is the web server and DBMS connected? Do they communicate across
the internet? LAN?
- How is your server licensed? Is it per-seat / per-processor? Have you
checked to see how many connections you have open in your connection
pool when it tries to open a new connection? (Running sa_conn_info() at
the server should give you this information...)
Regards,
--
Jeff Albion, Product Support Analyst
iAnywhere Solutions
iAnywhere Developer Community : http://www.ianywhere.com/developer
iAnywhere Documentation : http://www.ianywhere.com/developer/product_manuals
ASA Patches and EBFs :
http://downloads.sybase.com/swd/summary.do?baseprod=144&client=ianywhere&timeframe=0
Jeff Albion (iAnywhere Solutions)
I'm seeing that there's 156 concurrent connections in the log from this
one application server. How many other application servers attach to the
database?
The "-z -o [filename].log" should give us more context for what you're
seeing. Even better might be a "-zr SQL -zo reqlog.txt" to show what's
being executed at the database and why certain connections may be dropping.
(** WARNING: The above switches may consume a large amount of disk space
over time - if you do not have a large amount free, it's best to set
"-zs" to a reasonable size, such as: "-zs 512M")
AM> Yes. In addition, when this problem occurs, not all our application
servers
AM> (running their own connection pools) are affected. It's also never
the same
AM> application server.
This sounds very suspiciously like an internal limit you're hitting on
the engine based on the number of connections being used. What is the
output from:
C:\>"%ASANY9%\win32\dblic" "%ASANY9%\win32\dbsrv9.exe"
We'd need to see log files when the issue occurs to help you out further.
AM> Our license is "concurrent/CPU-based model"
That can't happen. You either have "Per-Seat (Concurrent)" or
"Processor" - not both. :D The dblic command above should tell us more.
Regards,
Anthony Mangione wrote:
> Jeff,
>
> Here are the answers to your questions:
>
>> - How are you connecting with jConnect? Do you have the URL string handy?
>
> Our connection pools are setup through a GUI console provided by BEA. Here
> are the associated properties:
>
> Driver Classname: com.sybase.jdbc2.jdbc.SybDriver
> Properties: user=[username],
> url=jdbc:sybase:Tds:[servername]:2638/[databasename], networkProtocol=Tds,
> userName=[username], portNumber=2638, databaseName=[databasename],
> serverName=[servername]
>
>> - How are you starting your server? (Service / command-line?)
>
> It is a Windows Service.
>
>> - How is the server running? What switches are in place?
>
>
> -x tcpip -c 1792M -p 1450 -n [databaseservername] -gp 2048
> [filepath]/[databasename].db -ek "[encryptionkey]" -z -o
> [filepath]\[filename].log
>
>> - Can you connect with another client outside of BEA at times when the
> connections are dropped?
>
> Yes. In addition, when this problem occurs, not all our application servers
> (running their own connection pools) are affected. It's also never the same
> application server.
>
>> - How is the web server and DBMS connected? Do they communicate across the
> internet? LAN?
>
> The web server doesn't access the DBMS directly. Our web server serves up
> static graphics only. All other request to the web server get forwarded to
> the BEA application servers (using a BEA-supplied plug-in DLL for IIS 5.0).
> The application servers access the DBMS. Everything communicates internally
> through our LAN. There is a firewall in front of the database for security
> reasons. Only the web server is accessable to the internet.
>
>> - How is your server licensed? Is it per-seat / per-processor? Have you
> checked to see how many connections you have open in your connection pool
> when it tries to open a new connection? (Running sa_conn_info() at the
> server should give you this information...)
>
> Our license is "concurrent/CPU-based model". I have not checked to see how
> many connections your I have open at that time. However, our configuration
> initially allocates 5, 5, 100 and 25 connections respectively for each of
> the 4 application server's 4 pools. Our load has been low, so I don't see
> these values incrementing higher at any point during these issues. If I've
> not answered these questions properly, please let me know. I've also
> attached a "netstat" that I ran on the application server machine that had
> the most recent issue and you'll see a bunch of established TCP connections
> to the database server.
>
> Anthony Mangione
> JAT Software
>
> "Jeff Albion (iAnywhere Solutions)" <firstname...@ianywhere.com> wrote
> in message news:47192547$1@forums-1-dub...
Unknown
to address this bug
================(Build #2532 - Engineering Case
#372605)================
If an application, connected to a server via jConnect, cancelled a
request
or closed a JDBC statement, the cancel or close could have failed
and/or
dropped the connection entirely. This problem has been fixed.
Your server is currently running with a build is earlier than that.
Further on the JConnect trail I find a number of bug fixes
mentioning that. In fact this one from the JConnect 5.5
ebf#13252 indicates you may be getting hit this same way
[see comments below about what I do not find in your traces
and `logout`s happening normally and at the connection
pools behest]
12896 401992 jConnect: Exception JZ0EM is being thrown on
Connection.close() internally and seen in debug
output. This causes some unnecessary code being
executed.
So... patching up JConnect too is a serious consideration as well.
Other than that . . . .
Unfortunately I do not see anything in those attachments
to indicate any of:
1 - Any abnormal disconnects. All disconnects seem to
be preceded with a TDS logout. There may have
been something in there but either the scrubbing of
the information (either only TDS was traced or any
errors and warnings are completely missing) or there
was not abnormal disconnect on the server side. I
also do not see and failed conections either.
2 - Any (of the recommended) JDBC SQLException chaining
seems to have been done to capture more of the context.
3 - If there has been any patching of JConnect since. I do see
you are still at 9.0.2 GA though.
The one thing that may be missing is the section of the SQL Anywhere
log just prior to the first snippet shown. Your snippets start at the
same second of the JDBC exception captured, but it is for a connection
and statement (a `set rowcount`) that starts occuring at 1:02:59. The
JDBC exception capture seems to have occured ending at that time
and was for a SELECT 1 statement.
So if there is anything in any SQL Anywhere output, it is the prior
file and not the ones you captured those snippets from.
One final observation. The origination exception is a JZOEM: End of data.
I would be expecting an 'End of data.' to be a normal (non-exception)
condition at the end of a result set. You may want to check into on
the JConnect newsgroup more. It that has a different meaning then
you need to chase that. If I am misinformed, you could be looking at
some kind of network traffic resource governer.
"Anthony Mangione" <aman...@jatsoftware.com> wrote in message
news:4727c8ff@forums-1-dub...
> To all:
>
> I've finally crashed again (never thought I'd be happy to say that...)
>
> Here's some interesting notes:
>
> 1) It's the same application server as the last time, in fact, it's the
> latest server I implemented into my application server cluster once I
> upgraded to the latest jRockit version (BEA suggestion). I thought for a
> minute that I configured that one incorrectly, but I doublechecked and I
> think I'm good.
> 2) All dbping's executed from the machine continue to SUCCEED outside of
> BEA
> WLS. I manually ran the dbping via DOS and have BATch jobs running and
> everything is succeeding. The failure seems to be internal to the
> connections established by the BEA WLS application server.
> 3) Jeff, to answer your question about the licensing, I've attached info
> on
> that and copied it below. Sorry for incorrectly stating my license
> situation...
>
> ------license-------
>>"%ASANY9%\win32\dblic" "%ASANY9%\win32\dbsrv9.exe"
> Adaptive Server Anywhere Server Licensing Utility Version 9.0.2.2451
> License read successfully.
> Licensed processors: 127
> User: JAT Software
> Company: JAT Software
> ------license-------
>
> I've attached the files because they're somewhat large to paste into the
> email body (especially the BEA one). You'll notice this is the first time
> the problem first appeared in the BEA WLS error log during a user's use of
> the application. There is still a "SELECT 1" occuring first because I
> believe I'm testing all created and reserved connections. Plus, after
> looking into the situation further, I'm testing all idle connections every
> 15 minutes.
>
> I'm unable to provide the entire iAnywhere log from top to bottom because
> of
> sensitive and proprietary information it the SQL queries. However, I've
> matched up the iAnywhere log with the WLS time-wise and it should give a
> representation as to what's going on here... The iAnywhere log is a
> result
> of "-z -o [filename].log" only. If you think I should implement "-zr
> SQL",
> etc...let me know. This log was up to 220MB... I had to split it up and
> sift through it... We have a lot of room on the hard drive... At least
> 20GB
> free.
>
> Instead of shutting down my cluster to get rid of the problem, I'm
> continuing to run the application server in it's current state but am
> disabling internet users from being redirected to it. If you have any
> "live" requests, just let me know. I'm doing this because I want to see
> if
> another one of the application servers fails as well...
>
> I apologize if I shouldn't be CC'ing some of you... If it's a problem,
> just
> let me know...
>
> Anthony
>
> "Jeff Albion (iAnywhere Solutions)" <firstname...@ianywhere.com>
> wrote
> in message news:472605b0$1@forums-1-dub...