TAF and FCF together ?
vis...@yahoo.com
I am desperately trying to find answers on how to use these two
technologies togther. As per Oracle docs, I cant use them together -
unless I am misreading it.
Could anyone suggest how I might accomplish these two together in my
application. I can use thin or thick application.
With thick we can get TAF. With thin we can get FCF (fast connection
failover).
Thanks
Vissu
joe.we...@gmail.com
What you are reading is true. The two are mutually exclusive. If your
application is only read-only with respect to the DBMS, then TAF can
work out, but if you're doing anything more serious, it can't actually
guarantee that everything will work or be transparent during a
failover.
If I remember when I get to work, I'll post the oracle document that
describes all the possible ways that TAF might fail to make failovers
transparent.
If you're doing clientside Java, TAF can actually leave the
connection
in a worse than dead state, where the basic connection can continue
to do plain new queries, but any PreparedStatements you already had
may be dead, and any query you may have been processing may fail
or return partial/incorrect results. Any XA transaction will have to be
rolled back and redone anyway, any DBMS session state you've changed
will be lost, etc.
All in all, it's much more reliable if you're writing to the DBMS or
doing
transactions etc, is to be able to handle a connection failure by
closing
the dead connection, asking for a new one (which will work with FCF
or some other middleware-based failover schemes), and redo your
current tx.
Joe Weinstein at BEA Systems
kuassi...@gmail.com
independent failover solution (works for both Thin and OCI) is FCF
which is based on RAC events.
Here is a brief comparison of the two (sorry for the display/layout).
FCF TAF
================== ===========================
Works for Both Client-side drivers Works only for JDBC-OCI
Rapid Node/Service Failure Rapid Node/Service Failure
Detection Detection
Connection Retry by Application Transparent Connection Retry
or Container and Query Retry
In-Flight Transactions Automatically Application must Roll
Rolled Back Back and notify TAF
Supports Dynamic Work Load No Support for Work Load
Balancing Balancing
I discuss Implicit Connection Cache, TAF, FCF and Runtime Connection
Load-Balancing in more details in chapter 7 in my book.
Kuassi,
http://db360.blogspot.com/2006/08/oracle-database-programming-using-java_01.html
joe.we...@gmail.com
Hi Kuassi, could you discuss the ramifications of these
TAF considerations as it relates to JDBC?
(metalink registration required)
http://metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=NOT&p_id=97926.1
Under what conditions would a running JDBC application
suffer from "Continuing work on existing cursors may raise an error
(eg: ORA-25401 "cannot continue fetches")"? Will existing
PreparedStatements
ever become defunct?
When will JDBC queries fail with "Failed over selects may raise an
error"
and what error?
Do JDBC connections ever rely on having altered the session state with
such calls as "ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD';"
or global temp tables, which will be lost on failover?
If a JDBC client is using the Oracle OCI driver, doing transactions,
how/when
to they know that a failover occurred so they know to roll back?
thanks,
Joe
DA Morgan
I think you've already received a good answer but there are two things
about your post I find interesting:
1. Why would you want to use them together?
2. Why would you want to use TAF at all if you can use FCF. It appears
that TAF is marked for deprecation.
--
Daniel Morgan
Puget Sound Oracle Users Group
kuassi...@gmail.com
These are very good questions that anyone contemplating TAF should
investigate..
I have discussed some of these in my book (chapter 7) and fairness
towards those who bought it suggests that i only give the following
summary:
a) things that TAF does well (transparently): connection failover and
query resubmit (or query failover) - with performance penalty
b) things that TAF does not do well (not transparent or does not work
at all): rolling back active transaction, and failing over session
states
As of 10g R2, FCF is bound to RAC while TAF may be used to failover
single-instance sites (Data Guard); TAF may also be customized through
its callback interface.
Hope this helps, Kuassi
joe.we...@gmail.com
clear best-practices and limitations that describe what
a programmer needs to do to make TAF and the Oracle
JDBC driver work together.
1 - That document seems to state that some queries will
*not* survive failover, whether with a delay or not. Is this
true?
2 - If I am in a transaction during a transparent failover,
what behavior will the Oracle driver exhibit that informs
me to roll back my transaction?
3 - If I have a set of PreparedStatements in my application
that I am continually re-using, will the failover ever affect
them and/or the DBMS cursors on which they depend?
4 - In cases where TAF is rerunning an in-flight query under-
the-covers, to get it back to where it was before the failover, I
have some questions. I am told that the position of the
new cursor is defined by fetching through the same *number*
of rows as had been fetched by the previous cursor. Given the
sizeable delay that may occur before the new cursor is
even started, is there any chance that the underlying data
may have changed so the new cursor counts past (and misses)
some new rows, and is set up to re-return rows that have
already been seen before the failover?
Thanks for your help on this,
Joe Weinstein
DA Morgan
> Hi Kuassi, what I'm looking for is a JDBC-specific list of
> clear best-practices and limitations that describe what
> a programmer needs to do to make TAF and the Oracle
> JDBC driver work together.
Go to www.psoug.org
Click on Resources
Click on Downloads
Click on TAF Download
It is coded in painfully commented detail.
--
joe.we...@gmail.com
Hi, and thank you very much. I'm looking over the download
now. It doesn't seem to do any serious queries. It only does
one tiny "select count(*) from ..." which will only return a
single one-row, one-column result set. This would never test
or expose the program to any TAF repositioned cursors. I
would suggest returning (and checking!) a significant data set,
at least 500 rows. I would make sure the count of rows is always
verifiable, and I would make the program update the data so
that if a query had to be rerun under-the-covers the order of the
data might change change. Also this example doesn't do any
transactions. If someone fluent with hacking that package would
like to contact me about suggestions for making it attempt
something more business-like, I would happily help/consult.
Joe Weinstein at BEA Systems
j
o
e
AT
b
e
a
DOT
c
o
m
DA Morgan
> Hi, and thank you very much. I'm looking over the download
> now. It doesn't seem to do any serious queries. It only does
> one tiny "select count(*) from ..." which will only return a
> single one-row, one-column result set. This would never test
> or expose the program to any TAF repositioned cursors. I
> would suggest returning (and checking!) a significant data set,
> at least 500 rows. I would make sure the count of rows is always
> verifiable, and I would make the program update the data so
> that if a query had to be rerun under-the-covers the order of the
> data might change change. Also this example doesn't do any
> transactions. If someone fluent with hacking that package would
> like to contact me about suggestions for making it attempt
> something more business-like, I would happily help/consult.
>
> Joe Weinstein at BEA Systems
Joe you've been here at c.d.o.s. long enough you know we vigorously
discourage top posting. Please adhere to the standard. Thanks.
With a TAF failover the query restarts with the same SCN. Thus the
result set is exactly the same.
--
Daniel Morgan
University of Washington
joe.we...@gmail.com
Sorry, my newsreader sometimes makes it very difficult
to bottom post!
My concern is that the SCN does not necessarily indicate the
same DBMS datum from DBMS to DBMS over time, for instance
if the table has changed between the failure of a connection and a
eventual reconnect to the other DBMS. The oracle TAF document
warns that on occasion, a query will fail, and I was interested
in expanding the example to actually exercise TAF's requery ability,
and wondering if these failures are always obvious.
I think we both agree that TAF is being deprecated, but there are
some, obviously that are still being lured by the siren-call of it's
promise. If we expanded the example to more thoroughly test
TAF, and if it exposed more of the Oracle-documented problems,
it might serve to sway these tempted souls from the rocks or
show how to handle them.
Joe
DA Morgan
>> With a TAF failover the query restarts with the same SCN. Thus the
>> result set is exactly the same.
>> --
>> Daniel Morgan
>> University of Washington
>> Puget Sound Oracle Users Group
>
> Sorry, my newsreader sometimes makes it very difficult
> to bottom post!
>
> My concern is that the SCN does not necessarily indicate the
> same DBMS datum from DBMS to DBMS over time, for instance
> if the table has changed between the failure of a connection and a
> eventual reconnect to the other DBMS.
Of course it does. That is precisely what Oracle's Multiversion
Consistency model is all about. This is Oracle Concepts and
Architecture 101. Oracle uses the SCN, which equates with a timestamp,
and the rollback/undo to reconstruct the data as it was at specific
moment in time.
There are members of your team working with RAC here in the Puget Sound
region, perhaps you are one of them, if so you might want to contact me
off line. At least one member of your team took my class at the U and
it seems there are quite a few misunderstandings on the part of BEA
about RAC.
> The oracle TAF document
> warns that on occasion, a query will fail, and I was interested
> in expanding the example to actually exercise TAF's requery ability,
> and wondering if these failures are always obvious.
Run the demo downloadable from the PSOUG website. That is precisely
what it reports and precisely what we use it to demonstrate.
Look in the code following the comment that begins:
* There are 7 possible failover event
> I think we both agree that TAF is being deprecated, but there are
> some, obviously that are still being lured by the siren-call of it's
> promise.
Hardly a siren call. It is a valuable technology not available from
any other RDBMS vendor. The only reason I prefer FCF is its relationship
with FAN and the fact that it works as well with SQL*Plus as with OCI,
or Java thick or think or, .NET.
> If we expanded the example to more thoroughly test
> TAF, and if it exposed more of the Oracle-documented problems,
> it might serve to sway these tempted souls from the rocks or
> show how to handle them.
>
> Joe
Perhaps I am mistaken but you don't need an expanded example. You
just need to understand, far better, the one you have already.
joe.we...@gmail.com
Alright, thanks, I'll work on it.
Joe
kuassi...@gmail.com
See elements of response inline
joeN...@BEA.com wrote:
> Hi Kuassi, what I'm looking for is a JDBC-specific list of
> clear best-practices and limitations that describe what
> a programmer needs to do to make TAF and the Oracle
> JDBC driver work together.
>
> 1 - That document seems to state that some queries will
> *not* survive failover, whether with a delay or not. Is this
> true?
Yes, TAF will not always successfully restart a query in progress. It
reopens the cursors and attempts to discard rows already returned. In
order to achieve that, it performs a chekcsum of to-be-discarded rows
and compares that checksum against a checksum for the rows already
returned. If the checksums are different, TAF knows the discarded rows
are not the same as the rows already returned. In such a case, it will
not resume returning rows and returns an error. Checksum discrepancies
are more likely to happen with replica databases which are not
block-for-block identical.
> 2 - If I am in a transaction during a transparent failover,
> what behavior will the Oracle driver exhibit that informs
> me to roll back my transaction?
JDBC is just a conduit b/w the RDBMS and TAF. (i) Upon node/instance
failure, the connection is dead and all in-flight transactions are
automatically rolled-back by the RDBMS. (ii) The JDBC Connection is
failed/switched over by TAF. (iii) TAF issues an exception. (iv) TAF
requires an ACK. (v) The application rolls back the transaction (which
can be see as an ACK to TAF) and replays the transaction.
> 3 - If I have a set of PreparedStatements in my application
> that I am continually re-using, will the failover ever affect
> them and/or the DBMS cursors on which they depend?
>
To my knowledge, PreparedStatement and their cursor are
re-openned/executed. However If the application uses any stored
procedures (i.e., CallableStatement), then the state of those
procedures is lost after failover.
> 4 - In cases where TAF is rerunning an in-flight query under-
> the-covers, to get it back to where it was before the failover, I
> have some questions. I am told that the position of the
> new cursor is defined by fetching through the same *number*
> of rows as had been fetched by the previous cursor. Given the
> sizeable delay that may occur before the new cursor is
> even started, is there any chance that the underlying data
> may have changed so the new cursor counts past (and misses)
> some new rows, and is set up to re-return rows that have
> already been seen before the failover?
See #1
joe.we...@gmail.com
kuassi...@gmail.com wrote:
> Joe,
>
> See elements of response inline
>
> joeN...@BEA.com wrote:
> > Hi Kuassi, what I'm looking for is a JDBC-specific list of
> > clear best-practices and limitations that describe what
> > a programmer needs to do to make TAF and the Oracle
> > JDBC driver work together.
> >
> > 1 - That document seems to state that some queries will
> > *not* survive failover, whether with a delay or not. Is this
> > true?
>
> Yes, TAF will not always successfully restart a query in progress. It
> reopens the cursors and attempts to discard rows already returned. In
> order to achieve that, it performs a chekcsum of to-be-discarded rows
> and compares that checksum against a checksum for the rows already
> returned. If the checksums are different, TAF knows the discarded rows
> are not the same as the rows already returned. In such a case, it will
> not resume returning rows and returns an error. Checksum discrepancies
> are more likely to happen with replica databases which are not
> block-for-block identical.
Kuassi, thank you very much. Your actual knowledge of the topic
is invaluable.
So, what will make DBMSes likely to be non-identical? For instance, I
suspect there is a timing issue for TAF failover. If an application has
paused (such as being one of many threads waiting to run in an
application server) when the DBMS failure occurs, I am wondering if
it's connection failover and repositioning of it's cursor will happen
only
when the application makes it's next use of the connection. Is that
true? If so, there may be other application instances who get going
sooner after the failover, and if any of these make any changes to the
data involved in the query, our first application's query will likely
fail,
correct? In fact, even if TAF endeavours to reposition all the
application
cursors immediately on failover, it the application has 200 connections
all being operated on independently by as many threads (maybe in
several JVMs), I'm guessing that it is unavoidable that TAF may allow
some of these threads to continue before others, and if any of these
change the data, other threads doing a (re)query of the data will fail.
> > 2 - If I am in a transaction during a transparent failover,
> > what behavior will the Oracle driver exhibit that informs
> > me to roll back my transaction?
>
> JDBC is just a conduit b/w the RDBMS and TAF. (i) Upon node/instance
> failure, the connection is dead and all in-flight transactions are
> automatically rolled-back by the RDBMS. (ii) The JDBC Connection is
> failed/switched over by TAF. (iii) TAF issues an exception. (iv) TAF
> requires an ACK. (v) The application rolls back the transaction (which
> can be see as an ACK to TAF) and replays the transaction.
The standard JDBC objects do not return any exception (due to the
failure/failover per se). It is only the callback that notifies the
application,
and the application must rollback/redo the transaction. If the
application is multi-threaded, and the callback is being processed by
and administrator thread, and the user thread is in the midst of any
possible JDBC call to the Oracle driver, does the Oracle driver (it's
internal synchronization etc) always allow the callback thread to
interrupt the user thread's JDBC operation and/or call a rollback on
the connection? The reason I ask is because I have seen deadlocks
on driver-internal locks when one user thread is doing it's JDBC and
another thread is trying to do management stuff like rollbacks, closing
JDBC objects etc.
> > 3 - If I have a set of PreparedStatements in my application
> > that I am continually re-using, will the failover ever affect
> > them and/or the DBMS cursors on which they depend?
>
> To my knowledge, PreparedStatement and their cursor are
> re-openned/executed. However If the application uses any stored
> procedures (i.e., CallableStatement), then the state of those
> procedures is lost after failover.
Thanks. So prepared or callable statements that use stored
procedures will throw exceptions, and must be closed and replaced.
Prepared statements that execute plain SQL should always survive,
correct?
kuassi...@gmail.com
See elements of response inline,
> > >
> > > 1 - That document seems to state that some queries will
> > > *not* survive failover, whether with a delay or not. Is this
> > > true?
> >
> > Yes, TAF will not always successfully restart a query in progress. It
> > reopens the cursors and attempts to discard rows already returned. In
> > order to achieve that, it performs a chekcsum of to-be-discarded rows
> > and compares that checksum against a checksum for the rows already
> > returned. If the checksums are different, TAF knows the discarded rows
> > are not the same as the rows already returned. In such a case, it will
> > not resume returning rows and returns an error. Checksum discrepancies
> > are more likely to happen with replica databases which are not
> > block-for-block identical.
>
> Kuassi, thank you very much. Your actual knowledge of the topic
> is invaluable.
Well, i am just the spokesman of our dev teams; you have a great
knowledge of the issues, yourself!
> So, what will make DBMSes likely to be non-identical? For instance, I
> suspect there is a timing issue for TAF failover. If an application has
> paused (such as being one of many threads waiting to run in an
> application server) when the DBMS failure occurs, I am wondering if
> it's connection failover and repositioning of it's cursor will happen
> only
> when the application makes it's next use of the connection. Is that
> true? If so, there may be other application instances who get going
> sooner after the failover, and if any of these make any changes to the
> data involved in the query, our first application's query will likely
> fail,
> correct? In fact, even if TAF endeavours to reposition all the
> application cursors immediately on failover, it the application has 200 connections
> all being operated on independently by as many threads (maybe in
> several JVMs), I'm guessing that it is unavoidable that TAF may allow
> some of these threads to continue before others, and if any of these
> change the data, other threads doing a (re)query of the data will fail.
Yes, there are some queries that will not survive failover. OCI
implicitly replays the query on the surviving instance using the
original SCN and discards the rows that were previously returned. If,
after failover, the same set of rows is not returned, then we throw an
error indicating that the query must be re-executed. With use of the
original SCN *and* the checksum verification on the re-fetched rows, it
is unlikely that OCI will erroneously re-return rows that had been
previously seen.
>
> > > 2 - If I am in a transaction during a transparent failover,
> > > what behavior will the Oracle driver exhibit that informs
> > > me to roll back my transaction?
> >
> > JDBC is just a conduit b/w the RDBMS and TAF. (i) Upon node/instance
> > failure, the connection is dead and all in-flight transactions are
> > automatically rolled-back by the RDBMS. (ii) The JDBC Connection is
> > failed/switched over by TAF. (iii) TAF issues an exception. (iv) TAF
> > requires an ACK. (v) The application rolls back the transaction (which
> > can be see as an ACK to TAF) and replays the transaction.
>
> The standard JDBC objects do not return any exception (due to the
> failure/failover per se). It is only the callback that notifies the
> application,
> and the application must rollback/redo the transaction. If the
> application is multi-threaded, and the callback is being processed by
> and administrator thread, and the user thread is in the midst of any
> possible JDBC call to the Oracle driver, does the Oracle driver (it's
> internal synchronization etc) always allow the callback thread to
> interrupt the user thread's JDBC operation and/or call a rollback on
> the connection? The reason I ask is because I have seen deadlocks
> on driver-internal locks when one user thread is doing it's JDBC and
> another thread is trying to do management stuff like rollbacks, closing
> JDBC objects etc.
>
You will most likely see an ORA-25402 error. However applications
should be prepared to handle errors in the 25400-25425 range and
rollback appropriately.
> > > 3 - If I have a set of PreparedStatements in my application
> > > that I am continually re-using, will the failover ever affect
> > > them and/or the DBMS cursors on which they depend?
> >
> > To my knowledge, PreparedStatement and their cursor are
> > re-openned/executed. However If the application uses any stored
> > procedures (i.e., CallableStatement), then the state of those
> > procedures is lost after failover.
>
> Thanks. So prepared or callable statements that use stored
> procedures will throw exceptions, and must be closed and replaced.
> Prepared statements that execute plain SQL should always survive,
> correct?
The surviving PreparedStatement assumes that the Statement handle's
bind values have not changed but this unlikey as different bind may
return a different result set.
At any rate, OCI transparently replays the execute-SQL/fetch during TAF
assuming same set of rows is returned from server.
Kuassi
- blog http://db360.blogspot.com/
- book
http://db360.blogspot.com/2006/08/oracle-database-programming-using-java_01.html
joe.we...@gmail.com
kuassi...@gmail.com wrote:
> Joe,
> See elements of response inline,
>
> > > > 1 - That document seems to state that some queries will
> > > > *not* survive failover, whether with a delay or not. Is this
> > > > true?
> > >
> > > Yes, TAF will not always successfully restart a query in progress. It
> > > reopens the cursors and attempts to discard rows already returned. In
> > > order to achieve that, it performs a chekcsum of to-be-discarded rows
> > > and compares that checksum against a checksum for the rows already
> > > returned. If the checksums are different, TAF knows the discarded rows
> > > are not the same as the rows already returned. In such a case, it will
> > > not resume returning rows and returns an error. Checksum discrepancies
> > > are more likely to happen with replica databases which are not
> > > block-for-block identical.
> >
> > Kuassi, thank you very much. Your actual knowledge of the topic
> > is invaluable.
>
> Well, i am just the spokesman of our dev teams; you have a great
> knowledge of the issues, yourself!
Well, thanks. Our chat will serve to educate others about TAF. Oracle
is
my favorite DBMS, I recommend it. It is only when I have problems that
I sound critical. I just want to make it explicit about how
untransparent
TAF can be, especially for JDBC in a complex working environment.
Ok. Can you home in for me on the timing question? For a given
TAF-enabled connection, is the failover and reposition done when
a user call makes OCI find that *this* connection has lost it's socket
(eg)?
I am guessing that OCI doesn't catalog connections by the DBMS, so
it can't/doesn't failover all the related connections at once, but will
do so
individually as each connection incurs a problem. A quiescent
connection
will not fail over until/unless used, right?
Again, can I ask you about multithreaded access to Oracle JDBC
objects? I am suggesting that if one Java thread gets the callback
that Connection A has failed over and it's DBMS transaction has been
rolled back, and some other Java thread is using connection A, and
because the failover is transparent at the user level, this other
thread is
still doing JDBC, perhaps querying and/or processing a Blob etc,
I am concerned that the admin thread may not be able to stop the
user thread it it's in the middle of a JDBC call.
Also, I assume that the autoCommit state of the session survives
a failover, correct? I would hate if a user thread that thought it was
operating in an autoCommit(false) mode, was suddenly doing updates
that were the back-half of the terminated transaction, but were now
being
committed individually. However, let's assume that the session stays
in autoCommit(false), and the application has a transactional
architecture
that is careful to avoid deadlocks by always locking datum A before
locking datum B. If there's a failover when a JDBC thread is doing a
transaction, and had locked datum A, the DBMS will now not have
datum A locked. Unless we're careful, fast and lucky, can the user
thread go on to issue it's update of datum B and a second update of
datum A, and risk a deadlock?
I noticed that the Oracle driver does do some session altering
during
initial log-in, eg: date format and NLS settings. This may only be the
thin
driver, but because these are lost with failover, I assume that if the
driver does the same via OCI, that it redoes it on failover.
> > > > 3 - If I have a set of PreparedStatements in my application
> > > > that I am continually re-using, will the failover ever affect
> > > > them and/or the DBMS cursors on which they depend?
> > >
> > > To my knowledge, PreparedStatement and their cursor are
> > > re-openned/executed. However If the application uses any stored
> > > procedures (i.e., CallableStatement), then the state of those
> > > procedures is lost after failover.
> >
> > Thanks. So prepared or callable statements that use stored
> > procedures will throw exceptions, and must be closed and replaced.
> > Prepared statements that execute plain SQL should always survive,
> > correct?
>
> The surviving PreparedStatement assumes that the Statement handle's
> bind values have not changed but this unlikey as different bind may
> return a different result set.
> At any rate, OCI transparently replays the execute-SQL/fetch during TAF
> assuming same set of rows is returned from server.
Ok, but to be clear, any prepared statement that was running a
stored procedure is dead, and must be replaced, correct? Or does
OCI also transparently re-execute the procedure? What about
executeBatch()? Does TAF/OCI re-execute the whole batch?
You could also maybe tell me about how much support for TAF there
is in the Oracle Application Server? For instance, with applications
built from generated EJBs etc, does Oracle generate EJBs that
buffer all their input so they can automatically replay their
transactional
input? If EJB A does a query and for each row it calls EJB B or EJB C
with the row's data etc, and these EJBs do updates based on
permutations
of the input, and after 30 loops of this and a similar descent from
each of
the other EJBs, when there's a failover, how often does the Oracle
Application
Server make it transparently OK?
kuassi...@gmail.com
Failover occurs when OCI detects that a particular socket has been
closed. A quiescent conneciton will not failover until it is used
Two different threads cannot both have an active call on the same
connection, so I am not sure what is meant by "stopping the user thread
if it's in the middle of a JDBC call."
this is correct
>Or does OCI also transparently re-execute the procedure?
Nope
> What about executeBatch()? Does TAF/OCI re-execute the whole batch?
No idea
> You could also maybe tell me about how much support for TAF there
> is in the Oracle Application Server?
Oracle AS rather integrates FCF and transparently handles connection
retry for CMP Beans. I don't have much details on the remaining part of
the question relative to EJBs