Rolling back tests -- status and open issues

[Karen Tracey]

One of the item on the list for 1.1 is "Run Django test cases inside a transaction".  The ticket for this is #8138:

http://code.djangoproject.com/ticket/8138

and previous list discussion can be found here:

http://groups.google.com/group/django-developers/browse_thread/thread/49aa551ad41fb919/3d75d940a1cc23d5

The general idea is to speed up the running of Django tests by using transaction rollbacks at the end of tests instead of flush/syncdb at the beginning of each test.  While this approach generally works fine, it did introduce a set of (some weird) failures into the Django test suite that needed to be tracked down before it could be seriously considered for checkin.  That, I believe, is done now (see ticket for details).  Broadly the failures could be categorized into two piles:

1 - Some resulted from subtle differences in the old and new approaches and could be fixed by tweaking the new approach. 

2 - Others resulted from tests that made invalid assumptions -- like that primary keys for objects created by the test would be assigned starting at one.  These failures needed to be addressed by fixing the testcases to not rely on such assumptions.  The need to do that, though, highlights that this change may not be 100% transparent to anyone with their own set of tests, unless they've been diligent about never relying on such things.  (A to-do still remaining for this ticket is to write some doc for stuff like this.)

With all the failures figured out, the new rollback approach shows significant speedups for running the Django test suite.  Early measurements I did showed a factor of 6 speedup for sqlite, nearer 10-12 for MySQL/InnoDB, PostgreSQL, and Oracle.  I tried measuring the latest patch today on sqlite and see an 8x speedup there and 16x on MySQL/InnoDB, but don't have time to test the others at present.  (MySQL/MyISAM has no improvement -- since it doesn't support transactions the old flush/syncdb approach still needs to be used when running on that combination.)

The one big remaining issue I'm not entirely sure what to do about is doctests.  These, today, don't have any sort of automatic flush/syncdb isolation done for them.  So DB changes made by one doctest may be seen by other tests (though an intervening unit test that did flush/syncdb might clear things out.)  The initial approach taken by the early patches on #8183 was to enclose doctest runs in a rolled-back transaction, just as was being done for unit tests.  At first glance this seems like a good idea (it provides isolation that is known to be lacking currently), but upon further investigation it has a couple of problems:

First, there is no "escape route" for a doctest to indicate that it needs to control transaction boundaries.  This is provided in the unit test alternative by providing a TransactionTestCase that can be used for tests that do not want the test running machinery interfering with its own use of transactions.  So if a unit test needs to control transactions, it can do so by by basing off of TransactionTestCase, in which case  flush/syncdb is used to initialize the DB instead of using the rolled back transaction.  There doesn't seem to be any nice clean way to provide a similar escape route for doctests. 

Second, if we start isolating doctests like this for DBs that support transactions, what do we do in the case(s) where the DB doesn't support transactions?  If we fall back to flush/syncdb as we do for unit tests, that's going to slow down the test suite even more for the MySQL/MyISAM combo.  If we don't fallback to flush/syncdb so as not to incur the performance penalty, then we start introducing a serious divergence in test environment depending on the DB used.  I really don't like either one of these alternatives.

So, the latest patch on #8183 (8138alternate-nodoctestxaction.diff) does not enclose doctest runs in a rolled-back transaction.  Thus the effects of doctests can still bleed over into subsequent tests.  This doesn't actually cause any failures or errors when running the Django test suite, as far as I've seen.  I'm inclined to think it's an OK alternative at least for the short term, but would be interested in hearing other opinions.  (Longer term I'm thinking perhaps something along the lines of Joseph's patch on #5624 for specifying doctest fixtures could be developed into a general mechanism for a doctest to specify some operational needs including fixtures, transactional control, isolation...or that may be overkill.)

Some other to-dos I have on my list:

1 - Documentation is currently completely lacking.
2 - Think some more about TransactionTestCase.  This isn't used by any of the Django tests but I think needs to be provided in case anyone has tests that really do need to make use of transactions internally.  I think, though, there is currently a bit of a problem with mixing TransactionTestCases with TestCases since the former clears the DB on entry but not afterward, so its effects are not cleaned up, while TestCases assume the DB is clean on entry.
3 - Anything else anyone notices?

Karen

[Russell Keith-Magee]

On Tue, Jan 13, 2009 at 4:59 AM, Karen Tracey <kmtr...@gmail.com> wrote:
> One of the item on the list for 1.1 is "Run Django test cases inside a
> transaction". The ticket for this is #8138:

Thanks for picking up on this one, Karen. I thought I would have time
to look at this, but life conspired against me.

> So, the latest patch on #8183 (8138alternate-nodoctestxaction.diff) does not
> enclose doctest runs in a rolled-back transaction. Thus the effects of
> doctests can still bleed over into subsequent tests. This doesn't actually
> cause any failures or errors when running the Django test suite, as far as
> I've seen. I'm inclined to think it's an OK alternative at least for the
> short term, but would be interested in hearing other opinions.

I can't say this concerns me particularly. Doctests are nice and all,
but they aren't a particularly sophisticated testing option. IMHO if
you have complex testing requirements, you should probably be looking
at moving beyond doctests.

> (Longer term
> I'm thinking perhaps something along the lines of Joseph's patch on #5624
> for specifying doctest fixtures could be developed into a general mechanism
> for a doctest to specify some operational needs including fixtures,
> transactional control, isolation...or that may be overkill.)

This is always an option, but it's an option we can add later if the
need arises.

> Some other to-dos I have on my list:
>
> 1 - Documentation is currently completely lacking.
> 2 - Think some more about TransactionTestCase. This isn't used by any of
> the Django tests but I think needs to be provided in case anyone has tests
> that really do need to make use of transactions internally. I think,
> though, there is currently a bit of a problem with mixing
> TransactionTestCases with TestCases since the former clears the DB on entry
> but not afterward, so its effects are not cleaned up, while TestCases assume
> the DB is clean on entry.

I agree that the TransactionTestCase needs to be provided, even if we
aren't using it.

The mixing of TransactionTestCase and TestCase doesn't worry me so
much; using doctests and TestCase requires that you are aware of the
way that they interact; this just introduces a different set of
expectations that we need to manage.

For me, a bigger concern is the change in behaviour of TestCase.
Moving to a transaction-based TestCase needs to be a zero-impact
operation, but under the current patch, it isn't. With the existing
code, a TestCase could assume that the database was clean at the start
of each test; this isn't the case after applying this patch.

> 3 - Anything else anyone notices?

I've seen the messages go back and forth about the timezone problem,
but the solution you have found makes me a little nervous. Modifying
the behavior of normal runtime code to fix test behavior strikes me as
the wrong thing to do. In particular, this modification means that
under Postgres, there is no longer any such thing as a passive read
operation on the database (e.g., retrieving a list of blog entries).
Every database connection will force a commit.

I'm not sure I have a better solution, but you asked if there was
anything I noticed :-)

Yours,
Russ Magee %-)

[Karen Tracey]

On Tue, Jan 13, 2009 at 9:05 AM, Russell Keith-Magee <freakb...@gmail.com> wrote:

> So, the latest patch on #8183 (8138alternate-nodoctestxaction.diff) does not
> enclose doctest runs in a rolled-back transaction.  Thus the effects of
> doctests can still bleed over into subsequent tests.  This doesn't actually
> cause any failures or errors when running the Django test suite, as far as
> I've seen.  I'm inclined to think it's an OK alternative at least for the
> short term, but would be interested in hearing other opinions.

I can't say this concerns me particularly. Doctests are nice and all,
but they aren't a particularly sophisticated testing option. IMHO if
you have complex testing requirements, you should probably be looking
at moving beyond doctests.

OK, so that sounds like one vote for leaving things as they are for doctests, that is with no rolled-back transaction cleaning up after them.

[snip]

 

>2 - Think some more about TransactionTestCase.  This isn't used by any of
> the Django tests but I think needs to be provided in case anyone has tests
> that really do need to make use of transactions internally.  I think,
> though, there is currently a bit of a problem with mixing
> TransactionTestCases with TestCases since the former clears the DB on entry
> but not afterward, so its effects are not cleaned up, while TestCases assume
> the DB is clean on entry.

I agree that the TransactionTestCase needs to be provided, even if we
aren't using it.

The mixing of TransactionTestCase and TestCase doesn't worry me so
much; using doctests and TestCase requires that you are aware of the
way that they interact; this just introduces a different set of
expectations that we need to manage.

For me, a bigger concern is the change in behaviour of TestCase.
Moving to a transaction-based TestCase needs to be a zero-impact
operation, but under the current patch, it isn't. With the existing
code, a TestCase could assume that the database was clean at the start
of each test; this isn't the case after applying this patch.

OK, so what to do?  Sources of pollution on entry to a TestCase using the rollback approach are previously run doctests and (given the way TransactionTestCase is currently written) previously run TransactionTestCases.  The latter could be dealt with by modifying TransactionTestCase to flush/syncdb after the run in addition to (or instead of, if we manage to guarantee clean DB on entry) before the run.  That leaves the doctests again. Some (not fully thought-out) ideas for dealing with them:

1 - Reconsider the idea of cleaning up after each doctest.

2 - I've not looked at the test discovery/running machinery at all.  Maybe there is a way to modify it to run all unit tests, then all doctests instead of interspersing them as it currently does?  That way we could guarantee clean DB on entry to unit tests because no doctests would have been run before the unit tests.

3 - Instead of changing TestCase to use the rollback approach and providing an opt-out in the form of TransactionTestCase, leave TestCase as it is and provide a new (e.g. LightningFastTestCase) option that uses rollback.  Change Django tests to use the new faster testcase, but require people with their own tests to make the decision as to whether they want to switch to it for their own tests.

Of the three #2 sounds most attractive to me, but I have no idea if it is feasible. 

#3 would also protect people from being bitten by invalid assumptions (e.g. primary keys are assigned starting at one) if we switch TestCase to the rollback approach.  Without doing something like #3 I'm not sure we can really guarantee zero-impact of this change for user-written tests. 

> 3 - Anything else anyone notices?

I've seen the messages go back and forth about the timezone problem,
but the solution you have found makes me a little nervous. Modifying
the behavior of normal runtime code to fix test behavior strikes me as
the wrong thing to do. In particular, this modification means that
under Postgres, there is no longer any such thing as a passive read
operation on the database (e.g., retrieving a list of blog entries).
Every database connection will force a commit.

I'm not sure I have a better solution, but you asked if there was
anything I noticed :-)

You are right, the commit of the timezone setting doesn't belong in this patch.  The fact that it is still there in the nodoctestxaction patch is partly laziness, partly a reminder that there may be an independent problem to be looked at here (having the timezone setting vulnerable to rollback seems a bit iffy to me).   If doctests are not enclosed in a transaction then the timezone commit isn't necessary.  So assuming we stick with not enclosing doctests in a transaction, it can be taken out.  If we decide to go back to enclosing doctests in a transaction, then we either need the timezone commit or a way for doctests to opt-out of an enclosing transaction.

Karen

[Jacob Kaplan-Moss]

On Tue, Jan 13, 2009 at 10:00 AM, Karen Tracey <kmtr...@gmail.com> wrote:
> OK, so that sounds like one vote for leaving things as they are for
> doctests, that is with no rolled-back transaction cleaning up after them.

Yeah, I agree. doctests are the 80% testing tool; if you need more
control, that's what formal test cases are for.

> Of the three #2 sounds most attractive to me, but I have no idea if it is
> feasible.

I also agree, and I *think* it ought to be possible by modifying the
way the test runner is constructed during discovery. Failing that, I
think keeping TestCase as it is in 1.0 and introducing a new
FasterTestCase would be the better approach -- that way people have to
opt-in to the new behavior, and can decide if the caveats are worth
it.

Jacob

[Russell Keith-Magee]

#3 sounds like the nicest option to me. #1 introduces even more
slowdowns caused by flush operations; #2 might be possible, but I
suspect it will be a complex solution to implement. #3 is a simple
approach to implement, easy to explain, and opt-in for any existing
usage, so it should be zero impact.

If I'm wrong on the complexity of #2, it might be preferable, but I
don't think it will be easy.

Russ %-)

[Karen Tracey]

OK, I looked into this idea a little more.  It's not too hard in django.test.simple.run_tests to take the full suite once it has been discovered and re-order it so that all of the unit tests that are using rollback are put first.  Then they are guaranteed a clean database since they run first and they clean up after themselves.

This can, however, still run into some problems.  For example the file_uploads test has trouble with this approach since it has  two component tests, one a django.test.TestCase and one a unittest.TestCase.  Ordinarily the unittest.TestCase DirectoryCreationTests runs first.  Pulling the django.test.TestCase FileUploadTests out and running it first results in DirectoryCreationTests failing because it tries to create as a file a directory that is created and not cleaned up by FileUploadTests.

This is easily fixed but highlights that reordering tests from the way they have been running to date may result in unexpected errors.   This, like assumptions about primary keys being assigned starting with one suddenly no longer being true, may make for some nasty surprises for people with their own set of tests if we just change django.test.TestCase to use the rollback approach.  It may be safer to force people to switch to using a new kind of test case if they want to see performance improvements for their tests.  But then, of course, they don't just get the nice performance benefit for free.  Opinions?

Karen

[Russell Keith-Magee]

Each test should be a unit, and should be able to be invoked
independently. If one test is depending on another test being executed
first, that's a problem with the test case that can be provoked right
now by running

$ manage test app.OrderDependentTestCase

so the problem with our test suite needs to be fixed, regardless.

As for making the whole thing opt-in, I'm of two minds. Introducing
potential test failures (however legitimate) as part of an upgrade to
the test system is something I'd rather avoid doing. On the other
hand, this will only affect people that have pre-existing problems
with their test suite (just problems they don't know about), and the
performance improvement is a huge boost that it seems a pity to
relegate it to sitting in the corner until it gets noticed.

I suppose we can manage this with a release note highlighting that
v1.1 will highlight any order dependent tests. Affected users can
replace TestCase with TransactionTestCase to make the problem go away,
or they can fix their tests.

Russ %-)

[Alex Gaynor]

Personally I think anyone with broken tests(and that's wat they are) should just have them break, though testing is a stable API I personally don't think that covers stuff that is broken(as Russ demonstrated it's broken  by running the test by itself).

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire
"The people's good is the highest law."--Cicero

[Karen Tracey]

Just to clarify -- the failure I hit in the Django test suite after re-ordering rolled back test cases to run first is not one where a test assumes another runs before it (rather the reverse) or one that can be triggered currently by just running that single test in isolation.  If you run just file_uploads on today's trunk, you'll see:

python runtests.py --settings=sqlite --verbosity=2 file_uploads
[snipped a bunch of output]
No fixtures found.
The correct IOError is raised when the upload directory name exists but isn't a directory ... ok
Permission errors are not swallowed ... ok
test_broken_custom_upload_handler (regressiontests.file_uploads.tests.FileUploadTests) ... ok
test_custom_upload_handler (regressiontests.file_uploads.tests.FileUploadTests) ... ok
Uploaded file names should be sanitized before ever reaching the view. ... ok
test_file_error_blocking (regressiontests.file_uploads.tests.FileUploadTests) ... ok
File names over 256 characters (dangerous on some platforms) get fixed up. ... ok
test_fileupload_getlist (regressiontests.file_uploads.tests.FileUploadTests) ... ok
test_large_upload (regressiontests.file_uploads.tests.FileUploadTests) ... ok
test_simple_upload (regressiontests.file_uploads.tests.FileUploadTests) ... ok

----------------------------------------------------------------------
Ran 10 tests in 3.392s

OK
Destroying test database...

If you run it, unchanged, with rolled back transactions that are re-ordered to be run first, you see:

python runtests.py --settings=sqlite --verbosity=2 file_uploads
[snipped a bunch of output]
test_broken_custom_upload_handler (regressiontests.file_uploads.tests.FileUploadTests) ... ok
test_custom_upload_handler (regressiontests.file_uploads.tests.FileUploadTests) ... ok
Uploaded file names should be sanitized before ever reaching the view. ... ok
test_file_error_blocking (regressiontests.file_uploads.tests.FileUploadTests) ... ok
File names over 256 characters (dangerous on some platforms) get fixed up. ... ok
test_fileupload_getlist (regressiontests.file_uploads.tests.FileUploadTests) ... ok
test_large_upload (regressiontests.file_uploads.tests.FileUploadTests) ... ok
test_simple_upload (regressiontests.file_uploads.tests.FileUploadTests) ... ok
The correct IOError is raised when the upload directory name exists but isn't a directory ... ERROR
Permission errors are not swallowed ... ok

======================================================================
ERROR: The correct IOError is raised when the upload directory name exists but isn't a directory
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/kmt/tmp/django/trunk/tests/regressiontests/file_uploads/tests.py", line 259, in test_not_a_directory
    fd = open(UPLOAD_TO, 'w')
IOError: [Errno 21] Is a directory: '/tmp/tmpud5FFp/test_upload'

----------------------------------------------------------------------
Ran 10 tests in 1.949s

FAILED (errors=1)

The problem is tests/regressiontests/file_uploads/test.py contains both a unittest.TestCase (named DirectoryCreationTests) and a django.test.TestCase (named FileUploadTests).  Given their names (Directory... sorts before File...), they are run in that order on today's trunk, and in that order they both pass.  If you switch their order (only the FileUploadTests becomes a rolled-back testcase that gets moved to the front of the suite) you get the "The correct IOError is raised when the upload directory name exists but isn't a directory" failure because the FileUploadTests has a (not cleaned up) side-effect of creating a directory where the unittest.TestCase assumes it can create a file. 

I tend to agree that this is a flaw in this test set, and it is easily fixed, but it is not a failure you can see today by just running a test in isolation.  You'd have to do something like change the test names so that the FileUploadTests sorted lexically (and thus ran before) the other test.

All that said, I, personally, would prefer to make the new rolled-back test case replace the existing django.test.TestCase and require people to opt-out (switch to TransactionTestCase) if they want/need the old behavior.  The rollback approach does provide a significant performance boost and it would be nice if people could just take advantage of it by default with 1.1.  I do fear it may be overlooked by some if code changes are required to use it.  But I did want to point out that the effects of the re-ordering done to put the rolled-back transaction test cases first may be a little more subtle than expected, in case that sways anyone's opinion towards requiring opt-in for getting the new behavior.

Karen

[Thomas Guettler]

Thank you Karen for your work.

Some weeks ago I wanted to write an "always_rollback" decorator, but found
that it was not that easy, since Postgres transactions can't be nested. The
first COMMIT commits everything. [1]

I want to run readonly unittests on my live data. I looked at your
patch, and
saw the way you do it: you redefine the transaction methods to "nop".

Are other developers interested in "readonly unitests on live data", too?

Of course you the tests must not alter data on the filesystem something
like this.

Thomas

[1] http://www.mail-archive.com/django...@googlegroups.com/msg58024.html
http://www.nnseek.com/e/comp.databases.postgresql/nested_transactions_49494491t.html

Karen Tracey schrieb:

> One of the item on the list for 1.1 is "Run Django test cases inside a
> transaction". The ticket for this is #8138:
>
> http://code.djangoproject.com/ticket/8138

> ...
>

--
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de

[Ville Säävuori]

> Are other developers interested in "readonly unitests on live data", too?

At our company we mostly test our Django apps with "live" data for
various reasons.

Our process is roughly following: 1. we make a sql-dump of the current
production data (or part of it if it's REALLY big) and move it to the
dev machine. (This is done via Fabric- or capistrano script.) 2. We
create a new test db from the fresh "live" data. 3. We run the tests
with a custom manage.py command that works around some problems in the
current testrunner. This process works quite well for us and it's also
mostly automated, so you just have to run one command on a dev machine
(and in some cases type password or two) to make it all happen.

I've been meaning to write a pony request relating to this but I
haven't found a suitable description for it, yet.

Basically, for our needs, it would be a great help if it would be
possible to use raw sql dumps as fixtures for tests.

- VS

[Eric Holscher]

I would also like to point out a kind of edge case. The Ellington test suite is currently using the approach in #5624[1], where we are injected fixtures into doctests by declaring a __fixtures__ argument inside of a doctest. This is currently being run with a patched Test Runner that applies the fixture (if present), and then calls the doctest.DocTestRunner.run itself. This is having to apply a flush before every call it loaddata (basically mimicking transactions, but much slower).

I think that if there is a plan to ever include fixtures into doctests, then we should put transaction management into them. We should also decide on a syntax (__fixtures__ really isn't too bad). This is mostly a bikeshed, where if it's going to happen, we just need to decide something that works and go with it. Otherwise we should go ahead and close #5624, and say that doctests are only for simple cases where you are generating the objects yourself. This means that we will internally write out doctests that require fixtures back into unit tests, but this isn't a huge deal.

I think we should just decide now, and stick with it. I really don't have a preference, because I don't think that doctests should be used as extensively as they are. I know a big reason they were used before is because they were faster, which now a moot point. If we just say "we will never have fixtures or transactions on doctests", then we can just be done with it.

1: http://code.djangoproject.com/ticket/5624

Cheers,
Eric

--
Eric Holscher
Web Developer at The World Company in Lawrence, Ks
http://ericholscher.com
er...@ericholscher.com

[Russell Keith-Magee]

On Thu, Jan 15, 2009 at 5:40 AM, Eric Holscher <eric.h...@gmail.com> wrote:
> I think that if there is a plan to ever include fixtures into doctests, then
> we should put transaction management into them. We should also decide on a
> syntax (__fixtures__ really isn't too bad). This is mostly a bikeshed, where
> if it's going to happen, we just need to decide something that works and go
> with it. Otherwise we should go ahead and close #5624, and say that doctests
> are only for simple cases where you are generating the objects yourself.
> This means that we will internally write out doctests that require fixtures
> back into unit tests, but this isn't a huge deal.
>
> I think we should just decide now, and stick with it. I really don't have a
> preference, because I don't think that doctests should be used as
> extensively as they are. I know a big reason they were used before is
> because they were faster, which now a moot point. If we just say "we will
> never have fixtures or transactions on doctests", then we can just be done
> with it.
>
> 1: http://code.djangoproject.com/ticket/5624

There is two issues at work here.
1) Is #5625 a good idea?
2) Do we need to implement it right now?

Regardless of the answer to (1), (2) is the more pressing concern -
given that we're days away from a v1.1 feature freeze, and we're
probably behind schedule as it is, now isn't the time to start with
unnecessary feature creep. As far as I can tell, #5625 can be safely
deferred to v1.2 without losing functionality or being boxed into a
corner by #8138. Given that there isn't any urgency brought on by a
need to meet v1.1 commitments, I suggest that we should defer this
discussion to the next planning cycle.

Russ %-)

[Karen Tracey]

I agree.  As it is now #8138 doesn't change the behavior of doctests, so far as I know.  Trying to do the same thing for them as was done for django.test.UnitTest code -- running them in a rolled-back transaction and preventing them from calling transaction routines -- led to more weird errors in the Django test suite.  Given time I'm sure they can be figured out, but I think there's some design work involved in order to come up with a way for them to opt-out of the change, as TestCases can do by switching to TransactionTestCase.  It seems there is similar work to be done for extending them to auto-load fixtures, and it would make sense coordinate that work so that all Django "doctest extensions" have a similar feel, use a consistent mechanism, etc.  But we don't have time to do that for 1.1, I don't think.  But I don't think it's necessary to say at this point we can never do it.

I'm hopeful #8138 can go in (today -- to make the major feature freeze deadline?) for 1.1 pretty much as it is now.  It provides good speedup for Django TestCases.  So far as I know doctests are now unaffected by it -- if I'm missing something there in terms of the latest patch on #8138 breaking doctests please let me know. 

Karen

[Eric Holscher]

You both are indeed correct. I certainly think that the current patch can go in today as presented. The Ellington test suite is passing with a 10x speedup. We can get it to 40x speedup if we change out doctests that load fixtures into unit tests (which probably makes more sense anyway). From 2 hours down to 3 minutes is amazing.

Thanks for all the hard work, and I'm really excited to see this going into Django. Django development will be much improved with this change.

[Karen Tracey]

On Thu, Jan 15, 2009 at 1:12 PM, Eric Holscher <eric.h...@gmail.com> wrote:

You both are indeed correct. I certainly think that the current patch can go in today as presented. The Ellington test suite is passing with a 10x speedup. We can get it to 40x speedup if we change out doctests that load fixtures into unit tests (which probably makes more sense anyway). From 2 hours down to 3 minutes is amazing.

Thanks for all the hard work, and I'm really excited to see this going into Django. Django development will be much improved with this change.

Cool -- I'm glad to hear the Ellington suite has no problems I was unaware of.

Earlier today I added some doc for the changes, and I've been verifying that the full Django test suite (including the aggregation changes) with the latest #8138 patch produces the same results as trunk on sqlite, Oracle, PostgreSQL, MySQL/InnoDB, and MySQL/MyISAM.  The first three are verified, the last two are currently running.

Assuming no problems crop up in the two that are still running, I'd like to check this in tonight -- if there are any objections, or more time is needed for others to review the current patch, please speak up in the next six hours or so. 

Karen