test-refactor update

[Paul McMillan]

I've posted an update with my project status. I made a little progress
this week, converting 7 model tests. In order to meet my goals for
finishing the project, I plan to be converting 25 directories a week
moving forward.

http://thefire.us/archives/450

-Paul

[Russell Keith-Magee]

On Tue, Jun 22, 2010 at 12:24 PM, Paul McMillan <Pa...@mcmillan.ws> wrote:
> I've posted an update with my project status. I made a little progress
> this week, converting 7 model tests. In order to meet my goals for
> finishing the project, I plan to be converting 25 directories a week
> moving forward.

I have two queries:

Firstly, do you have any evidence that this goal is achievable? If
you've converted 7 this week, what makes you think you will be able to
convert 25 next week -- especially when there are tests like
regressiontests/queries still out there? Can you provide stats in
terms of LOC converted, rather than test directories converted?

Secondly, are you tracking performance regressions in the test suite
-- i.e., is the test suite getting slower as a result of converting
from doctests to unittests? Can you share any indicative results?

Yours,
Russ Magee %-)

[Paul McMillan]

> Firstly, do you have any evidence that this goal is achievable? If
> you've converted 7 this week, what makes you think you will be able to
> convert 25 next week -- especially when there are tests like
> regressiontests/queries still out there? Can you provide stats in
> terms of LOC converted, rather than test directories converted?

That's probably a fair point. I don't have LOC counts, but I can work
on it (probably while waiting for tests to run on postgres). However
you slice it, it's a big project. I think I can work faster than I
have been, and I know that I'm becoming more proficient at the task as
I go along.

> Secondly, are you tracking performance regressions in the test suite
> -- i.e., is the test suite getting slower as a result of converting
> from doctests to unittests? Can you share any indicative results?

As I do the conversions, I've been keeping an eye on the runtime for
the individual tests I'm converting. If a test seems to be running
significantly slowly, I'll look for ways to improve that.

As far as runtime results on Python 2.6 and Postgres 8.4:
trunk
668.623s

test-refactor
655.897s

With 2.6 and sqlite3:
trunk
138.168s

test-refactor
124.452s

For my older testbed of Python 2.4 and Postgres 8.3 (which cheats a
bit by running in a vm on a ramdisk):
trunk
215.777s

test-refactor
289.505s

So yes, there have been some performance degradations on some
configurations, but it's slightly faster for the common cases. The
converted tests don't include any flush call removals, which usually
provide significant performance improvements.

-Paul

[Paul McMillan]

I've been ill, hence the late update and lack of progress. Last week I
only committed 2 test updates, but spent significant time tracking
down a fix for #13821 since it was preventing existing tests from
running on Postgres 8.3.

-Paul