Introduce a dynamic process pool for the local test driver (issue 275093002)

[mache...@chromium.org]

Reviewers: Jakob,

Message:
PTAL

Description:
Introduce a dynamic process pool for the local test driver

The new process pool allows adding jobs after testing has been started. It
will
also allow to restructure building the job queue (in a follow up CL), so
that
testing can start instantly while the queue is being built.

Also attempts to clean up the keyboard-interrupt logic. Idea: Only catch
keyboard interrupt once per process at the outermost level. Use
proper "finally"
clauses to clean up everywhere where a keyboard interrupt might occur. Never
turn named exceptions into none-exceptions using anonymous "raise".

TEST=python -m unittest pool_unittest

Please review this at https://codereview.chromium.org/275093002/

SVN Base: https://v8.googlecode.com/svn/branches/bleeding_edge

Affected files (+271, -134 lines):
M tools/run-tests.py
M tools/testrunner/local/commands.py
M tools/testrunner/local/execution.py
A tools/testrunner/local/pool.py
A tools/testrunner/local/pool_unittest.py

[jkum...@chromium.org]

LGTM with comments.


https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/execution.py
File tools/testrunner/local/execution.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/execution.py#newcode88
tools/testrunner/local/execution.py:88: # while the queue is filled.
Have you measured whether that would make a difference? Otherwise let's
drop the plan and the TODO.

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/pool.py
File tools/testrunner/local/pool.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/pool.py#newcode52
tools/testrunner/local/pool.py:52: BUFFER = 100
As discussed, maybe make this worker_count * 4 or something like that?

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/pool.py#newcode55
tools/testrunner/local/pool.py:55: self.workers = workers
nit: without context, "workers" make me expect a list of worker objects.
I'd prefer "num_workers" or "worker_count" (or "jobs" which we use
elsewhere).

https://codereview.chromium.org/275093002/

[mache...@chromium.org]

Done. The total approach is ~0.5 seconds slower. I don't know why yet. The
time
could be won back by starting to test earlier (see below).

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/execution.py
File tools/testrunner/local/execution.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/execution.py#newcode88
tools/testrunner/local/execution.py:88: # while the queue is filled.

It takes 0.5 seconds... I rather leave the todo and make a measurement
in a follow up CL if this can be improved. If not, I'll remove the todo.

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/pool.py
File tools/testrunner/local/pool.py (right):

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/pool.py#newcode52
tools/testrunner/local/pool.py:52: BUFFER = 100

On 2014/05/14 09:07:37, Jakob wrote:
> As discussed, maybe make this worker_count * 4 or something like that?

Done.

https://codereview.chromium.org/275093002/diff/60001/tools/testrunner/local/pool.py#newcode55
tools/testrunner/local/pool.py:55: self.workers = workers

On 2014/05/14 09:07:37, Jakob wrote:
> nit: without context, "workers" make me expect a list of worker
objects. I'd
> prefer "num_workers" or "worker_count" (or "jobs" which we use
elsewhere).

Done.

https://codereview.chromium.org/275093002/

[mache...@chromium.org]

Committed patchset #5 manually as r21310 (presubmit successful).

https://codereview.chromium.org/275093002/