Kyua turns parallel
16 views
Skip to first unread message
Julio Merino
Feb 28, 2015, 3:14:29 PM2/28/15
to kyua-d...@googlegroups.com
Hello!
[ Nicely-formatted version available at
http://julipedia.meroh.net/2015/02/kyua-turns-parallel.html . Posting
here for discussion. ]
-----
After three months of intensive work on Kyua's executor Git branch, I
am happy to announce that the new execution engine, whose crown
feature is the ability to run test cases in parallel, has just landed
in master and passes all self-tests!
You can head over to the commit message for more details on the merge,
read the NEWS entries, and skim throught the history of the executor
branch to understand how this feature has been built.
https://github.com/jmmv/kyua/commit/7197e51b8a86b8f2a031b50f8486b0cd3e952460
https://github.com/jmmv/kyua/blob/7197e51b8a86b8f2a031b50f8486b0cd3e952460/NEWS
One caveat: the history will look surprisingly short for a project
that has spanned over three months. The reason is that, in the
executor branch, I have routinely been using git rebase -i master to
build a reduced set of changes that tell the story behind the
implementation without distracting commits of the kind "Fix this
little bug I forgot about" here and there. An unfortunate side-effect
of this is that the temporal history of the commits makes no sense,
and also that all the work I've been doing is not properly accounted
for in GitHub's nice activity graphs; oh well, I consider a sane
semantical history more important than these tiny details.
Why is this work important? Why is running tests in parallel such a big deal?
First, because the computing industry has fully moved into
multiprocessor systems and thus taking advantage of multiple cores is
something that any piece of modern software should do. As a little fun
fact, Kyua is now able to stress my Mac Mini when running tests,
spinning its fans to the maximum; this did not happen at all before.
But second, and I would say more importantly, because many tests are
not resource-hungry or the system resources they stress do not overlap
the resources used by other tests. For example: a significant number
of tests are penalized by disk and time delays in them, which in turn
cause the whole test suite to run for much longer than it would
otherwise. Parallelization allows these long-running but not-heavy
tests to run without blocking forward progress.
Let me also add that a secondary goal of this work is to optimize the
inner workings of Kyua by reducing the system call overhead. In
particular, eliminating one fork(2) call from every test case has been
an explicit design goal. This especially helps Kyua when running on OS
X, as fork is particularly expensive on the Darwin kernel (yes,
citation needed).
As a very unscientific example: running the Kyua, ATF, Lutok, and shtk
test suites with the old sequential execution engine takes about 2
minutes and 15 seconds in my dual-core FreeBSD virtual machine running
on a dual-core Mac Mini. With the new implementation, the total run
time goes down to 1 minute and 3 seconds using a parallelism setting
of 4. Pretty cool I would say, but your mileage may (will) vary!
Are we done yet?
The merge of the executor branch marks the beginning of a major
restructuring of Kyua's internals. As things are today, only the kyua
test command has been switched to using the new execution engine, and
not fully: only the execution of the test's body and cleanup routines
happen through the executor; listing of test cases still happens as it
did before. Similarly, both kyua list and kyua debug still use the
out-of-process, testers-based, sequential implementation.
Therefore, there is a bunch of tricky work left to be done: the old
test case execution engine (the runner plus the out-of-process
testers) need to be fully removed, which in turn means that their
functionality has to first be integrated into the new executor; there
is a need for a feature to explicitly mark test programs as
"exclusive", which is a prerequisite for tests that modify system-wide
settings (as is done in the FreeBSD test suite); and an important
regression needs to be fixed.
So... if we are not done, why merge now?
Because the current code is complete enough as a first piece of the
whole puzzle. Even if the implementation does not yet meet my personal
quality standards, the behavior of the code is already 80% of the way
to my goal of fully switching to the new execution backend. You, as an
end user, care about the behavior (not so much about the
implementation), so by doing the merge now you can already start
taking advantage of the new parallel execution functionality.
Also, because I am tired of managing a relatively large set of commits
with git rebase -i. At this point, the set of commits that build the
executor provide a good foundation for the code and its design. From
now on, any other improvements to this codebase, such as the addition
of new features or the correction of the existing regressions, should
be properly tracked in the Git history.
And lastly because, at the beginning of 2015, I set myself the
personal goal of getting this code merged by the end of February... so
I just made the deadline! Which reminds me I gotta plan how the year's
timeline looks like to reach Kyua 1.0.
Can I try it?
Of course! Please do!
There is no release available yet, but you can obviously fetch the
code from the GitHub project page and and build it on your own! If you
do that, do not forget to set parallelism=4 (or some other value
greater than 1) in your ~/.kyua/kyua.conf file to enable the new
behavior.
https://github.com/jmmv/kyua/blob/master/INSTALL
In fact, I am not going to cut a new release just yet because some of
the issues mentioned above are of the "release-blocking" severity and
thus must be resolved first. What I am going to do, though, is file
bugs for each known issue so that they can be properly tracked.
Have fun and please share any feedback you may have!
--
Julio Merino / @jmmv
[ Nicely-formatted version available at
http://julipedia.meroh.net/2015/02/kyua-turns-parallel.html . Posting
here for discussion. ]
-----
After three months of intensive work on Kyua's executor Git branch, I
am happy to announce that the new execution engine, whose crown
feature is the ability to run test cases in parallel, has just landed
in master and passes all self-tests!
You can head over to the commit message for more details on the merge,
read the NEWS entries, and skim throught the history of the executor
branch to understand how this feature has been built.
https://github.com/jmmv/kyua/commit/7197e51b8a86b8f2a031b50f8486b0cd3e952460
https://github.com/jmmv/kyua/blob/7197e51b8a86b8f2a031b50f8486b0cd3e952460/NEWS
One caveat: the history will look surprisingly short for a project
that has spanned over three months. The reason is that, in the
executor branch, I have routinely been using git rebase -i master to
build a reduced set of changes that tell the story behind the
implementation without distracting commits of the kind "Fix this
little bug I forgot about" here and there. An unfortunate side-effect
of this is that the temporal history of the commits makes no sense,
and also that all the work I've been doing is not properly accounted
for in GitHub's nice activity graphs; oh well, I consider a sane
semantical history more important than these tiny details.
Why is this work important? Why is running tests in parallel such a big deal?
First, because the computing industry has fully moved into
multiprocessor systems and thus taking advantage of multiple cores is
something that any piece of modern software should do. As a little fun
fact, Kyua is now able to stress my Mac Mini when running tests,
spinning its fans to the maximum; this did not happen at all before.
But second, and I would say more importantly, because many tests are
not resource-hungry or the system resources they stress do not overlap
the resources used by other tests. For example: a significant number
of tests are penalized by disk and time delays in them, which in turn
cause the whole test suite to run for much longer than it would
otherwise. Parallelization allows these long-running but not-heavy
tests to run without blocking forward progress.
Let me also add that a secondary goal of this work is to optimize the
inner workings of Kyua by reducing the system call overhead. In
particular, eliminating one fork(2) call from every test case has been
an explicit design goal. This especially helps Kyua when running on OS
X, as fork is particularly expensive on the Darwin kernel (yes,
citation needed).
As a very unscientific example: running the Kyua, ATF, Lutok, and shtk
test suites with the old sequential execution engine takes about 2
minutes and 15 seconds in my dual-core FreeBSD virtual machine running
on a dual-core Mac Mini. With the new implementation, the total run
time goes down to 1 minute and 3 seconds using a parallelism setting
of 4. Pretty cool I would say, but your mileage may (will) vary!
Are we done yet?
The merge of the executor branch marks the beginning of a major
restructuring of Kyua's internals. As things are today, only the kyua
test command has been switched to using the new execution engine, and
not fully: only the execution of the test's body and cleanup routines
happen through the executor; listing of test cases still happens as it
did before. Similarly, both kyua list and kyua debug still use the
out-of-process, testers-based, sequential implementation.
Therefore, there is a bunch of tricky work left to be done: the old
test case execution engine (the runner plus the out-of-process
testers) need to be fully removed, which in turn means that their
functionality has to first be integrated into the new executor; there
is a need for a feature to explicitly mark test programs as
"exclusive", which is a prerequisite for tests that modify system-wide
settings (as is done in the FreeBSD test suite); and an important
regression needs to be fixed.
So... if we are not done, why merge now?
Because the current code is complete enough as a first piece of the
whole puzzle. Even if the implementation does not yet meet my personal
quality standards, the behavior of the code is already 80% of the way
to my goal of fully switching to the new execution backend. You, as an
end user, care about the behavior (not so much about the
implementation), so by doing the merge now you can already start
taking advantage of the new parallel execution functionality.
Also, because I am tired of managing a relatively large set of commits
with git rebase -i. At this point, the set of commits that build the
executor provide a good foundation for the code and its design. From
now on, any other improvements to this codebase, such as the addition
of new features or the correction of the existing regressions, should
be properly tracked in the Git history.
And lastly because, at the beginning of 2015, I set myself the
personal goal of getting this code merged by the end of February... so
I just made the deadline! Which reminds me I gotta plan how the year's
timeline looks like to reach Kyua 1.0.
Can I try it?
Of course! Please do!
There is no release available yet, but you can obviously fetch the
code from the GitHub project page and and build it on your own! If you
do that, do not forget to set parallelism=4 (or some other value
greater than 1) in your ~/.kyua/kyua.conf file to enable the new
behavior.
https://github.com/jmmv/kyua/blob/master/INSTALL
In fact, I am not going to cut a new release just yet because some of
the issues mentioned above are of the "release-blocking" severity and
thus must be resolved first. What I am going to do, though, is file
bugs for each known issue so that they can be properly tracked.
Have fun and please share any feedback you may have!
--
Julio Merino / @jmmv
Garrett Cooper
Mar 22, 2015, 5:12:52 AM3/22/15
to kyua-d...@googlegroups.com
On Feb 28, 2015, at 12:14, Julio Merino <jm...@meroh.net> wrote:
> Hello!
>
> [ Nicely-formatted version available at
> http://julipedia.meroh.net/2015/02/kyua-turns-parallel.html . Posting
> here for discussion. ]
>
> -----
>
> After three months of intensive work on Kyua's executor Git branch, I
> am happy to announce that the new execution engine, whose crown
> feature is the ability to run test cases in parallel, has just landed
> in master and passes all self-tests!
>
> You can head over to the commit message for more details on the merge,
> read the NEWS entries, and skim throught the history of the executor
> branch to understand how this feature has been built.
Thank you for doing this!
Julio Merino
Mar 23, 2015, 4:50:56 PM3/23/15
to kyua-d...@googlegroups.com
:)
Thank you for doing this!
Have you given it a chance with the FreeBSD test suite maybe? How did things go? (I did but, as I tested this within a VM only, the differences were not that impressive hehe.)
Julio Merino / jmmv@ / Persistent Disk SRE
Reply all
Reply to author
Forward
0 new messages