For the past week, I have been working on a design document detailing
what the requirements of configuration files for Kyua are and the
design decisions I am favoring most to implement these. No code has
been written yet, but I'm eager to do so because the current
configuration modules in Kyua are a real ugly hack.
You can find the design document here:
http://code.google.com/p/kyua/wiki/ConfigurationDesign
Probably, the most controversial point of this document is the usage
of Lua, but I believe it is a nice option to get going. We can always
revisit this at a later stage, but I'd rather not spend a lot of time
writing custom parsers *now* because there are way more exciting
things yet to work on ;-)
Comments welcome!
--
Julio Merino <jm...@google.com>
Google Ireland Ltd., Gordon House, Barrow Street, Dublin 4, Ireland
Registered in Dublin, Ireland; Registration Number: 368047
On 22 Δεκ, 11:40, Julio Merino <j...@google.com> wrote:One of the things that strikes me as particularly useful at this stage
> Hello,
>
> For the past week, I have been working on a design document detailing
> what the requirements of configuration files for Kyua are and the
> design decisions I am favoring most to implement these. No code has
> been written yet, but I'm eager to do so because the current
> configuration modules in Kyua are a real ugly hack.
>
> You can find the design document here:
>
> http://code.google.com/p/kyua/wiki/ConfigurationDesign
>
> Probably, the most controversial point of this document is the usage
> of Lua, but I believe it is a nice option to get going. We can always
> revisit this at a later stage, but I'd rather not spend a lot of time
> writing custom parsers *now* because there are way more exciting
> things yet to work on ;-)
>
> Comments welcome!
of the configuration design is that "kyua.suite" seems a bit
odd. "Atffile" is similar to the "Makefile" tradition, and it doesn't
have an extension at all. I'd probably prefer something like
"Kyuafile"
or at least an option like "kyua -f filename" that will be passed
recursively down to subdirs.
I'm still reading the rest of the config design. Back soon(-ish) with
any other comments.
I've noticed that converting test cases from the old ad-hoc format of FreeBSD's test suites to shell-based ATF test cases, there was a very noticeable slow-down. For example this code:
when converted to ATF test cases became:
Putting aside the code bloat from expanding the test cases to separate shell code for each test, there was another immediate result that irked me and is still unresolved in this bit of code. The runtime of the test-driving shell script shot up from 2 seconds to 41 seconds. AFAICT so far most of the time seems to be spent inside the shell code of the atf_init_test_cases() bits, and in the way each test case is run in a separate directory of its own.
So shell code for test cases is really really slow.
But we do need to be able to run arbitrary shell commands for test cases, e.g. to test utilities like sh(1) and test(1). With the proper insulation from previous test cases, or environment variables, filesystem location, etc.
» File layoutThe resolution order for the Kyua runtime configuration is:
Things are a bit different when specifying the configuration of the test suite variables:
Hi Julio & everyone else,
Attached below some notes I've been keeping while I was going through the configuration design document: http://code.google.com/p/kyua/wiki/ConfigurationDesign
» Configuration for every test suite
List of the test programs to run (and optionally subsets of their test cases). It must be possible to specify these test programs by their name or by a glob. (We probably want to use globs rather than regular expressions when referring to files to avoid confusion.)
It's probably ok to support both glob and regexp format for test suites. The Mercurial SCM people had a similar discussion for ".hgignore" files. Now the format of .hgignore files supports both glob-style and regexp-style file patterns, and the behavior is tunable with an option, e.g.:
At that point we may as well support regexp-based file matching too, as an optional test case filename pattern. We can probably even refactor away most of the glob matching code by converting globs to regexps internally, and use only regexp matching for the actual filename selection code.
» Decision: Kyua will have one single format for all external configuration files.
This is a nice decision. I've been watching the efforts to integrate Lua code with the main Kyua code, to make it the "officially sanctioned extension language". If the config files are Lua code that runs in its own namespace, I'd really love to see what interesting configuration modules people will come up with :-)
» Parser ImplementationShell is not really an option: ATF chose to implement hooks this way and, while it was a good idea at the time, the results have proven it was not. Shell scripts have too much external dependencies (binaries, environment variables, ...), they are noticeably slow, the syntax is too fragile and there is no way to provide a clean interface between the native code and the interpreted code.I've noticed that converting test cases from the old ad-hoc format of FreeBSD's test suites to shell-based ATF test cases, there was a very noticeable slow-down. For example this code:
t()
{
run test command on "$@" args
}
t 0 'b = b'
t 1 'b != b'
when converted to ATF test cases became:
Putting aside the code bloat from expanding the test cases to separate shell code for each test,
there was another immediate result that irked me and is still unresolved in this bit of code. The runtime of the test-driving shell script shot up from 2 seconds to 41 seconds. AFAICT so far most of the time seems to be spent inside the shell code of the atf_init_test_cases() bits, and in the way each test case is run in a separate directory of its own.
So shell code for test cases is really really slow.
But we do need to be able to run arbitrary shell commands for test cases, e.g. to test utilities like sh(1) and test(1). With the proper insulation from previous test cases, or environment variables, filesystem location, etc.
» File layout
It's a bit unclear what happens when both <home>/.kyua/kyua.conf and <sysconfdir>/kyua/kyua.conf exist. Will we load the <home> file and stop at that point? Will we load both? The text seems to imply that we load only the <home> file, but we should make sure this is clearly stated in the documentation, possibly by listing the paths we will look for config files in the same order that they will be looked up (now it's listed in reverse order, AFAICT).
» That's all
I don't have any other comments for now. I like the direction of using Lua for parsing config files, options, etc. We will probably see lots of interesting kyua modules & extensions if it's easy to hook into the existing API and "do stuff" :-)
"I do not see coincidence, I see providence."
Why not just keep things simple and stick to the rules defined by
fnmatch(3) [1]? Word of caution: according to kientzle@ there were
some issues with portability and functionality with fnmatch(3)
according to tar(1)'s requirements, so he basically ripped bits out of
libc, monkeyed with it a bit, and put it in libarchive.
...
> I really want to get rid of such boilerplate code (but I don't have ideas
> yet as to how to do it yet). After having written dozens of pyunit-based
> tests while at Google, I can't stress how nice it is to be able to define
> tests in a concise way. Loads of fine-grained tests == good (but the atf
> api kinda goes against this).
*puts on dusty Cisco QA/LTP hat*
Speaking from past experience, it makes sense to have a generalized
infrastructure that's portable anywhere and shim layers elsewhere to
make life easier in less portable cases (like with the BSDs for
instance). So I would make Kyua fast and general with as much useful
utility as possible without bloating it, and we (the BSD folks) can
pull our heads together and come up with less portable bits (but
portable within the BSDs) for testing out specific items. Granted, it
would probably require some consensus on format, way to do things,
etc, but I think that the 3 main BSDs (and their derivatives) would be
happy to have generalized tests that can port between the
distributions easily.
As time goes on bits from the BSD infrastructure could be ported into
Kyua as necessary if it's portable enough and makes sense to do so,
but probably as add-ons to the project.
>> there was another immediate result that irked me and is still unresolved
>> in this bit of code. The runtime of the test-driving shell script shot up
>> from 2 seconds to 41 seconds. AFAICT so far most of the time seems to be
>> spent inside the shell code of the atf_init_test_cases() bits, and in the
>> way each test case is run in a separate directory of its own.
>
> Wow, that's... surprising. How did you measure that? Is this running the
> test programs directly (./foo_test streq) or through atf-run? I have
> certainly seen a slowdown, but a blow up of 20x seems too much.
A ktrace run / some form of tracing needs to be done on the function
to determine why stuff is so slow because that's a lot of incurred
overhead :o... a recent copy of FreeBSD CURRENT might help too (there
were some shell bugs that were resolved recently that have been
present in FreeBSD since at least 7.x).
>> So shell code for test cases is really really slow.
>>
>> But we do need to be able to run arbitrary shell commands for test cases,
>> e.g. to test utilities like sh(1) and test(1). With the proper insulation
>> from previous test cases, or environment variables, filesystem location,
>> etc.
>
> Yes indeed. At some point I want to have bindings to write the test
> programs themselves in Lua (moving away from atf-sh as much as possible).
You're referring to the wrapper scripts which execute the tests though
and not the tests themselves, right :/? Requiring folks to learn Lua
to write testcases seems to put the bar up higher than it needs to be;
granted, it's a simple language from what I've seen, but I know a
kernel dev that hates to write shell code, and I doubt he would be
willing to learn Lua.
>> » File layout
>
> [...]
>>
>> It's a bit unclear what happens when both <home>/.kyua/kyua.conf and
>> <sysconfdir>/kyua/kyua.conf exist. Will we load the <home> file and stop at
>> that point? Will we load both? The text seems to imply that we load only the
>> <home> file, but we should make sure this is clearly stated in the
>> documentation, possibly by listing the paths we will look for config files
>> in the same order that they will be looked up (now it's listed in reverse
>> order, AFAICT).
>
> In atf, user-specific configuration files inherit the system-wide ones. I
> will have to think further on what the best approach is though... because
> inheriting system-wide files by default makes it quite hard to ensure your
> custom file does not bring in global definitions that might change over
> time.
> Civen that we are dealing with scripts, we can make the default to *not*
> inherit the system-wide settings and, if the user so desires, he can just
> include the system settings.
> Will add these thoughts to the design doc.
I think that having user settings masking global settings would be
better to match standard program behavior, but maybe an `override'
option should be added for anal retentive testcases that would be
affected a great deal by change in behavior when using Kyua (say for
folks doing a nightly regression, etc)?
>> » That's all
>>
>> I don't have any other comments for now. I like the direction of using Lua
>> for parsing config files, options, etc. We will probably see lots of
>> interesting kyua modules & extensions if it's easy to hook into the existing
>> API and "do stuff" :-)
>
> Yep, that would indeed be nice :-)
> Thanks for your detailed comments!
> PS: At the moment, there is virtually no subscribers to this mailing list.
> I expect we will eventually get people from multiple BSD backgrounds so we
> should avoid HTML emails.
<sarcasm>
As long as everything is kept down to 72 chars or less, Theo will be happy :).
</sarcasm>
I agree that HTML emails are ugly though..
Thanks!
-Garrett
1. http://www.opengroup.org/onlinepubs/000095399/functions/fnmatch.html
Basically because I was not aware of it.
> Word of caution: according to kientzle@ there were
> some issues with portability and functionality with fnmatch(3)
> according to tar(1)'s requirements, so he basically ripped bits out of
> libc, monkeyed with it a bit, and put it in libarchive.
That's a serious drawback. (FWIW atf currently implements a glob->re
mapping scheme. It supports only ? and * but seems to have been enough
so far!)
My current idea for implementing, e.g. the top-level Kyuafile that has
to recurse into a subset of directories not known beforehand is
something along the lines of:
for file in fs.directory('.') do
if fs.is_directory(file) then
local subfile = fs.join(file, 'Kyuafile')
if fs.exists(subfile) then
include(subfile)
end
end
end
You could also do something like this to add all *_test files:
for file in fs.directory('.') do
if fnmatch('*_test', file) then
AtfTestProgram {name=file}
end
end
While the above examples are way more verbose than their current atf
counterparts, I feel that they are clearer in intent and that the
transparency that they offer is a win. (Plus, if proven necessary due
to popular demand, it is extremely easy to write wrapping functions for
the code above.)
> > I really want to get rid of such boilerplate code (but I don't have ideas
> > yet as to how to do it yet). ?After having written dozens of pyunit-based
> > tests while at Google, I can't stress how nice it is to be able to define
> > tests in a concise way. ?Loads of fine-grained tests == good (but the atf
> > api kinda goes against this).
>
> *puts on dusty Cisco QA/LTP hat*
>
> Speaking from past experience, it makes sense to have a generalized
> infrastructure that's portable anywhere and shim layers elsewhere to
> make life easier in less portable cases (like with the BSDs for
> instance). So I would make Kyua fast and general with as much useful
> utility as possible without bloating it, and we (the BSD folks) can
> pull our heads together and come up with less portable bits (but
> portable within the BSDs) for testing out specific items. Granted, it
> would probably require some consensus on format, way to do things,
> etc, but I think that the 3 main BSDs (and their derivatives) would be
> happy to have generalized tests that can port between the
> distributions easily.
>
> As time goes on bits from the BSD infrastructure could be ported into
> Kyua as necessary if it's portable enough and makes sense to do so,
> but probably as add-ons to the project.
I think that's a nice idea and, actually, it kinda is in atf too. I
have refused several times to provide NetBSD-specific mechanisms in the
upstream version of the code (links to PRs is the one that comes to
mind now). The problem is... atf is not really extensible by third
parties to add the necessary "shim layers" for customization that you
mention.
As a side note, take a look at:
This is NetBSD's attempt of providing a subset of common functionality
useful within the realm of NetBSD tests. Some of such features could be
ported upstream to atf. (There is more stuff hidden in some other
directory of src/tests; in particular, for fs tests.)
> A ktrace run / some form of tracing needs to be done on the function
> to determine why stuff is so slow because that's a lot of incurred
> overhead :o... a recent copy of FreeBSD CURRENT might help too (there
> were some shell bugs that were resolved recently that have been
> present in FreeBSD since at least 7.x).
One thing I forgot to mention... what version of atf was that with?
Versions before 0.7 (IIRC) were incredibly slow -- but those are over a
year old already!
> > Yes indeed. ?At some point I want to have bindings to write the test
> > programs themselves in Lua (moving away from atf-sh as much as possible).
>
> You're referring to the wrapper scripts which execute the tests though
> and not the tests themselves, right :/? Requiring folks to learn Lua
> to write testcases seems to put the bar up higher than it needs to be;
> granted, it's a simple language from what I've seen, but I know a
> kernel dev that hates to write shell code, and I doubt he would be
> willing to learn Lua.
I was referring to both: the wrapper scripts and the tests themselves.
But I do not plan to kill atf-sh. I just want to offer an alternative
set of bindings so that a developer can write his tests in Lua if he so
wishes. The sh-based interface shall remain!
> I think that having user settings masking global settings would be
> better to match standard program behavior, but maybe an `override'
> option should be added for anal retentive testcases that would be
> affected a great deal by change in behavior when using Kyua (say for
> folks doing a nightly regression, etc)?
I think my prevous explanation was confusing (based on your response,
which confuses me ;-)
The approach I think makes more sense is:
* If you don't have ~/.foorc, kyua reads /etc/foo.conf and that's it.
* If you have ~/.foorc, then kyua reads ~/.foorc only and does _not_
read /etc/foo.conf. (But, if you wish, you can just add
'include /etc/foo.conf' or similar in your ~/.foorc if you desire to
import the global settings.) Pretty much like how .shrc behaves.
Does that make more sense?
Thanks for your comments!
Yeah, they probably aren't really usable in an infrastructure (it's
like saying you have a shiny Ferrari, but you have to drive 20MPH
because those are the limitations you have to deal with).
Honestly, as long as the semantics match fnmatch(3) within reason, it
should be ok to use whatever implementation necessary to achieve that
goal. fnmatch(3) just meshes well with other systems (BSD, Linux,
Windows, etc), so that's the reason why I suggested it first and
foremost.
Well... that's reasonably small in comparison to the kitchen sink that
libltp is in LTP today. If I was ambitious enough I would remove
interfaces and refactor code, but I don't feel like rototilling that
project more than I already have in the past couple of months.
Along those lines, that's why I suggest keeping Kyua minimal and
maximize things which are required at bare minimum to get the job
done. Otherwise this project will become another kitchen sink type
project.
>> A ktrace run / some form of tracing needs to be done on the function
>> to determine why stuff is so slow because that's a lot of incurred
>> overhead :o... a recent copy of FreeBSD CURRENT might help too (there
>> were some shell bugs that were resolved recently that have been
>> present in FreeBSD since at least 7.x).
>
> One thing I forgot to mention... what version of atf was that with?
> Versions before 0.7 (IIRC) were incredibly slow -- but those are over a
> year old already!
>
>> > Yes indeed. ?At some point I want to have bindings to write the test
>> > programs themselves in Lua (moving away from atf-sh as much as possible).
>>
>> You're referring to the wrapper scripts which execute the tests though
>> and not the tests themselves, right :/? Requiring folks to learn Lua
>> to write testcases seems to put the bar up higher than it needs to be;
>> granted, it's a simple language from what I've seen, but I know a
>> kernel dev that hates to write shell code, and I doubt he would be
>> willing to learn Lua.
>
> I was referring to both: the wrapper scripts and the tests themselves.
> But I do not plan to kill atf-sh. I just want to offer an alternative
> set of bindings so that a developer can write his tests in Lua if he so
> wishes. The sh-based interface shall remain!
Good to hear. As long as everything's binary compatible / communicable
then we should be able to use whatever language suitable to drive
tests with via C-type hooks/extensions.
>> I think that having user settings masking global settings would be
>> better to match standard program behavior, but maybe an `override'
>> option should be added for anal retentive testcases that would be
>> affected a great deal by change in behavior when using Kyua (say for
>> folks doing a nightly regression, etc)?
>
> I think my prevous explanation was confusing (based on your response,
> which confuses me ;-)
>
> The approach I think makes more sense is:
>
> * If you don't have ~/.foorc, kyua reads /etc/foo.conf and that's it.
>
> * If you have ~/.foorc, then kyua reads ~/.foorc only and does _not_
> read /etc/foo.conf. (But, if you wish, you can just add
> 'include /etc/foo.conf' or similar in your ~/.foorc if you desire to
> import the global settings.) Pretty much like how .shrc behaves.
>
> Does that make more sense?
What I meant is:
. /etc/foo.conf
. ~/.foorc
Values in foo.conf are superseded by values in .foorc . Unless you say
--norc, etc in which case it would function like bash --norc:
--norc Do not read and execute the personal initialization file
~/.bashrc if the shell is interactive. This option is on by
default if the shell is invoked as sh.
Thanks!
-Garrett
If demand for iterating through a directory and pattern-matching all
files before creating AtfTestProgram{} for them is sufficiently high, it
should be relatively easy to wrap these in a function or provide a
Lua-based path walker e.g.:
function testmatch(file, pattern)
if kyua.fs.fnmatch(pattern, file)
AtfTestProgram{name = file}
end
end
kyua.path.walk('.', {func = testmatch, arg = '*_test'})
> I think my prevous explanation was confusing (based on your response,
> which confuses me ;-)
>
> The approach I think makes more sense is:
>
> * If you don't have ~/.foorc, kyua reads /etc/foo.conf and that's it.
>
> * If you have ~/.foorc, then kyua reads ~/.foorc only and does _not_
> read /etc/foo.conf. (But, if you wish, you can just add
> 'include /etc/foo.conf' or similar in your ~/.foorc if you desire to
> import the global settings.) Pretty much like how .shrc behaves.
>
> Does that make more sense?
>
> Thanks for your comments!
Yes, that makes indeed a lot more sense, and it *permits* including the
system-wide configuration when necessary.
I'd like this to be the default for config file loading, so that users
can opt out of brain-damaged system defaults, which some distributors
may think it's a wise idea to force down the throad of everybody (the
'clear' command that wipes terminal scrollback from the default copy of
/etc/bash.bash.logout in Linux distribution packages is the canonical
example of this for me).