Design of configuration files and test suite definitions

13 views
Skip to first unread message

Julio Merino

unread,
Dec 22, 2010, 5:40:16 AM12/22/10
to kyua-d...@googlegroups.com
Hello,

For the past week, I have been working on a design document detailing
what the requirements of configuration files for Kyua are and the
design decisions I am favoring most to implement these. No code has
been written yet, but I'm eager to do so because the current
configuration modules in Kyua are a real ugly hack.

You can find the design document here:

    http://code.google.com/p/kyua/wiki/ConfigurationDesign

Probably, the most controversial point of this document is the usage
of Lua, but I believe it is a nice option to get going.  We can always
revisit this at a later stage, but I'd rather not spend a lot of time
writing custom parsers *now* because there are way more exciting
things yet to work on ;-)

Comments welcome!

--
Julio Merino <jm...@google.com>

Google Ireland Ltd., Gordon House, Barrow Street, Dublin 4, Ireland
Registered in Dublin, Ireland; Registration Number: 368047

Giorgos Keramidas

unread,
Dec 22, 2010, 2:42:09 PM12/22/10
to kyua-discuss
On 22 Δεκ, 11:40, Julio Merino <j...@google.com> wrote:
> Hello,
>
> For the past week, I have been working on a design document detailing
> what the requirements of configuration files for Kyua are and the
> design decisions I am favoring most to implement these. No code has
> been written yet, but I'm eager to do so because the current
> configuration modules in Kyua are a real ugly hack.
>
> You can find the design document here:
>
>     http://code.google.com/p/kyua/wiki/ConfigurationDesign
>
> Probably, the most controversial point of this document is the usage
> of Lua, but I believe it is a nice option to get going.  We can always
> revisit this at a later stage, but I'd rather not spend a lot of time
> writing custom parsers *now* because there are way more exciting
> things yet to work on ;-)
>
> Comments welcome!

One of the things that strikes me as particularly useful at this stage
of the configuration design is that "kyua.suite" seems a bit
odd. "Atffile" is similar to the "Makefile" tradition, and it doesn't
have an extension at all. I'd probably prefer something like
"Kyuafile"
or at least an option like "kyua -f filename" that will be passed
recursively down to subdirs.

I'm still reading the rest of the config design. Back soon(-ish) with
any other comments.

- Giorgos

Julio Merino

unread,
Dec 22, 2010, 2:58:17 PM12/22/10
to kyua-d...@googlegroups.com
On Wed, Dec 22, 2010 at 7:42 PM, Giorgos Keramidas <gkera...@gmail.com> wrote:
On 22 Δεκ, 11:40, Julio Merino <j...@google.com> wrote:
> Hello,
>
> For the past week, I have been working on a design document detailing
> what the requirements of configuration files for Kyua are and the
> design decisions I am favoring most to implement these. No code has
> been written yet, but I'm eager to do so because the current
> configuration modules in Kyua are a real ugly hack.
>
> You can find the design document here:
>
>     http://code.google.com/p/kyua/wiki/ConfigurationDesign
>
> Probably, the most controversial point of this document is the usage
> of Lua, but I believe it is a nice option to get going.  We can always
> revisit this at a later stage, but I'd rather not spend a lot of time
> writing custom parsers *now* because there are way more exciting
> things yet to work on ;-)
>
> Comments welcome!

One of the things that strikes me as particularly useful at this stage
of the configuration design is that "kyua.suite" seems a bit
odd. "Atffile" is similar to the "Makefile" tradition, and it doesn't
have an extension at all.  I'd probably prefer something like
"Kyuafile"

The names are not definitive at all at this point.  If Kyuafile makes people happier, we can go that route.  It's just that I personally 1) don't like the capitalization (because it's inconsistent with pretty much everything else except for Makefiles) and 2) I always found the Atffile name odd (a double f?), but that's about it :-)
 
or at least an option like "kyua -f filename" that will be passed
recursively down to subdirs.

There will definitely be a -f option so you can specify the filename.  See more details on the recursion vs. inclusion section.
 
I'm still reading the rest of the config design. Back soon(-ish) with
any other comments.

Thanks for taking a look! 

Giorgos Keramidas

unread,
Jan 23, 2011, 7:18:25 AM1/23/11
to kyua-d...@googlegroups.com
Hi Julio & everyone else,

Attached below some notes I've been keeping while I was going through the configuration design document: http://code.google.com/p/kyua/wiki/ConfigurationDesign

»
Configuration for every test suite

List of the test programs to run (and optionally subsets of their test cases). It must be possible to specify these test programs by their name or by a glob. (We probably want to use globs rather than regular expressions when referring to files to avoid confusion.)

It's probably ok to support both glob and regexp format for test suites. The Mercurial SCM people had a similar discussion for ".hgignore" files. Now the format of .hgignore files supports both glob-style and regexp-style file patterns, and the behavior is tunable with an option, e.g.:

syntax: glob
**/*.c

a similar ignore file, using regexp format patterns is:

syntax: regexp
.*/.*\.c

Glob formatted tests are ok for one path. They are ok if we know the subdir paths in advance too. But if we want, in the future, to support test matching patterns like "all the '*_test' files in all the subdirectories of the './regression/' subdir" we will have to invent glob-style expressions like:

./regression/.../*_test
./regression/**/*_test

with "..." or "**" for subdir-traversal.

At that point we may as well support regexp-based file matching too, as an optional test case filename pattern. We can probably even refactor away most of the glob matching code by converting globs to regexps internally, and use only regexp matching for the actual filename selection code.

» Decision: Kyua will have one single format for all external configuration files.

This is a nice decision. I've been watching the efforts to integrate Lua code with the main Kyua code, to make it the "officially sanctioned extension language". If the config files are Lua code that runs in its own namespace, I'd really love to see what interesting configuration modules people will come up with :-)

» Parser Implementation

Shell is not really an option: ATF chose to implement hooks this way and, while it was a good idea at the time, the results have proven it was not. Shell scripts have too much external dependencies (binaries, environment variables, ...), they are noticeably slow, the syntax is too fragile and there is no way to provide a clean interface between the native code and the interpreted code.

I've noticed that converting test cases from the old ad-hoc format of FreeBSD's test suites to shell-based ATF test cases, there was a very noticeable slow-down.  For example this code:

t()
{
    run test command on "$@" args
}

t 0 'b = b'
t 1 'b != b'

when converted to ATF test cases became:

atf_test_case streq
streq_head()
{
    atf_set "descr" "Test string equality predicate"
}
streq_body()
{
    atf_check -s 'exit:0' 'test' b = b
}
atf_init_test_cases()
{
    atf_add_test_case streq
}
 
atf_test_case notstrneq
notstrneq_head()
{
    atf_set "descr" "Test double-negative string equality predicate"
}
notstrneq_body()
{
    atf_check -s 'not-exit:0' 'test' b '!=' b
}

#
# Register all the test cases of this shell script.
#
atf_init_test_cases()
{
    atf_add_test_case streq
    atf_add_test_case notstrneq
}

Putting aside the code bloat from expanding the test cases to separate shell code for each test, there was another immediate result that irked me and is still unresolved in this bit of code. The runtime of the test-driving shell script shot up from 2 seconds to 41 seconds. AFAICT so far most of the time seems to be spent inside the shell code of the atf_init_test_cases() bits, and in the way each test case is run in a separate directory of its own.

So shell code for test cases is really really slow.

But we do need to be able to run arbitrary shell commands for test cases, e.g. to test utilities like sh(1) and test(1). With the proper insulation from previous test cases, or environment variables, filesystem location, etc.

» File layout

The resolution order for the Kyua runtime configuration is:

  1. Load one of:
    • The user-specific configuration file from <home>/.kyua/kyua.conf if it exists.
    • Otherwise the system-wide configuration file from <sysconfdir>/kyua/kyua.conf if it exists.
  2. Apply any overrides specified through the command line. E.g. kyua test -v kyua.architecture=amd64.

Things are a bit different when specifying the configuration of the test suite variables:

  1. Load the test suite configuration file (kyua.suite) to get the test suite name and the default values for the variables (if any).
  2. Apply any overrides defined in one of:
    • The user-specific configuration file from <home>/.kyua/suites/<suite>.conf if it exists.
    • Otherwise the system-wide configuration file from <sysconfdir>/kyua/suites/<suite>.conf if it exists.
  3. Apply any overrides specified through the command line. E.g. kyua test -v suite.netbsd.file_systems=ffs.
It's a bit unclear what happens when both <home>/.kyua/kyua.conf and <sysconfdir>/kyua/kyua.conf exist. Will we load the <home> file and stop at that point? Will we load both? The text seems to imply that we load only the <home> file, but we should make sure this is clearly stated in the documentation, possibly by listing the paths we will look for config files in the same order that they will be looked up (now it's listed in reverse order, AFAICT).

» That's all

I don't have any other comments for now. I like the direction of using Lua for parsing config files, options, etc. We will probably see lots of interesting kyua modules & extensions if it's easy to hook into the existing API and "do stuff" :-)

--

Giorgos Keramidas; gkera...@gmail.com

Julio Merino

unread,
Jan 23, 2011, 2:24:42 PM1/23/11
to kyua-d...@googlegroups.com
On Sun, Jan 23, 2011 at 12:18 PM, Giorgos Keramidas <gkera...@gmail.com> wrote:
Hi Julio & everyone else,

Hello Giorgos,

Your email cames right on time!  I have just submitted r49 to the repository, which implements basic processing of Lua-based Kyua-files (aka, test suite definitions).  All the initial work of integration with Lua is done (that is, me learning how things may work and creating the basic boilerplate code), so further improvements should be more iterative.

Coincidence? ;-)
 
Attached below some notes I've been keeping while I was going through the configuration design document: http://code.google.com/p/kyua/wiki/ConfigurationDesign

Comments inline. 

» Configuration for every test suite

List of the test programs to run (and optionally subsets of their test cases). It must be possible to specify these test programs by their name or by a glob. (We probably want to use globs rather than regular expressions when referring to files to avoid confusion.)

It's probably ok to support both glob and regexp format for test suites. The Mercurial SCM people had a similar discussion for ".hgignore" files. Now the format of .hgignore files supports both glob-style and regexp-style file patterns, and the behavior is tunable with an option, e.g.:
[...] 
At that point we may as well support regexp-based file matching too, as an optional test case filename pattern. We can probably even refactor away most of the glob matching code by converting globs to regexps internally, and use only regexp matching for the actual filename selection code.

My only concern is that inventing a new glob pattern might cause confusion.  I'd prefer to keep such patterns simple to prevent "abuses" like the ** recursion you mention.

Given that we are dealing with Lua scripts, we may as well expose primitives to iterate over file sets (something as simple as readdir) and regexp matching.  Couple that with some fancy wrapping functions that take the patterns you describe and translate them into the primitives and we can cover pretty much any use case.

» Decision: Kyua will have one single format for all external configuration files.

This is a nice decision. I've been watching the efforts to integrate Lua code with the main Kyua code, to make it the "officially sanctioned extension language". If the config files are Lua code that runs in its own namespace, I'd really love to see what interesting configuration modules people will come up with :-)
 

Are you talking about configuration files here or about writing test programs in Lua itself?  Both are worthy goals, but I have omitted the latter on purpose for now.

» Parser Implementation

Shell is not really an option: ATF chose to implement hooks this way and, while it was a good idea at the time, the results have proven it was not. Shell scripts have too much external dependencies (binaries, environment variables, ...), they are noticeably slow, the syntax is too fragile and there is no way to provide a clean interface between the native code and the interpreted code.

I've noticed that converting test cases from the old ad-hoc format of FreeBSD's test suites to shell-based ATF test cases, there was a very noticeable slow-down.  For example this code:

t()
{
    run test command on "$@" args
}

t 0 'b = b'
t 1 'b != b'

when converted to ATF test cases became:

[...] 
Putting aside the code bloat from expanding the test cases to separate shell code for each test,

I really want to get rid of such boilerplate code (but I don't have ideas yet as to how to do it yet).  After having written dozens of pyunit-based tests while at Google, I can't stress how nice it is to be able to define tests in a concise way.  Loads of fine-grained tests == good (but the atf api kinda goes against this).

there was another immediate result that irked me and is still unresolved in this bit of code. The runtime of the test-driving shell script shot up from 2 seconds to 41 seconds. AFAICT so far most of the time seems to be spent inside the shell code of the atf_init_test_cases() bits, and in the way each test case is run in a separate directory of its own.

Wow, that's... surprising.  How did you measure that?  Is this running the test programs directly (./foo_test streq) or through atf-run?  I have certainly seen a slowdown, but a blow up of 20x seems too much.

So shell code for test cases is really really slow.

But we do need to be able to run arbitrary shell commands for test cases, e.g. to test utilities like sh(1) and test(1). With the proper insulation from previous test cases, or environment variables, filesystem location, etc.

Yes indeed.  At some point I want to have bindings to write the test programs themselves in Lua (moving away from atf-sh as much as possible).
 
» File layout

[...] 

It's a bit unclear what happens when both <home>/.kyua/kyua.conf and <sysconfdir>/kyua/kyua.conf exist. Will we load the <home> file and stop at that point? Will we load both? The text seems to imply that we load only the <home> file, but we should make sure this is clearly stated in the documentation, possibly by listing the paths we will look for config files in the same order that they will be looked up (now it's listed in reverse order, AFAICT).


In atf, user-specific configuration files inherit the system-wide ones.  I will have to think further on what the best approach is though... because inheriting system-wide files by default makes it quite hard to ensure your custom file does not bring in global definitions that might change over time.

Civen that we are dealing with scripts, we can make the default to *not* inherit the system-wide settings and, if the user so desires, he can just include the system settings.

Will add these thoughts to the design doc.

» That's all

I don't have any other comments for now. I like the direction of using Lua for parsing config files, options, etc. We will probably see lots of interesting kyua modules & extensions if it's easy to hook into the existing API and "do stuff" :-)

Yep, that would indeed be nice :-)

Thanks for your detailed comments!

PS: At the moment, there is virtually no subscribers to this mailing list.  I expect we will eventually get people from multiple BSD backgrounds so we should avoid HTML emails.

Garrett Cooper

unread,
Jan 23, 2011, 3:52:22 PM1/23/11
to kyua-d...@googlegroups.com
On Sun, Jan 23, 2011 at 11:24 AM, Julio Merino <jm...@google.com> wrote:
> On Sun, Jan 23, 2011 at 12:18 PM, Giorgos Keramidas <gkera...@gmail.com>
> wrote:
>>
>> Hi Julio & everyone else,
>
> Hello Giorgos,
> Your email cames right on time!  I have just submitted r49 to the
> repository, which implements basic processing of Lua-based Kyua-files (aka,
> test suite definitions).  All the initial work of integration with Lua is
> done (that is, me learning how things may work and creating the basic
> boilerplate code), so further improvements should be more iterative.
> Coincidence? ;-)

"I do not see coincidence, I see providence."

Why not just keep things simple and stick to the rules defined by
fnmatch(3) [1]? Word of caution: according to kientzle@ there were
some issues with portability and functionality with fnmatch(3)
according to tar(1)'s requirements, so he basically ripped bits out of
libc, monkeyed with it a bit, and put it in libarchive.

...

> I really want to get rid of such boilerplate code (but I don't have ideas
> yet as to how to do it yet).  After having written dozens of pyunit-based
> tests while at Google, I can't stress how nice it is to be able to define
> tests in a concise way.  Loads of fine-grained tests == good (but the atf
> api kinda goes against this).

*puts on dusty Cisco QA/LTP hat*

Speaking from past experience, it makes sense to have a generalized
infrastructure that's portable anywhere and shim layers elsewhere to
make life easier in less portable cases (like with the BSDs for
instance). So I would make Kyua fast and general with as much useful
utility as possible without bloating it, and we (the BSD folks) can
pull our heads together and come up with less portable bits (but
portable within the BSDs) for testing out specific items. Granted, it
would probably require some consensus on format, way to do things,
etc, but I think that the 3 main BSDs (and their derivatives) would be
happy to have generalized tests that can port between the
distributions easily.

As time goes on bits from the BSD infrastructure could be ported into
Kyua as necessary if it's portable enough and makes sense to do so,
but probably as add-ons to the project.

>> there was another immediate result that irked me and is still unresolved
>> in this bit of code. The runtime of the test-driving shell script shot up
>> from 2 seconds to 41 seconds. AFAICT so far most of the time seems to be
>> spent inside the shell code of the atf_init_test_cases() bits, and in the
>> way each test case is run in a separate directory of its own.
>
> Wow, that's... surprising.  How did you measure that?  Is this running the
> test programs directly (./foo_test streq) or through atf-run?  I have
> certainly seen a slowdown, but a blow up of 20x seems too much.

A ktrace run / some form of tracing needs to be done on the function
to determine why stuff is so slow because that's a lot of incurred
overhead :o... a recent copy of FreeBSD CURRENT might help too (there
were some shell bugs that were resolved recently that have been
present in FreeBSD since at least 7.x).

>> So shell code for test cases is really really slow.
>>
>> But we do need to be able to run arbitrary shell commands for test cases,
>> e.g. to test utilities like sh(1) and test(1). With the proper insulation
>> from previous test cases, or environment variables, filesystem location,
>> etc.
>
> Yes indeed.  At some point I want to have bindings to write the test
> programs themselves in Lua (moving away from atf-sh as much as possible).

You're referring to the wrapper scripts which execute the tests though
and not the tests themselves, right :/? Requiring folks to learn Lua
to write testcases seems to put the bar up higher than it needs to be;
granted, it's a simple language from what I've seen, but I know a
kernel dev that hates to write shell code, and I doubt he would be
willing to learn Lua.

>> » File layout
>
> [...]
>>
>> It's a bit unclear what happens when both <home>/.kyua/kyua.conf and
>> <sysconfdir>/kyua/kyua.conf exist. Will we load the <home> file and stop at
>> that point? Will we load both? The text seems to imply that we load only the
>> <home> file, but we should make sure this is clearly stated in the
>> documentation, possibly by listing the paths we will look for config files
>> in the same order that they will be looked up (now it's listed in reverse
>> order, AFAICT).
>
> In atf, user-specific configuration files inherit the system-wide ones.  I
> will have to think further on what the best approach is though... because
> inheriting system-wide files by default makes it quite hard to ensure your
> custom file does not bring in global definitions that might change over
> time.
> Civen that we are dealing with scripts, we can make the default to *not*
> inherit the system-wide settings and, if the user so desires, he can just
> include the system settings.
> Will add these thoughts to the design doc.

I think that having user settings masking global settings would be
better to match standard program behavior, but maybe an `override'
option should be added for anal retentive testcases that would be
affected a great deal by change in behavior when using Kyua (say for
folks doing a nightly regression, etc)?

>> » That's all
>>
>> I don't have any other comments for now. I like the direction of using Lua
>> for parsing config files, options, etc. We will probably see lots of
>> interesting kyua modules & extensions if it's easy to hook into the existing
>> API and "do stuff" :-)
>
> Yep, that would indeed be nice :-)
> Thanks for your detailed comments!
> PS: At the moment, there is virtually no subscribers to this mailing list.
>  I expect we will eventually get people from multiple BSD backgrounds so we
> should avoid HTML emails.

<sarcasm>
As long as everything is kept down to 72 chars or less, Theo will be happy :).
</sarcasm>

I agree that HTML emails are ugly though..

Thanks!
-Garrett

1. http://www.opengroup.org/onlinepubs/000095399/functions/fnmatch.html

Julio Merino

unread,
Jan 23, 2011, 5:21:04 PM1/23/11
to kyua-d...@googlegroups.com
On Sun, Jan 23, 2011 at 12:52:22PM -0800, Garrett Cooper wrote:
> On Sun, Jan 23, 2011 at 11:24 AM, Julio Merino <jm...@google.com> wrote:
> > On Sun, Jan 23, 2011 at 12:18 PM, Giorgos Keramidas <gkera...@gmail.com>
> > wrote:
> >
> > My only concern is that inventing a new glob pattern might cause confusion.
> > ?I'd prefer to keep such patterns simple to prevent "abuses" like the **

> > recursion you mention.
> > Given that we are dealing with Lua scripts, we may as well expose primitives
> > to iterate over file sets (something as simple as readdir) and regexp
> > matching. ?Couple that with some fancy wrapping functions that take the

> > patterns you describe and translate them into the primitives and we can
> > cover pretty much any use case.
>
> Why not just keep things simple and stick to the rules defined by
> fnmatch(3) [1]?

Basically because I was not aware of it.

> Word of caution: according to kientzle@ there were
> some issues with portability and functionality with fnmatch(3)
> according to tar(1)'s requirements, so he basically ripped bits out of
> libc, monkeyed with it a bit, and put it in libarchive.

That's a serious drawback. (FWIW atf currently implements a glob->re
mapping scheme. It supports only ? and * but seems to have been enough
so far!)

My current idea for implementing, e.g. the top-level Kyuafile that has
to recurse into a subset of directories not known beforehand is
something along the lines of:

for file in fs.directory('.') do
if fs.is_directory(file) then
local subfile = fs.join(file, 'Kyuafile')
if fs.exists(subfile) then
include(subfile)
end
end
end

You could also do something like this to add all *_test files:

for file in fs.directory('.') do
if fnmatch('*_test', file) then
AtfTestProgram {name=file}
end
end

While the above examples are way more verbose than their current atf
counterparts, I feel that they are clearer in intent and that the
transparency that they offer is a win. (Plus, if proven necessary due
to popular demand, it is extremely easy to write wrapping functions for
the code above.)

> > I really want to get rid of such boilerplate code (but I don't have ideas

> > yet as to how to do it yet). ?After having written dozens of pyunit-based


> > tests while at Google, I can't stress how nice it is to be able to define

> > tests in a concise way. ?Loads of fine-grained tests == good (but the atf


> > api kinda goes against this).
>
> *puts on dusty Cisco QA/LTP hat*
>
> Speaking from past experience, it makes sense to have a generalized
> infrastructure that's portable anywhere and shim layers elsewhere to
> make life easier in less portable cases (like with the BSDs for
> instance). So I would make Kyua fast and general with as much useful
> utility as possible without bloating it, and we (the BSD folks) can
> pull our heads together and come up with less portable bits (but
> portable within the BSDs) for testing out specific items. Granted, it
> would probably require some consensus on format, way to do things,
> etc, but I think that the 3 main BSDs (and their derivatives) would be
> happy to have generalized tests that can port between the
> distributions easily.
>
> As time goes on bits from the BSD infrastructure could be ported into
> Kyua as necessary if it's portable enough and makes sense to do so,
> but probably as add-ons to the project.

I think that's a nice idea and, actually, it kinda is in atf too. I
have refused several times to provide NetBSD-specific mechanisms in the
upstream version of the code (links to PRs is the one that comes to
mind now). The problem is... atf is not really extensible by third
parties to add the necessary "shim layers" for customization that you
mention.

As a side note, take a look at:

http://cvsweb.netbsd.org/bsdweb.cgi/src/tests/h_macros.h?rev=1.6&content-type=text/x-cvsweb-markup&only_with_tag=MAIN

This is NetBSD's attempt of providing a subset of common functionality
useful within the realm of NetBSD tests. Some of such features could be
ported upstream to atf. (There is more stuff hidden in some other
directory of src/tests; in particular, for fs tests.)

> A ktrace run / some form of tracing needs to be done on the function
> to determine why stuff is so slow because that's a lot of incurred
> overhead :o... a recent copy of FreeBSD CURRENT might help too (there
> were some shell bugs that were resolved recently that have been
> present in FreeBSD since at least 7.x).

One thing I forgot to mention... what version of atf was that with?
Versions before 0.7 (IIRC) were incredibly slow -- but those are over a
year old already!

> > Yes indeed. ?At some point I want to have bindings to write the test


> > programs themselves in Lua (moving away from atf-sh as much as possible).
>
> You're referring to the wrapper scripts which execute the tests though
> and not the tests themselves, right :/? Requiring folks to learn Lua
> to write testcases seems to put the bar up higher than it needs to be;
> granted, it's a simple language from what I've seen, but I know a
> kernel dev that hates to write shell code, and I doubt he would be
> willing to learn Lua.

I was referring to both: the wrapper scripts and the tests themselves.
But I do not plan to kill atf-sh. I just want to offer an alternative
set of bindings so that a developer can write his tests in Lua if he so
wishes. The sh-based interface shall remain!

> I think that having user settings masking global settings would be
> better to match standard program behavior, but maybe an `override'
> option should be added for anal retentive testcases that would be
> affected a great deal by change in behavior when using Kyua (say for
> folks doing a nightly regression, etc)?

I think my prevous explanation was confusing (based on your response,
which confuses me ;-)

The approach I think makes more sense is:

* If you don't have ~/.foorc, kyua reads /etc/foo.conf and that's it.

* If you have ~/.foorc, then kyua reads ~/.foorc only and does _not_
read /etc/foo.conf. (But, if you wish, you can just add
'include /etc/foo.conf' or similar in your ~/.foorc if you desire to
import the global settings.) Pretty much like how .shrc behaves.

Does that make more sense?

Thanks for your comments!

Garrett Cooper

unread,
Jan 23, 2011, 5:59:16 PM1/23/11
to kyua-d...@googlegroups.com

Yeah, they probably aren't really usable in an infrastructure (it's
like saying you have a shiny Ferrari, but you have to drive 20MPH
because those are the limitations you have to deal with).

Honestly, as long as the semantics match fnmatch(3) within reason, it
should be ok to use whatever implementation necessary to achieve that
goal. fnmatch(3) just meshes well with other systems (BSD, Linux,
Windows, etc), so that's the reason why I suggested it first and
foremost.

Well... that's reasonably small in comparison to the kitchen sink that
libltp is in LTP today. If I was ambitious enough I would remove
interfaces and refactor code, but I don't feel like rototilling that
project more than I already have in the past couple of months.

Along those lines, that's why I suggest keeping Kyua minimal and
maximize things which are required at bare minimum to get the job
done. Otherwise this project will become another kitchen sink type
project.

>> A ktrace run / some form of tracing needs to be done on the function
>> to determine why stuff is so slow because that's a lot of incurred
>> overhead :o... a recent copy of FreeBSD CURRENT might help too (there
>> were some shell bugs that were resolved recently that have been
>> present in FreeBSD since at least 7.x).
>
> One thing I forgot to mention... what version of atf was that with?
> Versions before 0.7 (IIRC) were incredibly slow -- but those are over a
> year old already!
>
>> > Yes indeed. ?At some point I want to have bindings to write the test
>> > programs themselves in Lua (moving away from atf-sh as much as possible).
>>
>> You're referring to the wrapper scripts which execute the tests though
>> and not the tests themselves, right :/? Requiring folks to learn Lua
>> to write testcases seems to put the bar up higher than it needs to be;
>> granted, it's a simple language from what I've seen, but I know a
>> kernel dev that hates to write shell code, and I doubt he would be
>> willing to learn Lua.
>
> I was referring to both: the wrapper scripts and the tests themselves.
> But I do not plan to kill atf-sh.  I just want to offer an alternative
> set of bindings so that a developer can write his tests in Lua if he so
> wishes.  The sh-based interface shall remain!

Good to hear. As long as everything's binary compatible / communicable
then we should be able to use whatever language suitable to drive
tests with via C-type hooks/extensions.

>> I think that having user settings masking global settings would be
>> better to match standard program behavior, but maybe an `override'
>> option should be added for anal retentive testcases that would be
>> affected a great deal by change in behavior when using Kyua (say for
>> folks doing a nightly regression, etc)?
>
> I think my prevous explanation was confusing (based on your response,
> which confuses me ;-)
>
> The approach I think makes more sense is:
>
> * If you don't have ~/.foorc, kyua reads /etc/foo.conf and that's it.
>
> * If you have ~/.foorc, then kyua reads ~/.foorc only and does _not_
>  read /etc/foo.conf.  (But, if you wish, you can just add
>  'include /etc/foo.conf' or similar in your ~/.foorc if you desire to
>  import the global settings.)  Pretty much like how .shrc behaves.
>
> Does that make more sense?

What I meant is:

. /etc/foo.conf
. ~/.foorc

Values in foo.conf are superseded by values in .foorc . Unless you say
--norc, etc in which case it would function like bash --norc:

--norc Do not read and execute the personal initialization file
~/.bashrc if the shell is interactive. This option is on by
default if the shell is invoked as sh.

Thanks!
-Garrett

Giorgos Keramidas

unread,
Jan 24, 2011, 10:17:48 AM1/24/11
to kyua-d...@googlegroups.com
On Sun, 23 Jan 2011 22:21:04 +0000, Julio Merino <jm...@google.com> wrote:
> On Sun, Jan 23, 2011 at 12:52:22PM -0800, Garrett Cooper wrote:
> My current idea for implementing, e.g. the top-level Kyuafile that has
> to recurse into a subset of directories not known beforehand is
> something along the lines of:
>
> for file in fs.directory('.') do
> if fs.is_directory(file) then
> local subfile = fs.join(file, 'Kyuafile')
> if fs.exists(subfile) then
> include(subfile)
> end
> end
> end
>
> You could also do something like this to add all *_test files:
>
> for file in fs.directory('.') do
> if fnmatch('*_test', file) then
> AtfTestProgram {name=file}
> end
> end
>
> While the above examples are way more verbose than their current atf
> counterparts, I feel that they are clearer in intent and that the
> transparency that they offer is a win. (Plus, if proven necessary due
> to popular demand, it is extremely easy to write wrapping functions
> for the code above.)

If demand for iterating through a directory and pattern-matching all
files before creating AtfTestProgram{} for them is sufficiently high, it
should be relatively easy to wrap these in a function or provide a
Lua-based path walker e.g.:

function testmatch(file, pattern)
if kyua.fs.fnmatch(pattern, file)
AtfTestProgram{name = file}
end
end
kyua.path.walk('.', {func = testmatch, arg = '*_test'})

> I think my prevous explanation was confusing (based on your response,
> which confuses me ;-)
>
> The approach I think makes more sense is:
>
> * If you don't have ~/.foorc, kyua reads /etc/foo.conf and that's it.
>
> * If you have ~/.foorc, then kyua reads ~/.foorc only and does _not_
> read /etc/foo.conf. (But, if you wish, you can just add
> 'include /etc/foo.conf' or similar in your ~/.foorc if you desire to
> import the global settings.) Pretty much like how .shrc behaves.
>
> Does that make more sense?
>
> Thanks for your comments!

Yes, that makes indeed a lot more sense, and it *permits* including the
system-wide configuration when necessary.

I'd like this to be the default for config file loading, so that users
can opt out of brain-damaged system defaults, which some distributors
may think it's a wise idea to force down the throad of everybody (the
'clear' command that wipes terminal scrollback from the default copy of
/etc/bash.bash.logout in Linux distribution packages is the canonical
example of this for me).

Reply all
Reply to author
Forward
0 new messages