How to set ATF_SHELL per test program?

[Alan Somers]

If I set $ATF_SHELL in the environment before I invoke "kyua test",
then it's value will be used as the interpreter to execute all test
programs. However, I would like to use /bin/sh to execute most test
programs, with an alternate interpreter for just a few. How can I set
that variable for an individual test program or test case? If I put
it in the Kyuafile as I would do the "timeout" metadata property, then
I get this error:

kyua: E: Load of 'Kyuafile' failed: Failed to load Lua file
'Kyuafile': Kyuafile:5: Unknown metadata property atf_shell.

atf_test_program{name='fibs', atf_shell='/usr/local/bin/ksh93'}

So it seems that atf_shell is not a metadata property, and
atf_test_program() cannot specify config variables.

If I put it in the Atffile as suggested by atf-formats, then run
"atf2kyua", it's not present in the generated Kyuafile.

conf: atf_shell = "/usr/local/bin/ksh93"

It seems as though I can't specify it in the head of my test case,
because test programs have read-only access to configuration
variables.

If I put it in my global kyua.conf file, which I don't really want to
do, it doesn't seem to have an impact.

test_suites.FreeBSD.atf_shell="/usr/local/bin/ksh93"


So what is the correct way to specify ATF_SHELL for an individual test
program or test case ?

-Alan

[Julio Merino]

On Tue, May 27, 2014 at 12:16 PM, Alan Somers <aso...@freebsd.org> wrote:
> So what is the correct way to specify ATF_SHELL for an individual test
> program or test case ?

Well... you can't. atf-sh is "built" and _tested_ against a specific
shell. Dynamically changing the shell at run-time to something
different can (and probably will) cause all sorts of unexpected issues
in libatf-sh.subr. (Allowing the end user to customize this, as you
attempted via kyua.conf, is a non-option.)

What are you trying to do?

[Alan Somers]

Why is it ok to change the shell via an environment variable, but not
via any configuration files? Is it because the config files are
parsed after atf-sh has already been invoked?

>
> What are you trying to do?

I have a large number of tests that predate ATF. They were originally
written to run under ksh93 and no other shell. I've converted them to
run as ATF test programs, but I must specify ATF_SHELL in the
environment or they will fail in various ways. But ksh93 isn't 100%
backwards compatible with /bin/sh, so some of FreeBSD's builtin tests
fail when run with it. So the situation I'm in is that I can't run
all of my tests with a single invocation of Kyua. That in turn means
that I can't view all of my results in a single HTML output file.
What solution would you recommend?

-Alan

>
> --
> You received this message because you are subscribed to the Google Groups "kyua-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kyua-discuss...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Craig Rodrigues]

On Tue, May 27, 2014 at 9:44 AM, Alan Somers <aso...@freebsd.org> wrote:
>
> I have a large number of tests that predate ATF. They were originally
> written to run under ksh93 and no other shell. I've converted them to
> run as ATF test programs, but I must specify ATF_SHELL in the
> environment or they will fail in various ways. But ksh93 isn't 100%
> backwards compatible with /bin/sh, so some of FreeBSD's builtin tests
> fail when run with it. So the situation I'm in is that I can't run
> all of my tests with a single invocation of Kyua.

Are you referring to the ZFS tests (
http://blog.delphix.com/jkennedy/2013/09/17/testing-openzfs/ ,
https://github.com/delphix/zfs-test )?
At the OpenZFS Summit at Delphix in San Francisco last year, I looked
at those tests,
and the ksh93 dependencies were quite annoying. The only way I could get those
tests to run under FreeBSD was to take your port (
http://people.freebsd.org/~asomers/stf_zfs/ ) of those
tests to FreeBSD.

How hard would it be to upstream enough stuff to Illumos in order
to get those tests to run natively under Kyua under FreeBSD,
without maintaining a separate FreeBSD port of those tests?

--
Craig

[Julio Merino]

On Tue, May 27, 2014 at 12:44 PM, Alan Somers <aso...@freebsd.org> wrote:

On Tue, May 27, 2014 at 10:35 AM, Julio Merino <jm...@meroh.net> wrote:
> On Tue, May 27, 2014 at 12:16 PM, Alan Somers <aso...@freebsd.org> wrote:
>> So what is the correct way to specify ATF_SHELL for an individual test
>> program or test case ?
>
> Well... you can't.  atf-sh is "built" and _tested_ against a specific
> shell. Dynamically changing the shell at run-time to something
> different can (and probably will) cause all sorts of unexpected issues
> in libatf-sh.subr. (Allowing the end user to customize this, as you
> attempted via kyua.conf, is a non-option.)

Why is it ok to change the shell via an environment variable, but not
via any configuration files?  Is it because the config files are
parsed after atf-sh has already been invoked?

No, it's because you should always be allowed to override constants in binaries at run-time, for debugging/troubleshooting purposes. This is not intended to be a per-program knob and is why it cannot be specified in the configuration.

(That's the story today.  Historically, though, we used to have this thing called atf-build that took a source file and generated a script from it with the additional ATF boilerplate at the top; when using atf-build, you could actually provide a per-test-program shell.  With the switch to atf-sh, that feature was lost.)

 

> What are you trying to do?

I have a large number of tests that predate ATF.  They were originally
written to run under ksh93 and no other shell.  I've converted them to
run as ATF test programs, but I must specify ATF_SHELL in the
environment or they will fail in various ways.  But ksh93 isn't 100%
backwards compatible with /bin/sh, so some of FreeBSD's builtin tests
fail when run with it.

Right, and because of the incompatibilities, there is no guarantee that libatf-sh.subr will run correctly nor continue to run with ksh...

 

 So the situation I'm in is that I can't run
all of my tests with a single invocation of Kyua.  That in turn means
that I can't view all of my results in a single HTML output file.
What solution would you recommend?

The simplest would be to just shell out to ksh to run the actual code, just as you would to invoke a helper test program or any other binary written in a different language.

You'd, maybe, also do something like:

#! /usr/bin/env ATF_SHELL=.../ksh atf-sh

but be wary of the consequences.  Also, I'm not sure if any limits in the shebang line length will prevent this, but if not it should work.

Supporting the setting you request basically means having to have an alternate atf-ksh that is guaranteed to work with ksh because, as far as I know, they are different languages. I'm not sure we want to maintain that...

(And lastly, the "best" option would maybe be to rewrite those tests to use POSIX shell features instead of ksh, but I suppose that's out of the question.)

Do any of these make sense?

[Alan Somers]

On Tue, May 27, 2014 at 11:31 AM, Craig Rodrigues <rod...@freebsd.org> wrote:
> On Tue, May 27, 2014 at 9:44 AM, Alan Somers <aso...@freebsd.org> wrote:
>>
>> I have a large number of tests that predate ATF. They were originally
>> written to run under ksh93 and no other shell. I've converted them to
>> run as ATF test programs, but I must specify ATF_SHELL in the
>> environment or they will fail in various ways. But ksh93 isn't 100%
>> backwards compatible with /bin/sh, so some of FreeBSD's builtin tests
>> fail when run with it. So the situation I'm in is that I can't run
>> all of my tests with a single invocation of Kyua.
>
> Are you referring to the ZFS tests (
> http://blog.delphix.com/jkennedy/2013/09/17/testing-openzfs/ ,
> https://github.com/delphix/zfs-test )?
> At the OpenZFS Summit at Delphix in San Francisco last year, I looked
> at those tests,
> and the ksh93 dependencies were quite annoying. The only way I could get those
> tests to run under FreeBSD was to take your port (
> http://people.freebsd.org/~asomers/stf_zfs/ ) of those
> tests to FreeBSD.

I am indeed.

>
> How hard would it be to upstream enough stuff to Illumos in order
> to get those tests to run natively under Kyua under FreeBSD,
> without maintaining a separate FreeBSD port of those tests?

Hard. We have 383 local changes to the STF. Many have broken
compatibility with Illumos. Many others break conventions that
Illumos developers value. For example, my port no longer supports
running the tests using the stf tools; only ATF/Kyua. I think that
Illumos would reject such changes.

IMHO, the best way forward it to rewrite the STF build system to
follow FreeBSD conventions, then smooth out the remaining ATF
integration issues (like this one). ksh93 will have to stick around
for awhile. Porting 36000 SLOC of ksh93 code to sh is low on my list
of priorities ;)

-Alan

>
> --
> Craig

[Alan Somers]

This is an unattractive option for reasons that are too long and
boring to be contained within the margins of this forum.

>
> You'd, maybe, also do something like:
>
> #! /usr/bin/env ATF_SHELL=.../ksh atf-sh

That doesn't work. But using "#! /usr/bin/env atf-ksh93", where
atf-ksh93 is a /bin/sh script in my PATH that does
'ATF_SHELL=/path/to/ksh93 atf-sh "$@"', does work.

>
> but be wary of the consequences. Also, I'm not sure if any limits in the
> shebang line length will prevent this, but if not it should work.
>
> Supporting the setting you request basically means having to have an
> alternate atf-ksh that is guaranteed to work with ksh because, as far as I
> know, they are different languages. I'm not sure we want to maintain that...

I can understand why you wouldn't want to. It looks like I'll have to
maintain it for internal use, for now.

>
> (And lastly, the "best" option would maybe be to rewrite those tests to use
> POSIX shell features instead of ksh, but I suppose that's out of the
> question.)

You and Craig have renewed my interested in ditching ksh93. I'll give
it a shot, after I fix the build system.

-Alan

>
> Do any of these make sense?
>

[Craig Rodrigues]

On Tue, May 27, 2014 at 12:00 PM, Alan Somers <aso...@freebsd.org> wrote:
> On Tue, May 27, 2014 at 11:31 AM, Craig Rodrigues <rod...@freebsd.org> wrote:
>> On Tue, May 27, 2014 at 9:44 AM, Alan Somers <aso...@freebsd.org> wrote:
>>>
>>> I have a large number of tests that predate ATF. They were originally
>>> written to run under ksh93 and no other shell. I've converted them to
>>> run as ATF test programs, but I must specify ATF_SHELL in the
>>> environment or they will fail in various ways. But ksh93 isn't 100%
>>> backwards compatible with /bin/sh, so some of FreeBSD's builtin tests
>>> fail when run with it. So the situation I'm in is that I can't run
>>> all of my tests with a single invocation of Kyua.
>>
>> Are you referring to the ZFS tests (
>> http://blog.delphix.com/jkennedy/2013/09/17/testing-openzfs/ ,
>> https://github.com/delphix/zfs-test )?
>> At the OpenZFS Summit at Delphix in San Francisco last year, I looked
>> at those tests,
>> and the ksh93 dependencies were quite annoying. The only way I could get those
>> tests to run under FreeBSD was to take your port (
>> http://people.freebsd.org/~asomers/stf_zfs/ ) of those
>> tests to FreeBSD.
>
> I am indeed.

Good stuff. That's a lot of hard work you did to port the tests.
I'm glad you did it, because I would have tried and given up. :)

>> How hard would it be to upstream enough stuff to Illumos in order
>> to get those tests to run natively under Kyua under FreeBSD,
>> without maintaining a separate FreeBSD port of those tests?
>
> Hard. We have 383 local changes to the STF. Many have broken
> compatibility with Illumos. Many others break conventions that
> Illumos developers value. For example, my port no longer supports
> running the tests using the stf tools; only ATF/Kyua. I think that
> Illumos would reject such changes.

Oh that's too bad. Yes, those sound like changes that are not upstreamable.

> IMHO, the best way forward it to rewrite the STF build system to
> follow FreeBSD conventions, then smooth out the remaining ATF
> integration issues (like this one). ksh93 will have to stick around
> for awhile. Porting 36000 SLOC of ksh93 code to sh is low on my list
> of priorities ;)

To me, porting ksh93 to sh is as exciting as porting COBOL to FORTRAN-77. :)

If you are talking about rewrites, what do you think about the feasibility
of rewriting the STF build system to use WAF ( http://code.google.com/p/waf/ ).
To end-users, WAF looks like autoconf, but it is written entirely in Python.
I recently looked at the use of WAF in Samba 4.0 uses it, and it is
quite reasonable...
certainly more fun than messing with m4 macros/autoconf/automake/etc.

Using WAF would be a way to sneak in more Python
into the STF. Also, few people care if changes are made which make things
run better on FreeBSD, but WAF works great under Linux, so being able
to make the STF more Linux-friendly.


If we could get a nice combination of STF + Kyua + Jenkins going,
that would be a really nice thing to showcase. It would be even nicer
if that combination of tools worked well under Illumos, FreeBSD, and Linux.

--
Craig

[Alan Somers]

LOL

>
> If you are talking about rewrites, what do you think about the feasibility
> of rewriting the STF build system to use WAF ( http://code.google.com/p/waf/ ).
> To end-users, WAF looks like autoconf, but it is written entirely in Python.
> I recently looked at the use of WAF in Samba 4.0 uses it, and it is
> quite reasonable...
> certainly more fun than messing with m4 macros/autoconf/automake/etc.
>
> Using WAF would be a way to sneak in more Python
> into the STF. Also, few people care if changes are made which make things
> run better on FreeBSD, but WAF works great under Linux, so being able
> to make the STF more Linux-friendly.

I've used WAF for C and C++. It's amazingly fast! But it wouldn't
solve the hard parts of porting the STF. For example, some tests use
gpart on FreeBSD but would need to use parted on Linux. The
infrastructure of the tests is insufficiently abstract to cleanly
support that distinction, even if somebody had the time to do all the
porting. Frankly, I don't think that any sh-based language can handle
tests of this size cleanly. We would probably have to port the tests
to Ruby or Python or something if we wanted them to be completely
cross-platform.

>
>
> If we could get a nice combination of STF + Kyua + Jenkins going,
> that would be a really nice thing to showcase. It would be even nicer
> if that combination of tools worked well under Illumos, FreeBSD, and Linux.

I only care about FreeBSD, and nobody's going to pay me for Illumos or
Linux suppport. My goal will be to get the STF tests to build and run
using FreeBSD-friendly makefiles and tools, and to integrate it with
Kyua and Jenkins.

[Craig Rodrigues]

On Tue, May 27, 2014 at 1:52 PM, Alan Somers <aso...@freebsd.org> wrote:
>
> I've used WAF for C and C++. It's amazingly fast! But it wouldn't
> solve the hard parts of porting the STF. For example, some tests use
> gpart on FreeBSD but would need to use parted on Linux. The
> infrastructure of the tests is insufficiently abstract to cleanly
> support that distinction, even if somebody had the time to do all the
> porting. Frankly, I don't think that any sh-based language can handle
> tests of this size cleanly. We would probably have to port the tests
> to Ruby or Python or something if we wanted them to be completely
> cross-platform.

Well, at the end of the day, we can't get away from the fact that
at some layer, there will be some machine dependent code that
needs to be ported to each platform.

>> If we could get a nice combination of STF + Kyua + Jenkins going,
>> that would be a really nice thing to showcase. It would be even nicer
>> if that combination of tools worked well under Illumos, FreeBSD, and Linux.
>
> I only care about FreeBSD, and nobody's going to pay me for Illumos or
> Linux suppport. My goal will be to get the STF tests to build and run
> using FreeBSD-friendly makefiles and tools, and to integrate it with
> Kyua and Jenkins.

The only reason I mention Linux is because at the OpenZFS summit
at Delphix in San Francisco, I saw a lot of ZFS/Linux developers.
At least for now, ZFS/Linux seems to have quite a few developers,
many of them working at Lawrence Livermore National Laboratory.
There seem to be more ZFS/Linux developers at LLNL than
ZFS/FreeBSD developers where I work (iXsystems). :)

If we could upstream changes to STF to work better on FreeBSD *and*
Linux, with more
modern things like WAF and Python, then
that might be the bait to pull more of the LLNL developers to work on
STF and make it better all around. Of course, their focus would be
Linux, but maybe we could ride
their coattails and benefit in FreeBSD.

Even better if we can encourage use of kyua on Linux, to pull in more
developers from that ecosystem.
--
Craig

[Craig Rodrigues]

Now I am understanding the problems that Alan faced with this.

I have a whole lot of tests, some are written as /bin/sh compatible scripts,
and some are /bin/bash compatible scripts.

Is there a way to override the ATF_SHELL on a per test basis?

The atf-sh(1) man page mentions atf-config(1) , but that man page doesn't

seem to exist in atf-0.20.

Rewriting all the test cases to /bin/sh, or creating wrapper scripts to call the /bin/bash scripts
from a /bin/sh scripts is not practical.

--

Craig

[Julio Merino]

On Tue, Sep 23, 2014 at 8:58 PM, Craig Rodrigues <rod...@freebsd.org> wrote:
>
> Now I am understanding the problems that Alan faced with this.
> I have a whole lot of tests, some are written as /bin/sh compatible scripts,
> and some are /bin/bash compatible scripts.
>
> Is there a way to override the ATF_SHELL on a per test basis?

There isn't.

I'm wondering if adding a flag to atf-sh to specify the shell would be
sufficient and a good idea for this use case? Then you'd do "#!
/somewhere/atf-sh -s /bin/bash".

Doesn't solve the "compatibility issues" mentioned above, but those
are very unlikely anyway.

> The atf-sh(1) man page mentions atf-config(1) , but that man page doesn't
> seem to exist in atf-0.20.

Filed https://github.com/jmmv/atf/issues/18 to track the atf-config(1)
manpage cleanup.

[Craig Rodrigues]

On Sep 25, 2014 12:30 AM, "Julio Merino" <jm...@meroh.net> wrote:
>
> On Tue, Sep 23, 2014 at 8:58 PM, Craig Rodrigues <rod...@freebsd.org> wrote:
> >
> > Now I am understanding the problems that Alan faced with this.
> > I have a whole lot of tests, some are written as /bin/sh compatible scripts,
> > and some are /bin/bash compatible scripts.
> >
> > Is there a way to override the ATF_SHELL on a per test basis?
>
> There isn't.
>
> I'm wondering if adding a flag to atf-sh to specify the shell would be
> sufficient and a good idea for this use case? Then you'd do "#!
> /somewhere/atf-sh -s /bin/bash".
>

Adding a flag to atf-sh would work for me.  On Linux and Mac, compiling atf results in atf-sh using bash internally, and atf still works.  So the internal shell routines in atf-sh are clean and portable to bash...dunno about ksh, but I think it should work there too.

The restriction would have to be that folks could only use "POSIX-compatible" shells.  No csh,  emacs, python, etc.
--
Craig

[Julio Merino]

On Thu, Sep 25, 2014 at 7:59 PM, Craig Rodrigues <rod...@freebsd.org> wrote:
>
> Adding a flag to atf-sh would work for me.

Alright then. This is done in head. Please let me know if things work
for you. Also, Alan, it'd be great if you'd check that this removes
the need for your ksh93 hack.

Will try to cut a new release next week, when I'm back from EuroBSDCon.

[Craig Rodrigues]

On Fri, Sep 26, 2014 at 9:49 AM, Julio Merino <jm...@meroh.net> wrote:

Alright then. This is done in head. Please let me know if things work
for you. Also, Alan, it'd be great if you'd check that this removes
the need for your ksh93 hack.

Hi,

I tried with bash, and it worked.  Thanks!

Here is a quick test I wrote for the CVE-2014-6721 bash problem.

--

Craig