[Haskell-cafe] ANNOUNCE: test-framework-golden-1.1
Roman Cheplyaka
golden testing library.
Hackage: http://hackage.haskell.org/package/test-framework-golden
GitHub: https://github.com/feuerbach/test-framework-golden
Golden tests are similar to unit tests (as implemented in HUnit), but the idea
is to store the expected result in files (called «golden» files).
I was introduced to the idea of golden testing by Bohdan Vlasyuk at ZuriHac in
2010. Since then I've discovered that this is a nice approach and it is already
used in variety of projects (e.g. ghc, haddock).
Surprisingly, not much is written about golden testing, and I've been unable to
find any golden testing libraries for Haskell or any other programming language.
Every project has its own ad-hoc implementation in Haskell/Python/Shell/etc.
The closest match on the market is Simon Michael's shelltestrunner. But to use
it you have to expose the tested functionality via command line, which may be
inconvenient.
So this is my attempt at a general golden testing library.
It consists of two modules.
Test.Golden has a simple interface that helps you get started very quickly.
Test.Golden.Advanced provides a very generic testing function that you can use
to implement the testing system you dream about.
The library is integrated with test-framework, so you can use golden tests in
addition to SmallCheck/QuickCheck/HUnit tests.
In future there's a plan to support some golden test management capabilities.
Roman
_______________________________________________
Haskell-Cafe mailing list
Haskel...@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
Alfredo Di Napoli
A.
Sent from my iPad
Gianfranco Alongi
how to TDD in them.
As such, you mention that this is similar, could you please send some
links about this?
Cheers!
Simon Hengel
> I am glad to announce the first public release of
> test-framework-golden — a golden testing library.
Nice!
> The library is integrated with test-framework, so you can use golden
> tests in addition to SmallCheck/QuickCheck/HUnit tests.
I would suggest to rename the modules to
Test.Framework.Golden
and
Test.Framework.Golden.Advanced
to more clearly represent that fact.
Cheers,
Simon
Roman Cheplyaka
> Hi,
>
> > I am glad to announce the first public release of
> > test-framework-golden — a golden testing library.
>
> Nice!
>
> > The library is integrated with test-framework, so you can use golden
> > tests in addition to SmallCheck/QuickCheck/HUnit tests.
>
> I would suggest to rename the modules to
>
> Test.Framework.Golden
>
> and
>
> Test.Framework.Golden.Advanced
>
> to more clearly represent that fact.
According to the test-framework convention, they ought to be named
Test.Framework.Providers.Golden
and
Test.Framework.Providers.Golden.Advanced
respectively, and you can see that it was the case in the early
versions:
http://hackage.haskell.org/package/test-framework-golden-1.0
But it's just too much to type.
My justification (which you may or may not buy) is that, unlike, say,
Test.Framework.Providers.HUnit, this is not an adaptation of an existing
testing library to test-framework, but is a new library that just
happens to use test-framework. So it's more like Test.HUnit, although it
already subsumes what would become Test.Framework.Providers.Golden.
Roman
Roman Cheplyaka
> I'm a big fan of TDD and tend to approach all languages by learning
> how to TDD in them.
>
> As such, you mention that this is similar, could you please send some
> links about this?
Sorry, I'm not sure I understand the question. Links about what?
The library documentation is available on Hackage (the first link
below).
If you're asking about TDD in Haskell in general, I doubt I can tell
you anything that Google can't.
Roman
Roman Cheplyaka
> Hi Roman,
>
> This sounds like a great idea. As you suggest, I've put this kind of
> thing together in scripts many times, without a consistent framework.
>
> I haven't tried it yet, but have one question. When you call a
> function like
>
> goldenVsString
>
> I assume one generally provides a relative path. what is the file
> path relative to? How does one arrange a package of haskell code and
> associated golden files, and reference the golden files from within
> the haskell so that the tree works from any location?
The paths are naturally relative to the working directory of the test
executable.
In my projects, I typically have
./test.hs
./tests/case1.golden
./tests/case2.golden
./tests/case3.golden
And so test.hs refers to test cases by relative paths like
"tests/case1.golden". Of course, that'll only work if you run test.hs
from its directory, but it is the case most of the time anyway.
Also, since paths are relative to the directory containing the project.cabal
file, "cabal test" works, too.
If, for some reason, you want to be able to run tests from any location,
you could provide the path to golden paths e.g. via an environment
variable.
Roman
Simon Hengel
> Test.Framework.Providers.HUnit, this is not an adaptation of an existing
> testing library to test-framework, but is a new library that just
> happens to use test-framework. So it's more like Test.HUnit, although it
> already subsumes what would become Test.Framework.Providers.Golden.
testing frameworks. So I think it is less general. Anyway, it's up to
you.
Cheers,
Simon
Janek S.
announcement came up, so your library made it as a last minute news :) I never used golden
approach to testing but it is good to know that it exists.
*I* think that it might be a good idea to separete your library into two parts: the golden
functionality itself and test-framework provider. Right now it seems impossible to use your
library outside of test-framework. For me that's not a big deal, cause I use test-framework
anyway, but I suspect you could get more users that way.
> According to the test-framework convention, they ought to be named
>
> Test.Framework.Providers.Golden
>
> and
>
> Test.Framework.Providers.Golden.Advanced
>
> respectively, and you can see that it was the case in the early
> versions:
> http://hackage.haskell.org/package/test-framework-golden-1.0
>
> But it's just too much to type.
Well, I'm already typing Test.Framework.Providers.QuickCheck2, so that's not a big deal really :)
Jan
Roman Cheplyaka
> announcement came up, so your library made it as a last minute news :) I never used golden
> approach to testing but it is good to know that it exists.
I hope you won't forget to cover SmallCheck in your article as well.
Being also the maintainer of SmallCheck, I want it to steal some fame
from QuickCheck :)
> *I* think that it might be a good idea to separete your library into two parts: the golden
> functionality itself and test-framework provider. Right now it seems impossible to use your
> library outside of test-framework. For me that's not a big deal, cause I use test-framework
> anyway, but I suspect you could get more users that way.
1. It's hard to guess at the moment how a good interface to the "pure
golden" part should look like.
2. I'd rather avoid doing work when there's no demand.
So when I see someone interested in using it outside of test-framework,
I'm sure we'll figure something out.
Roman
Roman Cheplyaka
> http://ics.p.lodz.pl/~stolarek/blog/2012/10/code-testing-in-haskell/
> I was just going to publish it and then your email came up on the list.
> > I hope you won't forget to cover SmallCheck in your article as well.
> > Being also the maintainer of SmallCheck, I want it to steal some fame
> > from QuickCheck :)
> didn't feel like I need to look for other testing library. I might be wrong of course.
There are some technical advantages to SmallCheck (determinism, no need
to shrink etc.), but the main reason I prefer it is because it gives me more
confidence. With quickcheck, I know that it generated 100 tests, but
I've no idea what those tests are, and whether the RNG missed some
important corner cases. With SmallCheck I know that it tried *all* cases
up to certain depth, so it's much more reassuring. (Except for cases
when the notion of depth isn't that informative, like floating-point
numbers.)
Roman
Janek S.
http://ics.p.lodz.pl/~stolarek/blog/2012/10/code-testing-in-haskell/
I was just going to publish it and then your email came up on the list.
> Being also the maintainer of SmallCheck, I want it to steal some fame
> from QuickCheck :)
didn't feel like I need to look for other testing library. I might be wrong of course.
>
> 1. It's hard to guess at the moment how a good interface to the "pure
> golden" part should look like.
> 2. I'd rather avoid doing work when there's no demand.
Janek S.
> to shrink etc.), but the main reason I prefer it is because it gives me
> more confidence. With quickcheck, I know that it generated 100 tests, but
> I've no idea what those tests are, and whether the RNG missed some
> important corner cases. With SmallCheck I know that it tried *all* cases up
> to certain depth, so it's much more reassuring. (Except for cases when the
> notion of depth isn't that informative, like floating-point numbers.)
repeatibility sounds very good - that's something I actually miss in QC.
Jan
Simon Hengel
> golden" part should look like.
goldenVsFile :: FilePath -> FilePath -> IO () -> Assertion
That way it works with plain HUnit
main = runTestTT $ TestLabel "someAction produce desired output $
goldenVsFile "ref.txt" "out.txt" someAction
test-framework
main = defaultMain [
testGroup "someAction" [
testCase "produces some desired output" $
goldenVsFile "ref.txt" "out.txt" someAction
]
]
Hspec
main = hspec $ do
describe "someAction" $ do
it "produces some desired output" $ do
goldenVsFile "ref.txt" "out.txt" someAction
and probably every other current or future Haskell test framework.
Cheers,
Simon
Roman Cheplyaka
at the moment on top of HUnit.
However, an important part of functionality isn't there at the moment —
golden file management. You should be able to say, "for this test,
take its current output and write it to the corresponding golden file".
In order to do that, you need to have access to the list of golden tests
in the suite. This is where implementation details of different test
frameworks start to matter. Probably one can make an abstraction over
test frameworks that would give the list of all golden tests.
(Although when you start abstracting over test frameworks, which are
abstractions themselves, it becomes somewhat funny.)
Speaking of such functionality, correct me if I'm wrong, but neither
HUnit nor hspec won't be able to support it anyway, because they
represent tests as opaque IO actions.
Nor can HTF provide such support — although its TestSort type could be
extended to indicate that the test is a golden test, still there's no
way to get hold of the golden file name.
test-framework will support this once the Typeable constraint is added
for tests.
Roman
* Simon Hengel <s...@typeful.net> [2012-10-05 13:43:57+0200]
Simon Hengel
> I can do that indeed, and I guess I could reimplement everything I have
> at the moment on top of HUnit.
>
> However, an important part of functionality isn't there at the moment —
> golden file management. You should be able to say, "for this test,
> take its current output and write it to the corresponding golden file".
>
> In order to do that, you need to have access to the list of golden tests
> in the suite. This is where implementation details of different test
> frameworks start to matter. Probably one can make an abstraction over
> test frameworks that would give the list of all golden tests.
> (Although when you start abstracting over test frameworks, which are
> abstractions themselves, it becomes somewhat funny.)
Ok, makes sense.
I'm looking forward to give it a try, and see how it compares to using
operating system primitives (say `cp') for "golden file management".
> Speaking of such functionality, correct me if I'm wrong, but neither
> HUnit nor hspec won't be able to support it anyway, because they
> represent tests as opaque IO actions.
It would be easy to extend Hspec to support this in the same way you
extend test-framework to support this. It requires existentials; the
only substantial difference that I can see is that test-framework
already uses existentials, while Hspec does not.
Roman Cheplyaka
> On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:
> > I can do that indeed, and I guess I could reimplement everything I have
> > at the moment on top of HUnit.
> >
> > However, an important part of functionality isn't there at the moment —
> > golden file management. You should be able to say, "for this test,
> > take its current output and write it to the corresponding golden file".
> >
> > In order to do that, you need to have access to the list of golden tests
> > in the suite. This is where implementation details of different test
> > frameworks start to matter. Probably one can make an abstraction over
> > test frameworks that would give the list of all golden tests.
> > (Although when you start abstracting over test frameworks, which are
> > abstractions themselves, it becomes somewhat funny.)
>
> Ok, makes sense.
>
> I'm looking forward to give it a try, and see how it compares to using
> operating system primitives (say `cp') for "golden file management".
1. You often want to update not just one test, but all, or some of the
tests (when you've made a change and verified that the changes in
output are expected). Doing it in command line is certainly possible,
but not trivial nor convenient.
2. For some tests (like goldenVsString) the output is not captured in a
file, so using cp directly is not possible.
> > Speaking of such functionality, correct me if I'm wrong, but neither
> > HUnit nor hspec won't be able to support it anyway, because they
> > represent tests as opaque IO actions.
>
> It would be easy to extend Hspec to support this in the same way you
> extend test-framework to support this. It requires existentials; the
> only substantial difference that I can see is that test-framework
> already uses existentials, while Hspec does not.
Well, if you are willing to make this change, then I'll try to do my
part of the job and expose a useful abstraction.
Roman
Simon Hengel
> * Simon Hengel <s...@typeful.net> [2012-10-07 15:45:21+0200]
> > On Fri, Oct 05, 2012 at 05:17:18PM +0300, Roman Cheplyaka wrote:
> > > I can do that indeed, and I guess I could reimplement everything I have
> > > at the moment on top of HUnit.
> > >
> > > However, an important part of functionality isn't there at the moment —
> > > golden file management. You should be able to say, "for this test,
> > > take its current output and write it to the corresponding golden file".
> > >
> > > In order to do that, you need to have access to the list of golden tests
> > > in the suite. This is where implementation details of different test
> > > frameworks start to matter. Probably one can make an abstraction over
> > > test frameworks that would give the list of all golden tests.
> > > (Although when you start abstracting over test frameworks, which are
> > > abstractions themselves, it becomes somewhat funny.)
> >
> > Ok, makes sense.
> >
> > I'm looking forward to give it a try, and see how it compares to using
> > operating system primitives (say `cp') for "golden file management".
>
> 1. You often want to update not just one test, but all, or some of the
> tests (when you've made a change and verified that the changes in
> output are expected). Doing it in command line is certainly possible,
> but not trivial nor convenient.
> 2. For some tests (like goldenVsString) the output is not captured in a
> file, so using cp directly is not possible.
Yes and yes. I have no fixed expectations nor any idea how an "ideal"
interface would look like, but I guess there is room for improvement.
So I'm really looking forward to try it ;)
> > > Speaking of such functionality, correct me if I'm wrong, but neither
> > > HUnit nor hspec won't be able to support it anyway, because they
> > > represent tests as opaque IO actions.
> >
> > It would be easy to extend Hspec to support this in the same way you
> > extend test-framework to support this. It requires existentials; the
> > only substantial difference that I can see is that test-framework
> > already uses existentials, while Hspec does not.
>
> Well, if you are willing to make this change, then I'll try to do my
> part of the job and expose a useful abstraction.
If it gives us something that is useful from a users perspective, I'm
happy to make that change. I would hope that something like [1] works,
e.g.:
instance Example GoldenTest where
evaluateExample c = evaluateExample c . goldenTestToHUnitAssetion
exampleMetadata = Just . Metadata
That way the Typeable instance is optional.
Personally, I still think that it may be a good idea to first explore
the design space with test-framework before trying to abstract over it.
Cheers,
Simon
[1] https://github.com/sol/hspec/commit/6927f642aea44803b57c2b77548931f6865b0c38