managed fixtures [ann]
rssh
Good day, colleagues.
I want to announce managed fixture -- reimplementation of scalatest
fixture tests to situation, where our fixture have external state
(like relational database) and test author can
- mark each test with description of state preconditions and changes
in simple DSL
- provide functions for loading given state.
Then test engine can split all tests into parts, which require same
state and can be executed in parallel, then execute this parts in
order of execution, which minimize state loading. I.e. tests can be
written in next style:
<pre>
class MyFlatSpec extends managedfixture.[DBFixtureStateTypes.type]
{
val fuxtureStateTypes = DBFixtureStateTypes
val fuxtureAccess = DBFixtureAccess
import DBFixtureStateTypes.States._
behavior of "My datababase"
start state(INITIAL) finish state(WITH_ONE)
it should "be able to add user " in { db =>
.....
}
start state(DATASET1) change(nothing) parallel
it should "retrieve user with name Jon in our test dataset" in { db
=>
inTransaction {
val x = db.selectUser("Jon").headOption;
assert(x!=None)
assert(x.name == "Jon")
}
}
</pre>
Basic implementation and docs are here: http://rssh.github.com/fixture-state-management/
sources: https://github.com/rssh/fixture-state-management
It's now in experimental stage: works at first glance but not yet
fully tested.
In future I want to submit all into one of next version of scalatest,
if this is possible.
So, question - if this possible (?), and if yes -- how to do this
(?)
[May be mirror on github for pull requests ?]
Also it would be good to see some changes in next scalatest core:
- now managed-fixture suites contains reimplementations of each
DSL. This can be simplicifed if scalatest core would allows hook
arround registerTest .. registerBranch calls in fixture suite.
- distruptors - we can't load different database states at the
same time from different tests, so we need from distruptors ability to
run set of tasks inside external lock or may-be just callback call
when all tests in distruptors are finished [?]
// As alternative, we can implement own parallel execution scheme,
does not depend from distruptors.
Bill Venners
That looks interesting. From what I gather, you want to "annotate"
tests with a bit of code (in a DSL) that defines what fixture stuff
should happen before and after a test. Is that an accurate summary?
I'm guessing that "start state (<value>)" means that before the test
starts, the passed state should be setup. I'm not sure what "finish
state(WITH_ONE)" means. Can you clarify? Also, what does
"change(nothing)" mean? And what about "parallel"?
With "parallel" do you want to mark those tests that can run in
parallel as opposed to those that can't perhaps?
As far as github. I have considered having a mirror over there of
major releases. Almost like a tag, that people could do pull requests
against. I'll check into how easy that is. The reason I didn't move
over to begin with when I moved to Google code from java.net is I
didn't want to lose all the history. But it would be nice if people
could fork a release on GitHub and submit a pull request. Especially
if there was some easy way for me to get that back into Google code.
As far as Distributors go, I believe you can define your own
distributor that has the features you need. You'd need to downcast it
with a pattern match to your more specific type, but I think that
would be the way to do it. Why don't you try implementing that so I
can see what it looks like. It is not impossible that we could add
such a thing to ScalaTest, but I would want to make sure it is very
very well understood before it goes in. I need something like that in
the afterAll method, for example.
I'm not too keen on opening up registerTest and registerBranch. Those
are not generic enough to be lifecycle methods. For one thing, they
don't work on traits where tests are methods like Suite (and in the
future, the new Spec). For another example look at the new path
traits. Those methods don't work on them either. Instead what you've
done is actually the intended way to customize: you made some custom
style traits. They look like the built-in ones but support your
managed fixture feature.
That said, the other thing I was wondering about is did you consider
just executing the DSL code inside the test body? Instead of:
start state(INITIAL) finish state(WITH_ONE)
it should "be able to add user " in { db =>
//...
}
It might be something like:
it should "be able to add user " in {
withFixture(start state(INITIAL) finish state(WITH_ONE)) { db =>
// ...
}
}
This withFixture method you could place in a trait and users could mix
that into any style trait, even path traits or traits with test
methods, JUnit, TestNG, and so on. A lot more general. This probably
doesn't cover the parallel thing, but that depends on what you were
trying to achieve there. But I think it could take care of the setup
and teardown stuff if I'm understanding correctly what you're trying
to do.
Thanks. Cool stuff.
Bill
> --
> You received this message because you are subscribed to the Google
> Groups "scalatest-users" group.
> To post to this group, send email to scalate...@googlegroups.com
> To unsubscribe from this group, send email to
> scalatest-use...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/scalatest-users?hl=en
> ScalaTest itself, and documentation, is available here:
> http://www.artima.com/scalatest
--
Bill Venners
Artima, Inc.
http://www.artima.com
Esko Luontola
As far as github. I have considered having a mirror over there of
major releases. Almost like a tag, that people could do pull requests
against. I'll check into how easy that is. The reason I didn't move
over to begin with when I moved to Google code from java.net is I
didn't want to lose all the history. But it would be nice if people
could fork a release on GitHub and submit a pull request. Especially
if there was some easy way for me to get that back into Google code.
Bill Venners
Actually the problem was the *size* of the history. ScalaTest started
as a "port" from the Java test framework SuiteRunner, which was
started in 2001, thus it still had jar files checked in. So
ScalaTest's history includes a lot of jar files, and that is
impractical import into git. I did try it before deciding to move to
Google Code, and it never completed the import. So either I had to let
go of the history and start fresh, or stick with SVN. I chose the
latter. But I think I'd like to copy release tags over to my github
account so people can more easily contribute that way.
Say, did you get a chance to look at the path traits? Any suggestions
for improvements?
Thanks.
Bill
Kevin Wright
Bill Venners
I've been wanting to clean out all the dead branches. I haven't had
time, but most are unused. For the past year or so, we have been
putting 2.0 changes into the trunk, and doing 1.7 and 1.8 changes in
their own branches, but once 1.8 goes final thankfully we'll be back
in the trunk trunk heading to 2.0. Other than that, we tag release as
they come out under /tags, and that's pretty standard I think for svn.
These are the dirs I was thinking we'd pull into github, each tagged
release. Any other suggestions other than removing old branches once
we're done with them?
Thanks.
Bill
--
Esko Luontola
On Tuesday, 28 February 2012 23:38:20 UTC+2, Bill Venners wrote:
Actually the problem was the *size* of the history. ScalaTest started
as a "port" from the Java test framework SuiteRunner, which was
started in 2001, thus it still had jar files checked in. So
ScalaTest's history includes a lot of jar files, and that is
impractical import into git. I did try it before deciding to move to
Google Code, and it never completed the import.
Say, did you get a chance to look at the path traits? Any suggestions
for improvements?
I've also looked into this. The main problem with Scalatest is that the SVN branch structure is very much non-standard, so it's not particularly amenable to any automated conversion.
Esko Luontola
On Wednesday, 29 February 2012 00:08:15 UTC+2, Kevin Wright wrote:I've also looked into this. The main problem with Scalatest is that the SVN branch structure is very much non-standard, so it's not particularly amenable to any automated conversion.
I can have a look at the branch structure and estimate how much effort would be needed to import it to Git and make it clean.
Bill Venners
On Tue, Feb 28, 2012 at 2:50 PM, Esko Luontola <esko.l...@gmail.com> wrote:
>
> On Tuesday, 28 February 2012 23:38:20 UTC+2, Bill Venners wrote:
>>
>> Actually the problem was the *size* of the history. ScalaTest started
>> as a "port" from the Java test framework SuiteRunner, which was
>> started in 2001, thus it still had jar files checked in. So
>> ScalaTest's history includes a lot of jar files, and that is
>> impractical import into git. I did try it before deciding to move to
>> Google Code, and it never completed the import.
>
>
> Did you try importing on your own machine, or using GitHub's import service?
> The former should be more reliable. Even more reliable is if you run the SVN
> repository on localhost, to avoid network problems and make it a bit faster.
>
I was trying via the Github import service.
> After importing, it's possible to filter the history to remove the JAR
> files.
>
OK.
>
>> Say, did you get a chance to look at the path traits? Any suggestions
>> for improvements?
>
>
> I read the documentation. Looks good.
>
> The documentation about ignored tests says "Note that a separate instance
> will be created for an ignored test". Does this mean that the fixture setup
> will be executed, but then no tests are executed? If the ignored test is
> followed by siblings, then it should be possible to execute one of them, to
> avoid wasting the effort which was spent on running the fixture setup.
>
Good idea, and you're right again. I'll jot that down as a potential
performance enhancement for the future. It does complicate the "one
instance per leaf node" explanation of the execution model. I tried
hard to make it easy for users to form an accurate mental model of
what's going on, to help them reason about their code, so I like that
the explanation is simple without exceptions like "one instance per
leaf node, unless there are ignores with siblings, in which case it
will keep executing until..."
>
> On Wednesday, 29 February 2012 00:08:15 UTC+2, Kevin Wright wrote:
>>
>> I've also looked into this. The main problem with Scalatest is that the
>> SVN branch structure is very much non-standard, so it's not particularly
>> amenable to any automated conversion.
>
>
> Git is very flexible in that regard (though it would require more or less
> manual work). The history can be modified to fix any weirdness, and even
> merges can be added to histories from pre-1.5 SVN repositories using "Git
> grafts".
>
> I can have a look at the branch structure and estimate how much effort would
> be needed to import it to Git and make it clean.
>
Well I wouldn't bother until I clean it all up. Only a few branches
will likely survive such a purge.
Also, now that the history is safely in Google Code, I could also
possibly just actually develop via github repos, and then tag them
back on Google Code once released. The problem I had before was that
ScalaTest started out on java.net, and when Oracle bought Sun, they
decided to migrate off of Collabnet, whose platform really was a pain
to use actually. So that was a good decision, but it meant that my
history was going away there so I need to find a new home for it. Long
story but that took 3 painful months because of the trouble I had
importing to github and a bug importing between subversion versions.
After being told I'd need to pay $1000 to get Collabnet to make a zip
file of my subversion files, I finally ended up importing from
java.net to the artima server using an older subversion version, then
upgrading subversion on the artima server for that repo to work around
the subversion version bug, and finally I was able to import it from
the artima server to Google Code.
In some way I'd like to get the project on Github because it is so
much easier for people to clone, make tweaks, and contribute them
back. As a next step I'll clear out the dead branches, sometime after
1.8 goes to RC probably.
Bill
Bill Venners
Ah, I see. Yes that's true. On java.net they put you under trunk/app,
because they also had the home page under /trunk. So that's in the
history. After I moved to Google Code I probably moved things to
/trunk, /branches, and /tags like I was used to.
Bill
Esko Luontola
In some way I'd like to get the project on Github because it is so
much easier for people to clone, make tweaks, and contribute them
back. As a next step I'll clear out the dead branches, sometime after
1.8 goes to RC probably.
Bill Venners
Cool. Thanks.
Bill
Ruslan Shevchenko
> Hi rssh,
>
> That looks interesting. From what I gather, you want to "annotate"
> tests with a bit of code (in a DSL) that defines what fixture stuff
> should happen before and after a test. Is that an accurate summary?
>
This is accurate summary of near 1/2 of functionality. I.e. full
summary can be:
1. define what fixture stuff should happen before and after a test.
2. run group of tests in some sequential order.
3. prevent setting the state of fixture to different states at the
same time in parallel and allow parallel execution of
tests on the same state.
Let I describe use-case for which such type of tests is needed:
So, let we develop software which later deployed in multiple client location.
For example -- let we build CRM system on top of relational database.
Usually, when we build our system, we start with empty database (which
is our state0), than
we have test which add 100 client records (which is our state1) than
we have sequence of
tests which can run in state1 and change nothing (state) than we check
removing of
one user, than we entry to our CRM information about deals for left
99 persons, than
we try to remove user which have associated information ... etc.
I.e. we model life-cycle of client as sequence of database states.
We can say that can exists 3 types of tests:
- tests, which do transition change (add 100 user: start
state(empty) finish state(justCreated100Users)
- tests, which check something and change nothing
Jon is in our db: (start state(justCreated100Users)
change(nothing) parallel)
- tests, which leave state after themself undefined
remove random used.
Now, .let look on next use-case:
1. administrator set price of item X to x1
2. client bought one such item and receive invoice
3. administrator set price of item X to x2
4. client bought one such item and receive invoice
5. summary of invoices must be x1+x2
And imagine that we want to add one to set of our tests.
1. we must have state where exists record for item X and for client
which bought this item and admin which set the price. As
alternative, we can setup such state inside each tests, but this will
be long.
2 may-be we introduce next state (which we know) where exists users
and items and history of payments.
May be we will want use it as basic for next test. (for example --
delete item X fro database, look at summary of
invoices again)
Also note, that run this test inside one database transaction is
possible only in simple cases,
in real life we have interactions with other systems, etc. So
fully create all environment inside each tests
will not only cost time, but in some cases impossible.
> I'm guessing that "start state (<value>)" means that before the test
> starts, the passed state should be setup. I'm not sure what "finish
> state(WITH_ONE)" means. Can you clarify? Also, what does
test which change state to state(WITH_ONE). Example of such state --
adding 100 users.
> "change(nothing)" mean? And what about "parallel"?
>
tests which does not change nothing. (for example -- implemented
via rollbacked transaction)
> With "parallel" do you want to mark those tests that can run in
> parallel as opposed to those that can't perhaps?
>
Yes. How this implemented: we collect such group of tests in nested
suites and
override runNestedSuite to run group tests in given order. Tests
which share same
start state, change nothing and marked as parallel can be executed in parallel.
> As far as github. I have considered having a mirror over there of
> major releases. Almost like a tag, that people could do pull requests
> against. I'll check into how easy that is. The reason I didn't move
> over to begin with when I moved to Google code from java.net is I
> didn't want to lose all the history. But it would be nice if people
> could fork a release on GitHub and submit a pull request. Especially
> if there was some easy way for me to get that back into Google code.
>
Exist bidirectional bridge git-svn http://schacon.github.com/git/git-svn.html
[it was too complex for me when I was have project with few branches,
but whith one main branch its works]
> As far as Distributors go, I believe you can define your own
> distributor that has the features you need. You'd need to downcast it
> with a pattern match to your more specific type, but I think that
> would be the way to do it. Why don't you try implementing that so I
> can see what it looks like. It is not impossible that we could add
> such a thing to ScalaTest, but I would want to make sure it is very
> very well understood before it goes in. I need something like that in
> the afterAll method, for example.
>
Thank's I will look.
> I'm not too keen on opening up registerTest and registerBranch. Those
> are not generic enough to be lifecycle methods. For one thing, they
> don't work on traits where tests are methods like Suite (and in the
> future, the new Spec). For another example look at the new path
> traits. Those methods don't work on them either. Instead what you've
> done is actually the intended way to customize: you made some custom
> style traits. They look like the built-in ones but support your
> managed fixture feature.
>
Hmm, good. [I was afraid that translating DSL to fixture DSL layes
is overkill]
> That said, the other thing I was wondering about is did you consider
> just executing the DSL code inside the test body? Instead of:
>
> start state(INITIAL) finish state(WITH_ONE)
> it should "be able to add user " in { db =>
> //...
> }
>
> It might be something like:
>
> it should "be able to add user " in {
> withFixture(start state(INITIAL) finish state(WITH_ONE)) { db =>
> // ...
> }
> }
>
> This withFixture method you could place in a trait and users could mix
> that into any style trait, even path traits or traits with test
> methods, JUnit, TestNG, and so on. A lot more general. This probably
> doesn't cover the parallel thing, but that depends on what you were
> trying to achieve there. But I think it could take care of the setup
> and teardown stuff if I'm understanding correctly what you're trying
> to do.
>
Hmm, this can be useful for use-cases, when order of evaluation is unimportant,
i.e. loading of state is cheap. I will think about real-life example.
Bill Venners
Very interesting. I think I see where you want to go now. Nevermind
what I said about withFixture for the time being. After reading your
latest email, I think this sounds more like a job for overriding
runTests and a creative application of tagging.
ScalaTest tests can be tagged in any style trait, and those tags can
be accessed and used inside any lifecycle method (including runTests).
My intuition is that there could be a way to tag tests to indicate the
required starting state and the ending state, and whether or not it
can run in parallel with others that have the same starting state.
(Maybe you could make a simplifying assumption that any tests that
require the same starting state and do not change the state can run in
parallel.)
Anyway, to do this the user would need to define tags for all their
starting and ending states, including one for the "NoChange" ending
state. (Or, maybe if they don't specify an end state you could
interpret that to mean NoChange, but that sounds error prone and less
obvious to me.)
Then, you give them a way using your DSL to specify the order to
perform the tests in based on their tags. Or maybe you could give them
a way just to say the starting state, and use some algo to find an
optimum way through the tests given the state changes. Let's call this
the "script."
Lastly, override runTest to read the script and the tags, and figure
out what order to run tests in, and run them in that order. As far as
running in parallel, you're right it would be nice to be able to use
the distributor if it is passed in, but you could also just fire up
some threads and run in parallel yourself. Then you could do a
countdown latch and figure out when to move on. But it does seem to me
that if a Distributor is passed, its pool of threads should be used.
And if a Distributor is *NOT* passed, maybe that means you should do
those things sequentially anyway. Perhaps. Maybe you want to always do
things in parallel that can, and if a distributor isn't passed you use
your own.
But regardless, the upshot is that I think if you took this approach,
you'd have a general library that could be used with any strait
including ones where tests are methods. You may need some enhancements
from ScalaTest in how tags are done and/or the Distributor. Might be
worth coding up to see what you discover is actually needed from
ScalaTest.
Bill