Tips for getting started with testing for first large app?
Ram
Well I have this fairly large application with restful authentication
and a few other plugins in it. I developed it wholly otherwise.
This is my first ROR app. And I learnt of Test Driven Development
quite late.
Ive reached the point where I need to cover my code with good test
coverage.
Ive done quite a lot of reading on testing but I still need to take
the plunge into it and I have no idea where to start.
My request to those of you out there with the experience to put me in
the right direction..
a few questions..
- should i start with unit tests? how paranoid should i be about them
(if at all there's a unit of measurement for that)?
- i understand too much dependency on fixtures can make my tests
brittle. So i keep fixtures to a minimum? when exactly should I
absolutely use a fixture?
- what tools should i definitely use besides test/unit? RSpec,
Autotest, Cucumber, Selenium, ... ?
Any and many tips welcome as welcome can be!
What would probably be most important is how do I analyze my code to
see what tests are most important and start writing them? How do I
approach writing tests?
Thank you!
Fernando Perez
- Don't bother with RSpec's hype. I tried it and got me pissed off so
many times. Don't care about autospec, it will distract you more than it
will help. Test::Unit is much better to use. AWDWR has a good enough
chapter to get you started.
- Use mocha to mock and/or stub your method calls so that you test
objects as isolated as possible from other objects.
- rcov can come in handy to know what parts of your app are not yet
tested. Be careful: 100% test coverage, doesn't mean your app is still
correctly tested.
- Don't ever use fixtures at all! Run away from any person who tells you
they are fine, even if he is famous and rich.
- Write your tests (even unit tests) so that they hit DB as least as
possible. When tests have to hit DB such as :include or :joins, then
setup sqlite in-memory for faster testing.
- I have tried plenty factories: Machinist, factory girl, etc. I even
wrote my own, but once you understand how to write tests that don't hit
DB, you'll discover you will probably not need a factory at all. Except
when you must hit DB, it can be a little bit easier to insert valid
objects in DB. But at the beginning don't bother.
- use a notifier plugin on your production server to catch any
additional edge case you haven't thought of (I use getexceptional), and
write tests to prevent against them in the future
Writing tests is easy when you know how to write them and which tools to
use.
Remember to have fun. If you are not having fun (even writing tests)
then something is wrong.
--
Posted via http://www.ruby-forum.com/.
Marnen Laibow-Koser
> My advise:
>
> - Don't bother with RSpec's hype. I tried it and got me pissed off so
> many times.
I completely disagree. RSpec works very well, and its syntax is
generally more natural (at least to me) than Test::Unit. In what way
does RSpec not work for you?
> Don't care about autospec, it will distract you more than it
> will help.
Again, I disagree 100%. I find it extremely helpful to get immediate
feedback when my tests break. If you find autospec "distracting", I
wonder if you're really paying enough attention to your tests...
[...]
> - Use mocha to mock and/or stub your method calls so that you test
> objects as isolated as possible from other objects.
Yes. Mock everything that isn't what you're trying to test. (You only
need Mocha if you're not using RSpec, which has its own mocking
framework; however, it too will work with Mocha if you prefer it.)
>
> - rcov can come in handy to know what parts of your app are not yet
> tested. Be careful: 100% test coverage, doesn't mean your app is still
> correctly tested.
Heck yes.
>
> - Don't ever use fixtures at all! Run away from any person who tells you
> they are fine, even if he is famous and rich.
Agreed, with the possible caveat that Phlip seems to like them, and I
trust everything *else* he says about testing. :)
>
> - Write your tests (even unit tests) so that they hit DB as least as
> possible.
Yes, but don't go through silly contortions to religiously avoid the DB
altogether if it's more trouble than it's worth. RSpec's mock_model and
stub_model are lovely here.
> When tests have to hit DB such as :include or :joins, then
> setup sqlite in-memory for faster testing.
I don't see how this would work if you're using the DB in any reasonably
sophisticated way, given that SQLite tends not to support such things.
[...]
>
> - use a notifier plugin on your production server to catch any
> additional edge case you haven't thought of (I use getexceptional), and
> write tests to prevent against them in the future
Good advice.
>
>
> Writing tests is easy when you know how to write them and which tools to
> use.
>
> Remember to have fun. If you are not having fun (even writing tests)
> then something is wrong.
:)
Best,
--
Marnen Laibow-Koser
http://www.marnen.org
mar...@marnen.org
Phlip
> Well I have this fairly large application with restful authentication
> and a few other plugins in it. I developed it wholly otherwise.
> This is my first ROR app. And I learnt of Test Driven Development
> quite late.
> Ive reached the point where I need to cover my code with good test
> coverage.
Open a new Rails project in a new folder.
Rewrite every tiny detail of your old project into the new one.
If your old project has a model Foo, you start with script/generate model Foo,
and get the foo_test.rb. You edit that to write a _useful_ test on one or two
fields of Foo, and you pass the test by adding those fields to the migration and
doing something with them in a method inside Foo.
(BTW, don't add a migration for each field - edit the existing migrations until
you deploy them...)
Going this way prevents you from cheating, and helps you refactor. The new
project should be finished very soon.
> - should i start with unit tests? how paranoid should i be about them
> (if at all there's a unit of measurement for that)?
The test to code ratio should be 2:1. And start with useful features, not
layers. To write a useful feature, write new tests and code into each layer that
it needs.
> - i understand too much dependency on fixtures can make my tests
> brittle. So i keep fixtures to a minimum? when exactly should I
> absolutely use a fixture?
Using my schedule, your fixtures will not be brittle. Just don't write
assertions that expect useless things, like a count of found records. Test that
all the records matched the find criteria.
> - what tools should i definitely use besides test/unit? RSpec,
> Autotest, Cucumber, Selenium, ... ?
Those are mostly for documenting features to customers, not for TDD. Stick with
unit tests, because RSpec would only add clutter if used as a TDD tool.
Autotest is because all the Rails editors truly suck, and do not let you run one
test at a time. (Yes, textmate lets you run one test at a time - if you rotate
the freaking document back to the correct test between every edit. The quality
of these editors makes me wonder if anyone who develops them actually uses TDD.)
> What would probably be most important is how do I analyze my code to
> see what tests are most important and start writing them? How do I
> approach writing tests?
Add features to your new project in order from highest business value and on down.
--
Phlip
Fernando Perez
> generally more natural (at least to me) than Test::Unit. In what way
> does RSpec not work for you?
Each time I would write a test, something would go wrong in the spec,
specially when specing controllers, so I only speced models. With
Test::Unit everything works well and I can test my controllers and views
so easily, I definitely don't need to watch any screencast to teach me
how it works.
Moreover to my great surprise, Test::Unit's syntax is now pretty so you
can write:
test "..." do
end
And behind the scenes it transforms it into def
my_cool_test_with_plenty_underscores
I prefer 100 times test::unit to rspec, and when people ask about which
tool to use, I recommend test::unit and I am glad to share my experience
in testing so people won't do the same mistakes I did.
> Again, I disagree 100%. I find it extremely helpful to get immediate
> feedback when my tests break. If you find autospec "distracting", I
> wonder if you're really paying enough attention to your tests...
No autospec is distracting as you often find yourself waiting for the
test result, I won't even talk about growl which makes things even
worse.
I got fed up when saving 3 files quickly and one after the other just
because I did some indentation editing. Autospec would then run 3 times
for nothing! Waste of time and resource. I prefer to run my tests
manually.
> Yes, but don't go through silly contortions to religiously avoid the DB
> altogether if it's more trouble than it's worth.
You are right. But for instance, instead of update_attributes I prefer
to set the attributes and then call save! which I can stub in a test and
still have the attributes get set, assert their values and save a slow
DB hit.
By doing this I divided by 10 my testing time!!! Tests that hit DB are
in their own file so that they don't slow down the other tests.
> I don't see how this would work if you're using the DB in any reasonably
> sophisticated way, given that SQLite tends not to support such things.
What things does sqlite not support? joins?
Pick your poison whether it is rspec, test/unit, whatever, as long as
you test, your choice is always a good one.
Fernando Perez
app could have a bug and you would still not detect it. And remember
that the higher the ratio, the more pain it will be to maintain the
tests. So don't care about ratio.
> Autotest is because all the Rails editors truly suck, and do not let you
> run one test at a time.
Is there a benefit to run 1 test at a time? I don't mind running 20
tests each time because they take less then 1 second to perform.
Phlip
> Yes. Mock everything that isn't what you're trying to test.
Absolutely false. Mock almost nothing - the clock, the wire out of your
workstation, and system errors.
Everything else should be so clean and decoupled that you can use it as stubs
without mocking it. Mocks just tell your tests what they think they want to
hear, and the mocks interfere with refactoring and decoupling.
Phlip
>> - Don't bother with RSpec's hype. I tried it and got me pissed off so
>> many times.
>
> I completely disagree. RSpec works very well, and its syntax is
> generally more natural (at least to me) than Test::Unit. In what way
> does RSpec not work for you?
RSpec is for BDD, like FIT or FITnesse. It's for communicating with your
business-side analysts about all your features. For raw development, it adds
clutter to the syntax without adding much new innovation to the test flow.
And, yes, all the developers are eating it up like popcorn...
>> Don't care about autospec, it will distract you more than it
>> will help.
>
> Again, I disagree 100%. I find it extremely helpful to get immediate
> feedback when my tests break. If you find autospec "distracting", I
> wonder if you're really paying enough attention to your tests...
Right - leave it running and save your files over and over again. Each time,
practice successfully predicting what the tests will do.
And it bears repeating that TDD requires writing new tests and
_watching_them_fail_, before adding the tested code. Don't just go write the
test and then write the code. Don't write the code and then get around to
testing what you got. Only write new code in response to failing tests.
And refactor refactor refactor. If your boss doesn't like it, keep refactoring
until you get fired!
>> - Don't ever use fixtures at all! Run away from any person who tells you
>> they are fine, even if he is famous and rich.
>
> Agreed, with the possible caveat that Phlip seems to like them, and I
> trust everything *else* he says about testing. :)
I _enjoy_ their fragility (in their current incarnation). It forces you to
review your tests.
If anything in TDD is fragile, or leads to too much debugging (p statements),
revert and try again. TDD makes fragility an asset.
>> - Write your tests (even unit tests) so that they hit DB as least as
>> possible.
>
> Yes, but don't go through silly contortions to religiously avoid the DB
> altogether if it's more trouble than it's worth. RSpec's mock_model and
> stub_model are lovely here.
Again, use the database to provide ready-made objects, and TDD your find()
statements at the db. The decoupling will come, and "don't hit the db in
testing" is all but a myth. (Darn you, Mike Feathers!)
>> Writing tests is easy when you know how to write them and which tools to
>> use.
>>
>> Remember to have fun. If you are not having fun (even writing tests)
>> then something is wrong.
>
> :)
Yep. Tests should be easier and easier to write, until you feel guilty and think
they must not be doing anything for you.
--
Phlip
Marnen Laibow-Koser
[...]
>
> RSpec is for BDD, like FIT or FITnesse.
I would not say that FIT is a BDD tool, at least the way I understand
that term.
> It's for communicating with your
> business-side analysts about all your features.
No. That's what a tool like FITnesse or Cucumber is for. Perhaps your
usage patterns are different, but for me at least, RSpec is for
describing the behavior of modules, but at a much more technical level
than business-side analysts would be interested in. I can't see using
RSpec for business-domain tests at all!
> For raw development, it
> adds
> clutter to the syntax without adding much new innovation to the test
> flow.
How so? I use RSpec for all raw development, largely for two reasons:
* I like the syntax.
* I like the focus on external interface rather than (Test::Unit's
orientation toward?) internal behavior.
Where's the clutter? I really don't understand. Test::Unit feels *far*
more cluttered to me, or did the last time I tried it.
>
> And, yes, all the developers are eating it up like popcorn...
Sure! It's fun to use. :)
[...]
>
> And it bears repeating that TDD requires writing new tests and
> _watching_them_fail_, before adding the tested code. Don't just go write
> the
> test and then write the code. Don't write the code and then get around
> to
> testing what you got. Only write new code in response to failing tests.
I agree wholeheartedly (although in practice I'm not always as good at
this as I should be).
[regarding fixtures]
> I _enjoy_ their fragility (in their current incarnation). It forces you
> to
> review your tests.
I *really* don't understand this. Could you clarify?
>
> If anything in TDD is fragile, or leads to too much debugging (p
> statements),
> revert and try again.
Doesn't this contradict the previous sentence?
> TDD makes fragility an asset.
Doesn't *this* contradict the previous sentence?
[...]
> Again, use the database to provide ready-made objects, and TDD your
> find()
> statements at the db. The decoupling will come, and "don't hit the db in
> testing" is all but a myth. (Darn you, Mike Feathers!)
It's probably a very good general idea (or ideal), but Rails
ActiveRecord is too intimate with the DB to make it practical. I *do*
think it's worth not having the tests *overly* dependent on the DB,
though.
<sarcasm>But Jay Fields says that Rails tests shouldn't touch the DB.
That's reason enough to agree with you that it's a myth.</sarcasm>
>
>>> Writing tests is easy when you know how to write them and which tools to
>>> use.
>>>
>>> Remember to have fun. If you are not having fun (even writing tests)
>>> then something is wrong.
>>
>> :)
>
> Yep. Tests should be easier and easier to write, until you feel guilty
> and think
> they must not be doing anything for you.
I like that. But Master Phlip, I still think my tests are
significant...I guess I have not yet achieved enlightenment!
>
> --
> Phlip
Fernando Perez
> to review your tests.
Fixtures make tests really slow, and they will eventually not reflect
the actual model if you forget to update some attribute you changed.
Moreover, you have to actually open the fixture file to know what's in
it. I prefer to setup each test inside the test method so that I can
immediately predict what should be the end result.
> The decoupling will come, and "don't hit the db in
> testing" is all but a myth. (Darn you, Mike Feathers!)
Although everybody talks about him, I still don't know who he is...
Anyway, I got the idea of not hitting the DB from Jay Fields, and from
my own experience. Havin super fast tests is so enjoyable.
> Absolutely false. Mock almost nothing - the clock, the wire out of your
> workstation, and system errors.
I think nobody is right nor wrong. The truth lies somewhere in between.
I know that mocking, subbing and using unsaved objects can let in some
bugs in my tests, but on the other hand it speeds up the tests. So I
guess some objects can be mocked/stubbed/unsaved in order to save time,
and some objects must be saved in order to make sure they are valid and
legit in DB.
What's funny is that in 2009, there is still a lot of debate about the
testing workflow, although programming has been around for a few years
:-)
Fernando Perez
> orientation toward?) internal behavior.
I don't see how test::unit focuses on internal behavior and how rspec
focuses on external interface. For me only the developer and his love of
the tool makes the difference.
Marnen Laibow-Koser
[...]
> Mock almost nothing - the clock, the wire out of your
> workstation, and system errors.
I was at a loss to understand this until I read the following sentence.
>
> Everything else should be so clean and decoupled that you can use it as
> stubs
> without mocking it.
I suppose I was speaking inexactly. I think of a stub as a special case
of mock, and was writing from that point of view. My original statement
should have read:
Stub everything that isn't what you're directly testing. If for some
reason you can't stub it, *then* mock it.
Or more simply:
If it isn't what you're testing, fake it!
Is that a more acceptable method?
> Mocks just tell your tests what they think they want
> to
> hear,
True, although sometimes that can be useful to isolate problems.
> and the mocks interfere with refactoring and decoupling.
How? By making a test too dependent on another object's interface? But
don't stubs do that too?
Brandon Olivares
> -----Original Message-----
> From: rubyonra...@googlegroups.com [mailto:rubyonrails-
> ta...@googlegroups.com] On Behalf Of Marnen Laibow-Koser
> Sent: Friday, April 17, 2009 10:56 AM
> To: rubyonra...@googlegroups.com
> Subject: [Rails] Re: Tips for getting started with testing for first
> large app?
>
>
> Phlip wrote:
> [...]
> >
> > RSpec is for BDD, like FIT or FITnesse.
>
> I would not say that FIT is a BDD tool, at least the way I understand
> that term.
>
> > It's for communicating with your
> > business-side analysts about all your features.
>
> No. That's what a tool like FITnesse or Cucumber is for. Perhaps your
> usage patterns are different, but for me at least, RSpec is for
> describing the behavior of modules, but at a much more technical level
> than business-side analysts would be interested in. I can't see using
> RSpec for business-domain tests at all!
>
Brandon
Hans
I started with fixtures, then in my second attempt to be a good tester
I used factory girl, that worked fine for me.
In my current third attempt, after upgrading to rails 2.3.2, I do not
know how to proceed.
I have a lot of dependencies, which makes fixtures very hard and time
consuming to use, but you learn a lot about your application when you
use them. (Are there any tools that generates fixtures considering all
associations?). Most people tell you that you should use a factory
approach instead (see e.g. the other replies about fixtures) and I am
thinking of using machinist, but ..... I hesitate.
The reason is that the rails developing team has selected the fixture
approach (which are their reasons?) and the development of rails seems
to follow that line. Is it not then wiser to go that way and take
advantage of the improvements and tools that will be realized?
So what to select factories or fixtures ?????
On Apr 17, 2:52 pm, Fernando Perez <rails-mailing-l...@andreas-s.net>
wrote:
Phlip
> How so? I use RSpec for all raw development, largely for two reasons:
>
> * I like the syntax.
you.should be(:flogged)
> [regarding fixtures]
>> I _enjoy_ their fragility (in their current incarnation). It forces you
>> to
>> review your tests.
>
> I *really* don't understand this. Could you clarify?
When we say "fragile", we mean a test failed because we changed the fixtures for
another test.
When we say "failed", we mean SUCCEEDED in SLOWING YOU DOWN. A car has brakes,
headlights, and red lights not so we can stop, but so we can go very fast
between stoppings.
When another test fails unexpectedly, you look at it and think about it. It
might remind you of the original meaning of the fixture you are changing. It
might force you to improve the assertion, to actually detect that test's
situation better.
Or it might not, and you just get pissed off, and make a better change to the
fixtures, or you might clone your fixture, or you might even start abusing mocks
in frustration.
Either way, it's the overlap between tests, test-side code, and production code,
that keeps everyone honest.
>> If anything in TDD is fragile, or leads to too much debugging (p
>> statements),
>> revert and try again.
>
> Doesn't this contradict the previous sentence?
Revert early and often.
>> TDD makes fragility an asset.
>
> Doesn't *this* contradict the previous sentence?
Reverting forces you to try a narrower change next time. Narrow changes are
overwhelmingly better than wide sweeping changes.
Phlip
> Concerning fixtures
> I started with fixtures, then in my second attempt to be a good tester
> I used factory girl, that worked fine for me.
> In my current third attempt, after upgrading to rails 2.3.2, I do not
> know how to proceed.
>
> I have a lot of dependencies, which makes fixtures very hard and time
> consuming to use
Expensive setup is a design smell. How about all your tests only use the fewest
records possible?
> The reason is that the rails developing team has selected the fixture
> approach (which are their reasons?) and the development of rails seems
> to follow that line. Is it not then wiser to go that way and take
> advantage of the improvements and tools that will be realized?
The reason the Rails team went with the current style of Fixtures was because in
2004 there was _nothing_ in the sector "TDD for databases, with migrations and
refactoring", besides high-end guru consultant work at big companies. So the
Railsters thought outside the box, and decided to load a complete DB for each
test. To remain efficient, they loaded YAML directly into INSERT statements,
bypassing the models.
For the first hundred records, that's not fragile...
> So what to select factories or fixtures ?????
Write your first test and let it tell you. The point of all of this is nothing
is carved in stone...
Phlip
>> Mock almost nothing - the clock, the wire out of your
>> workstation, and system errors.
>
> I was at a loss to understand this until I read the following sentence.
Only mock things that are too expensive to call directly.
We are not writing QA "Unit Tests" - the kind CPU manufacturers write. They need
pure isolation between units, so when a test fails it only implicates one unit.
TDD does not do that.
> Stub everything that isn't what you're directly testing. If for some
> reason you can't stub it, *then* mock it.
>
> Or more simply:
>
> If it isn't what you're testing, fake it!
>
> Is that a more acceptable method?
No. I meant "expensive setup is a design smell".
Given an object model Post -> Author -> Access, a test on the Post->Author
relationship could conceivably ignore the Access. And a test on Author->Access
could ignore Post. The -> goes nowhere; it is stubbed.
Fixtures are among the systems that keep object models easy to whip out and use.
If our system were as coupled as a Vendor-Lock-In system, such as MFC, we can't
just create one object. We have to pull in everything just to construct and use
one stinking CEdit control. Don't do that. All objects should construct cleanly
without their runtime dependencies, if you don't need them.
>> and the mocks interfere with refactoring and decoupling.
>
> How? By making a test too dependent on another object's interface? But
> don't stubs do that too?
When you change the raw object, what part of the system forces you to remember
to change the mock the same way?
Marnen Laibow-Koser
[...]
> We are not writing QA "Unit Tests" - the kind CPU manufacturers write.
> They need
> pure isolation between units, so when a test fails it only implicates
> one unit.
Aren't we, though? What's the point of a test that doesn't tell you
what failed?
> TDD does not do that.
That statement is in direct contradiction to just about everything else
I've read and striven for on the subject, at least if we're talking
about fairly granular unit tests. Acceptance/functional/integration
tests are another matter, of course.
> No. I meant "expensive setup is a design smell".
Then I guess I shouldn't be using Rails. I don't know how to test
anything based on ActiveRecord without reasonably expensive (in dev
time, anyway) setup. What am I missing?
>
> Given an object model Post -> Author -> Access, a test on the
> Post->Author
> relationship could conceivably ignore the Access.
Right.
> And a test on
> Author->Access
> could ignore Post. The -> goes nowhere; it is stubbed.
Yes. No argument there.
>
> Fixtures are among the systems that keep object models easy to whip out
> and use.
Why not mock/stub model objects instead? I don't see a single benefit
from using fixtures, except that the setup expense is shifted from the
developer (lots of mocks) to the computer (excessive DB access resulting
in *slow* tests -- see Asynchronous Unit Testing antipattern). Is there
a third way that I'm missing?
> If our system were as coupled as a Vendor-Lock-In system, such as MFC,
> we can't
> just create one object. We have to pull in everything just to construct
> and use
> one stinking CEdit control.
I'm not familiar with MFC, but I *think* I understand. However, it
seems to me that fixtures for everything is doing exactly what you're
saying *not* to do.
> Don't do that. All objects should construct
> cleanly
> without their runtime dependencies, if you don't need them.
I agree. That's why a test double should be sufficient.
[...]
> When you change the raw object, what part of the system forces you to
> remember
> to change the mock the same way?
In many cases, an integration/functioanl test would break. In
others...well, I'm not sure. I'll have to think about that.
But this is a red herring: mocks and stubs (my original question) suffer
from the same problem here. OTOH, I'm not sure it's appropriate to
couple your unit tests to a live foreign object. Hmmm...
I really appreciate being able to discuss these issues with those who
know so much more about them than I do. Thanks for making me think!
Phlip
>> We are not writing QA "Unit Tests" - the kind CPU manufacturers write.
>> They need
>> pure isolation between units, so when a test fails it only implicates
>> one unit.
>
> Aren't we, though? What's the point of a test that doesn't tell you
> what failed?
Under TDD, the element that failed must be among your recent edits.
Therefor, tests are free to cross-polinate any of your modules together...
>> No. I meant "expensive setup is a design smell".
>
> Then I guess I shouldn't be using Rails. I don't know how to test
> anything based on ActiveRecord without reasonably expensive (in dev
> time, anyway) setup. What am I missing?
I don't mean this is expensive...
foo = foos(:my_foo_fixture)
...I mean a zillion lines of pure clutter, building object after object and
glueing them all together just the right way. That's expensive.
>> Fixtures are among the systems that keep object models easy to whip out
>> and use.
>
> Why not mock/stub model objects instead? I don't see a single benefit
> from using fixtures, except that the setup expense is shifted from the
> developer (lots of mocks) to the computer (excessive DB access resulting
> in *slow* tests -- see Asynchronous Unit Testing antipattern). Is there
> a third way that I'm missing?
Better the computer is slow than the developer. That's orders of magnitude
difference there.
If your tests get too slow, install CruiseControl.rb for a build server.
--
Phlip
Ram
ok that was some intense stuff. I dint understand a lot of stuff going
around because, as established before, I dont have experience writing
tests.
But the following are the things Im taking out of this discussion.
- jump in the dirt. Do TDD for your existing app by starting all over
again if you have to. Thats the only way to go (?) / thats the best
way to go to become a better programmer (?)
- i dont *absolutely* need any external tools besides test::unit. I
probably would use the right tools if i ever felt the need for them in
the first place
- use fixtures when I have to test something that uses complex
associations between models. especially when the alternative is to
spend a lot of time and brain on mocking/stubbing them instead. (I
have to mention I dont know the difference/similarities/functions of
mocking and stubbing completely)
- i dont have any client documentation to show for these tests. theyre
only for the sake of testing my code itself. So ill skip over RSpec
and Cucumber (until i feel a need for them)
- test code keeps evolving. I need to optimize test code as much (or
more ?) as i optimize dev code. and tests breaking out of nowhere will
be a way of life (?)
i am going to be adding features to the app in between too. anything I
have to keep in mind while doing that?
when do I start on integration and functional tests?
Also, any good articles that could help me gain some perspective on
the task i have ahead of me?
my 2 cents worth.. i personally like the feel autotest gives.
especially for TDD. but hey.. no experience. only tryouts and
screencasts.
Please DO let me know if I have taken away the right things from here
*to get started* with writing tests for my existing app.
I hope this discussion goes further and provides insights for many
more in my place.
Thanks Fernando, Marnen, Phlip, Brandon and Hans :)
Marnen Laibow-Koser
> <Yikes!>
>
> ok that was some intense stuff. I dint understand a lot of stuff going
> around because, as established before, I dont have experience writing
> tests.
> But the following are the things Im taking out of this discussion.
>
> - jump in the dirt. Do TDD for your existing app by starting all over
> again if you have to. Thats the only way to go (?) / thats the best
> way to go to become a better programmer (?)
Starting over, as Phlip suggested, is pretty drastic, but it *will* make
sure that everything is tested.
>
> - i dont *absolutely* need any external tools besides test::unit.
Right, although you should try RSpec as a replacement for Test::Unit and
see which you prefer.
> I
> probably would use the right tools if i ever felt the need for them in
> the first place
Huh?
>
> - use fixtures when I have to test something that uses complex
> associations between models. especially when the alternative is to
> spend a lot of time and brain on mocking/stubbing them instead. (I
> have to mention I dont know the difference/similarities/functions of
> mocking and stubbing completely)
Check out Martin Fowler's article "Mocks aren't Stubs", as well as the
RSpec docs. There's also a hell of a lot of info in the wiki at c2.com,
though some of it may not apply to what you're doing...
>
> - i dont have any client documentation to show for these tests. theyre
> only for the sake of testing my code itself. So ill skip over RSpec
> and Cucumber (until i feel a need for them)
Cucumber is definitely for client-documentation-type tests. RSpec,
however, is useful as a replacement for Test::Unit if (like me) you find
that it's easier to use it for unit tests. (Phlip's characterization of
RSpec is IMHO inaccurate.)
>
> - test code keeps evolving. I need to optimize test code as much (or
> more ?) as i optimize dev code.
Maybe. But differently. Optimize test code for readability and
accuracy -- unlike production code, you're not going to be running
thousands of tests every second. :)
> and tests breaking out of nowhere will
> be a way of life (?)
Perhaps. At least you'll know right away when something needs work.
>
> i am going to be adding features to the app in between too. anything I
> have to keep in mind while doing that?
Try to get everything working before adding the next feature.
> when do I start on integration and functional tests?
Good question. And many Rails developers are less gung-ho about these
than about unit tests. (I'm one such developer at the moment, so I don't
really have a good answer for you yet.)
> Also, any good articles that could help me gain some perspective on
> the task i have ahead of me?
>
Other than the c2 wiki...hmmm. There's an interesting forum thread on
SimpleTest (PHP) that was very helpful to me...I'll see if I can dig up
the URL. Phlip, do I remember correctly that you've got some stuff
around the Web on the subject?
> my 2 cents worth.. i personally like the feel autotest gives.
> especially for TDD. but hey.. no experience. only tryouts and
> screencasts.
Autotest is just about essential for TDD. On Mac OS, I find it helpful
to integrate it with Growl (there are a number of ways to do that -- a
search should turn them up).
>
> Please DO let me know if I have taken away the right things from here
> *to get started* with writing tests for my existing app.
I think so.
> I hope this discussion goes further and provides insights for many
> more in my place.
>
> Thanks Fernando, Marnen, Phlip, Brandon and Hans :)
You're welcome! Good luck.
Fernando Perez
> especially for TDD. but hey.. no experience. only tryouts and
> screencasts.
to developing a real application that has a few more features than a
blog or a weather forecast parser, you will eventually turn autospec
off.
As you said, the best solution is to get started now!
Andrew Timberlake
>
> <Yikes!>
>
> ok that was some intense stuff. I dint understand a lot of stuff going
> around because, as established before, I dont have experience writing
> tests.
> But the following are the things Im taking out of this discussion.
>
> - jump in the dirt. Do TDD for your existing app by starting all over
> again if you have to. Thats the only way to go (?) / thats the best
> way to go to become a better programmer (?)
I don't think that this is a good idea at all.
Definitely start testing immediately but I wouldn't rewrite the app at all.
For every bug fix and change, make sure you write tests for the
functionality you are working on (in a TDD/BDD way - before changing
the code.)
Jim Wierich did an excellend talk at Scotland on Rails about Comment
Driven Development which, if I can summarise it accurately, is as
follows:
When changing/bug fixing existing code
1. Comment out the code in question
2. Write tests on how the code should work
3. Only uncomment the code that makes the test pass
Repeat 2 & 3 until you have the code completely covered in tests
4. Write a test for the new feature/bug fix (if not already fixed by now)
5. Write the new feature/bug fix
Done!
If you do this whenever you work on existing code, you'll grow the
test coverage of the app over time.
>
> - use fixtures when I have to test something that uses complex
> associations between models. especially when the alternative is to
> spend a lot of time and brain on mocking/stubbing them instead. (I
> have to mention I dont know the difference/similarities/functions of
> mocking and stubbing completely)
I still wouldn't use fixtures. I work with factories because then you
can put together the complex associations and data needed for just the
question at hand.
I have recently started creating a test library for very complex
applications that will build up a complex set of data via
TestData.load_data type calls. I do this so that I only load up the
data for those areas where the tests need them and not on every test.
>
> - i dont have any client documentation to show for these tests. theyre
> only for the sake of testing my code itself. So ill skip over RSpec
> and Cucumber (until i feel a need for them)
Cucumber is not just for client documentation but does end-to-end or
full-stack testing which is useful for getting
as-close-as-the-client-experience testing
> i am going to be adding features to the app in between too. anything I
> have to keep in mind while doing that?
> when do I start on integration and functional tests?
> Also, any good articles that could help me gain some perspective on
> the task i have ahead of me?
This I covered above. Start your testing with the new features
(Another reason not to re-write)
Include all the testing you want on a per-feature basis.
>
> my 2 cents worth.. i personally like the feel autotest gives.
> especially for TDD. but hey.. no experience. only tryouts and
> screencasts.
I have no problem with autotest running on large projects as I find
that when it starts running the full suite, I'm busy writing the next
test/spec. When working with a failing test, autotest only re-runs the
failing test anyway.
Andrew Timberlake
http://ramblingsonrails.com
http://www.linkedin.com/in/andrewtimberlake
"I have never let my schooling interfere with my education" - Mark Twain
Ram
Thanks for the responses to all my concerns. And I will take your
advice and give RSpec a go too. It definitely is more appealing to the
eyes at least on the test method name level (which is how much i know
of RSpec +/ test::unit :) )
> Huh?
I meant that once i get into writing tests, ill probably know what
tools i need if at all i need them.
@Fernando,
good point well made :) ... I have it installed anyway so Ill go along
with it until it starts getting in my way :)
@Andrew,
>I don't think that this is a good idea at all.
>Definitely start testing immediately but I wouldn't rewrite the app at all.
conveniently ignored writing tests for so long is because its
daunting. Rewriting the whole app and doing it the TDD way makes it
even more daunting! I must say though that I realize rewriting with
TDD is probably gonna do more good than covering my code with tests
now. It will make me revisit my app and clarify requirements and
functionality and help me understand my app itself a whole lot better
overall.
But for now, I think im gonna go with writing tests bitwise, covering
existing code. If i feel insecure about it at some point, maybe ill
rewrite with TDD to give the app a more solid foundation. Ive spent
months on this app and rewriting with TDD is just too much added time
investment.
The Comment Driven Development part is very close to my situation
right now. Its an approach that I probably wouldve taken naturally out
of necessity rather than in order to follow a technique. :) Thanks a
lot for mentioning that. I will look it up in detail.
>Cucumber is not just for client documentation but does end-to-end or full-stack testing which is useful for getting as-close-as-the-client-experience testing
tests? or integration and functional tests too for that matter?
Does one always start writing tests with unit tests and work through
functional, integration and then full-stack-tests? Isnt that like
going from backend to frontend? Would that be the right approach for
ANY kind of application? Even if its largely user centric and has got
very little magic going on in the models?
On Apr 18, 12:35 pm, Andrew Timberlake <and...@andrewtimberlake.com>
wrote:
Andrew Timberlake
>
> @Andrew,
>>Cucumber is not just for client documentation but does end-to-end or full-stack testing which is useful for getting as-close-as-the-client-experience testing
>
> That is actually a good point. But when do I start writing full stack
> tests? or integration and functional tests too for that matter?
>
> Does one always start writing tests with unit tests and work through
> functional, integration and then full-stack-tests? Isnt that like
> going from backend to frontend? Would that be the right approach for
> ANY kind of application? Even if its largely user centric and has got
> very little magic going on in the models?
>
Start writing integration tests immediately as the first step, then
work down to view, controller and then unit tests
You want to first cover the expectation and then drill down into the
implementation detail.
I found that when writing Cucumber scenarios on an existing project,
it clarified my thinking around how I already expected features to
work
If I was only able to write one kind of test, it would be integration
or full-stack tests.
The rest of the tests are great because they document how you want
various functionality to work at a technical level but the core of an
app is to get it used (and working) for a customer and that is what
the integration test covers
Andrew Timberlake
http://ramblingsonrails.com
Phlip
> I don't think that this is a good idea at all.
> Definitely start testing immediately but I wouldn't rewrite the app at all.
It ain't a rewrite. Imagine if you could put a bookmark at one end of all your
source. Above the bookmark is completely tested and below the bookmark is
incidentally tested or not at all.
If you open a new project folder and start TDDing each feature, passing each
test by copying code in, then what you are really doing is simply moving that
bookmark down thru all your source until it hits the end.
If opening a new folder squicks you out, then comment out _all_ the code, and
TDD it to uncomment it. Either way, the resulting velocity is much higher than
raw development.
> For every bug fix and change, make sure you write tests for the
> functionality you are working on (in a TDD/BDD way - before changing
> the code.)
That is the advice from /Working Effectively with Legacy Code/ by Mike Feathers.
Andrew Timberlake
I'm not advocating commenting out ALL the code, just the code around
the feature you're working with.
When you've got an app with thousands of hours work, a rewrite is just too huge.
What I'm suggesting is an iterative approach to eventually getting the
entire project covered based on the features you're working with at
any given time.