autotest

[Bill Burcham]

Do folks developing Calagator use autotest? If not, what's the preferred way to run tests, just "rake spec"?

[Igal Koshevoy]

Bill Burcham wrote:
> Do folks developing Calagator use autotest? If not, what's the
> preferred way to run tests, just "rake spec"?

I frequently run "./script/spec spec/..." to execute the individual spec
I'm working on, then I run "rake spec" when I'm done and want to make
sure that the whole suite still works, and periodically I run "rake
spec:rcov" to check the coverage report.

I don't get autotest. It seems to simply runs the equivalent of "rake
spec" over and over again without pause, which completely swamps a CPU
core and abuses the disk. When you make a change to a file, rather than
manually running a spec and immediately confirming that it works, you
have to instead bounce over to the terminal running autotest and then
watch and wait until it finally gets around to running the spec that you
actually care about to see if it works before it scrolls off the screen,
or somehow wait for however long you think is right (?!) for the
non-presence of a warning message to assume your code runs. This seems
like a complete waste of time and resources.

If someone can explain the value of autotest, I'd appreciate it.

-igal

[Ben Kerney]

I try to use it most of the time. I can't speak to disk abuse, but I'm
pretty sure it is more intelligent than simply running 'rake spec.' On
startup, it will run all your tests, but from then on only runs the
tests that apply to changes you've made -- not the whole suite
continuously. From the RDoc (http://zentest.rubyforge.org/ZenTest/):

Autotest continuously scans the files in your project for changes
and runs
the appropriate tests. Test failures are run until they have all
passed. Then
the full test suite is run to ensure that nothing else was
inadvertantly broken.

If you want Autotest to start over from the top, hit ^C once. If you
want
Autotest to quit, hit ^C twice.

As for jumping to the terminal, AutoTest has Growl integration, so a
few seconds after you hit save you can get a little red or green HUD
telling you the state of your suite. Igor, I don't know if it works
with Mumble, but I heard that the API is the same for each, so maybe?

- Ben

[Bryan Stearns]

I *do* use autotest, pretty much all the time while developing anything
in Rails: I arrange my windows to always be able to see the bottom of
the autotest terminal window. It saves me time when it shows me that
I've broken an unrelated test, and having it run all the time saves me
from having to remember to manually run it. I certainly don't worry
about "swamping a CPU core" or "abusing the disk" while I'm developing:
I don't notice any performance impact on my slower development box, and
I'm often playing music on it too.

My two biggest problems with autotest is that it doesn't rerun enough
tests sometimes (eg, when I change a partial or something in lib/ - I
know I can help its guesses, but haven't done so often enough), and that
when a test raises an exception, the emitted stacktrace swamps the
output (something smarter could elide everything below the test, for
instance; so far, though, I've avoided the temptation to go off and
write my own client for its hooks).

...Bryan

[Bill Burcham]

Looking at these docs: 

http://zentest.rubyforge.org/ZenTest/

Autotest is purported to not just blindly cycle through all specs repeatedly. Instead:

As soon as you save a file, autotest will run the corresponding dependent tests.

That being said, I have not used autotest myself but just wondered what the norm was on this project.

Oh and while we're talking specs. I notice that story support was added (to rspec) fairly recently (more at evangelist). It's not really documented AFAIK. But stories are the BDD answer to Rails Integration tests. 

I don't see stories for Calagator. What do folks think about stories? Has anybody got experience w/ the new story support in rspec?

[Sam Livingston-Gray]

For my projects, I usually just run an individual _test.rb file in
TextMate while I'm working on it, then run all tests to check for
regression. Autotest does try to be smart about selectively running
certain tests, and it stays active so that it doesn't have to
constantly reload Rails, which lets it give you rapid feedback.
However, its implementation does drain resources by continuously
polling the filesystem to find out which files have changed (so it can
figure out which tests it needs to rerun).

For you Mac folks, there is a Leopard-only alternative, though I
haven't actually gotten it to work on my machine.
http://rails.aizatto.com/2007/11/28/taming-the-autotest-beast-with-fsevents/

-Sam

[Igal Koshevoy]

Thanks all for the descriptions of how you're using autotest. It sounds
a lot more useful now.

I've let autotest run for 10 minutes, and it completely, continuously
swamps one of the CPU cores. I believe the problem is that I'm using svn
and hg, which means that I have ~6000 total files in my Calagator
checkout directory. Using strace to watch the autotest process, I can
see it poking at every single one of those to run stat to check the
timestamps. This is a lousy way to do things. Unfortunately, Ryan Davis
has refused to support efficient kernel-level filesystem notification
systems, such as inotify.

Autotest apparently runs much more reasonably when it has fewer files to
monitor. Ryan claims it's possible to tell autotest to ignore stuff
(e.g. .hg, .svn, .git), but my Google-fu is weak and I absolutely can't
figure out how to do this, even after reading the rdoc and scanning
through autotest's code.

Ideas?

-igal

[M. Edward (Ed) Borasky]

How many layers does your directory tree have? If it's relatively
shallow, you could try something like

list_of_files = `ls -lt`

This gives you the files sorted by most recently modified first. You
can then iterate through the list and stop when you hit a file that
was modified before the last time you checked. The date parsing is a
bit tricky, so you might want to just parse out the file names and do
the actual time stamp checking in Ruby.

I suppose there's also a "pure Ruby" (aka "will run on Windows") way
to do this, but I don't know what it is ... maybe something with
"Find" and "sort"?? Or you could look at the code to "ls" and see how
they do it.

[Ben Kerney]

Just looked at the link Sam posted. FSEvents looks cool, but the
discussion about why autotest's CPU usage is so high was more helpful.
I honestly have been turning a blind eye to what it must be doing to
poll those files (just get the job done, I don't care how -- I want my
test results, dammit!)

My ~/.autotest looks like this now:

require 'autotest/redgreen'

Autotest.add_hook :run do |autotest|
autotest.add_exception(/^\.\/vendor/)
autotest.add_exception(/\.svn/)
autotest.add_exception(/\.git/)
end

Autotest is now sitting at 2.5% CPU (down from 40-50%, which I hadn't
even been aware of.)

[Igal Koshevoy]

Ben,

Awesome. That configuration file, plus an extra line for ignoring .hg
directories, completely solves my problem. The autotest process now
consumes just a few CPU cycles and picks up changes promptly because
it's only scanning the relevant files.

Why the heck aren't these exclusions in there by default? Why isn't the
documentation clearer about this? Why did Ryan decide to call these
"exceptions" -- a rather common term with a different meaning -- instead
of something like "exclude" or "ignore"? Ugh.

Thank you for the help, you've helped make this tool usable for me.

-igal

PS: I've committed the .autotest file to the application's Rails root.
Running autotest from there will load the contents of the file so
everyone else can take advantage of these exclusions.