a client-side debugging UI for buster

[Martijn Faassen]

Hi there,

I've been struggling with using client-side debugging for buster. For
Firefox with Firebug, there's a kind of trick:

* use the -p option with "buster test" to run the tests with static paths

* this causes firebug to retain script urls to the scripts you want to
debug next time around (otherwise it keeps changing the URLs to defeat
potential caching issues)

* you can then set a break point in either the test code or the code under test

* you can then run the tests again with "buster test -p" and the break
point will be triggered

* during this process, it's often useful to filter out individual
tests on the command-line using the "buster test" using their names.

There are some related issues, such as the way buster handles errors.
Any error in the browser is captured by buster and reported to the
server. But in case of a
test that is error-ring for some reason, it's useful to see the stack
trace in Firebug instead (as you can click around).

For IE, it's harder to get the IE debugger to work. The "buster test"
command cleans up the frame in which the tests run after the tests are
done. IE
responds to this and does not retain the URLs of the scripts in the
frame, and those you want to debug. So unlike with FF, you cannot wait
to the end of the
test run to set break points. Instead, you can hack around this by
putting an 'alert()' in your tests to halt execution, during that time
use the IE debugger to set
breakpoints, and then press ok on the alert message in the browser.

It strikes me that we could make the whole debugging experience better
by offering a special debugging UI in buster. I think a web UI is
appropriate for client-side debugging, because the environment would
aim to be helpful to the person doing debugging on the client-side. A
command-line UI is great to verify quickly whether all tests pass, and
it's good for automated reporting and integration with other systems,
but less ideal to debug client-side tests (especially in the case of
errors).

What could such a UI offer?

* a way to get to the tests in "debugging mode". This means load up a
web page or a frame where the resources are loaded with static paths.
No tests are run yet. This way the developer can start the debugger
and set break points before any test has run. In addition, to support
systems which dynamically load .js resources, it would be good if
there was a way to prevent the loaded resources from being removed in
the end (but I haven't investigated the latter topic a lot).

* offer an overview of test cases and tests. offer a search interface
so you can filter on particular test cases. This way it becomes easy
to isolate a test. One can do this on the command-line as well but
that's a bit more cumbersome in the case of client-side debugging as
one has to switch to a different modality.

* the ability to start running one or more tests from the UI. (over
and over again)

* when a test is changed on the filesystem by the developer, it'd be
good if we could guarantee it being reloaded when the tests are
re-run. But since URL mangling is not what we want as that would
defeat the breakpoint story, that would require another
cache-defeating mechanism, so this might not be feasible. In that case
good integration with force-reload (and disable cache & reload)
facilities in the browser would be useful.

* when the tests run in this environment, a qunit like mechanism that
doesn't catch all errors but instead lets them bubble up to the
browser. this is useful when debugging errors (instead of failures),
as tools like firebug can help you introspect the stack trace and
such, and put in breakpoints. the test runner will in this case stop
running the first error it will encounter, because since it cannot
catch the error it cannot recover and run the next test. But if you
have your firebug console on you'll see something interesting. This
feature shouldn't be always on of course; it should be possible to run
the tests with catching and reporting instead.

It could be integrated into buster server, but since it has different
requirements (no slave attaching, for instance), it might be better to
make it a separate tool, buster debugserver (it'd be good think about
integration between the two, though). I also realize that 'buster
static' was attempting to tackle some of these use cases, but it
doesn't really help very much yet in practice, and 'static' is more a
mechanism to help with debugging rather than the goal, so perhaps also
not ideal naming.

If people are interested in such an effort, I'd be willing to brave
the node.js and buster coding world and help build it. Please let me
know what you think!

Regards,

Martijn

[Martijn Faassen]

Hey,

Thanks to sriley on IRC, here's an alternative way to trigger the IE 8 debugger:

* press F12 in IE

* go to the 'script' tab, press 'start debugging'

* then in the javascript code to place a breakpoint use the
"debugger;" statement

* this will get you into the IE tests in a way that you can set other
breakpoints.

I was confused as I had 'Microsoft Script Debugger' installed which
was triggered preferentially by the 'debugger;' statement, and this
one is older and has a worse UI. Uninstalling it stops that one from
popping up instead.

Regards,

Martijn