Future of Crosscheck

16 views
Skip to first unread message

Charles Lowell

unread,
May 14, 2009, 10:16:05 PM5/14/09
to cross...@googlegroups.com
All,

There has been a distinct uptick of interest in Crosscheck over the past
few weeks. Mostly, people have been emailing the frontside directly, but
there has also been a few messages on the list as well. While I still
believe very much in crosscheck the idea, I've had to turn people away
because the fact is that in its current state the chances are it's not
going to do what they need.

What this has made clear to me is that the time has come to either move
things forward, or take down the sign and bury Crosscheck for good.

If it's going to continue, then it needs to be informed by a detailed
vision of what it should strive to become, and how it's going to get
there. To that end, we spent some time talking on the IRC channel
yesterday about what that vision might be, and here are some of the
ideas that came out of the discussion:

We still believe that there is a need for a *headless* environment for
functional testing and unit testing of javascript in which the tests are
written *in javascript*. As as it stands however, Crosscheck has not yet
realized this niche for several reasons.

1) There has been too much focus on supporting *all* browser
environments which dramatically slowed the implementation of other
features which people needed.

While this might have been the right thing to concentrate on in 2005, we
are now in 2009 and most people use very mature platforms like jquery,
mootools, etc... which, except for a few minor instances, pretty much
sweep cross browser issues under the rug. Instead, crosscheck should
focus first on achieving a complete vertical stack before fanning out
horizontally to verify a host of potentially obscure browser
implementations. When I say full stack, I mean the script runner, build
integration, IDE integration, and a one fully functioning virtual
browser environment that is "close enough" to the browsers people use today.

This is not to say that the possibility of cross browser testing should
be abandoned, only that it should be a second priority to getting that
full stack which people can extend for their needs.

2) The code has historically been opaque to the majority of the users.

Crosscheck executes javascripts, but is itself implemented almost
entirely in Java. Because of the way rhino (circa 1.5) works, this has
necessarily entailed a large and complex code base, especially for the
simulated browser implementation. Multiply that by all the environments
supported, and you have a hot mess that no one person can fully
comprehend. Not only that, but only a fraction of potential crosscheck
users can or want to write Java, which presents a serious impediment to
community contributions. The obvious answer is to implement crosscheck
in the language which it targets and the language for which most people
use it (I'm talking about javascript of course). With the recent
releases of rhino this is a very possible and very preferable way forward.

3) Lack of an active Community.

The is perhaps our biggest failing, and if crosscheck is to procede, the
one that needs to be addressed most urgently. It's not worth dwelling
on, but for whatever reason we never managed to consistently convert
interest into involvement. I believe that the interest is still there,
and will probably increase, but if there is no clear pathway from
interest to action, then it will never go anywhere.

Also, we never think there should be more outreach and cooperation with
other projects which are in this space such as jspec, jsunit, and http
unit. There is a lot of code to be shared there.

These are the major points that we came up with, but it doesn't mean
that there aren't more, or that they all need to be addressed at once.
If something does start back up, it will start, as it always does: in a
small way.


But these are just the thoughts of a few people. I'm more interested in
what do you think.

What are the problems that you want to see solved by crosscheck that
aren't solved by tools you use today?

What are the things that have frustrated you in your efforts to use
crosscheck?

How might they be addressed and/or temporarily mitigated?

What do you think crosscheck gets right (if anything)?


I think the opportunity is there for a very useful, very successful open
source project, but I as well as others are not going to find the time
to work on it unless there is a clear way ahead. Are you interested in
being a part of it?

cheers,
Charles


Jeff Hemminger

unread,
May 19, 2009, 10:30:17 AM5/19/09
to cross...@googlegroups.com
I'm nearing the end of  a client engagement; a major webapp rewrite using a Java backend delivering restful services, and a sophisticated client written in ExtJs.
What I need from a headless client testing framework are two things:

1) Support a full-featured DOM
2) Allow me to debug my JavaScript in my IDE.

I honestly don't care about the browser differences. Many of the javascript frameworks mitigate those pretty well. The volume of javascript code I'm writing now is where real troubles lie- I need a way to write effective unit tests, and tests those within my ide.

I realize this may be a very different goal than the original crosscheck's, but these are the problems that drove me to sign up for the crosscheck mailing list, and this is what I'm still looking for.

thanks,
Jeff

John

unread,
May 20, 2009, 12:56:51 PM5/20/09
to crosscheck
I agree whole mentally. I would say, considering the dearth of full-
featured DOM testing environments, the #1 area that crosscheck could
contribute in terms of a headless unit testing environment is a
complete DOM. As it is, I have to use a ridiculously complex
assortment of tools to accomplish headless javascript testing. This
includes using a virtual frame buffer (Xvfb) in conjunction with
Selenium RC to automate a browser to run JsUnit tests. I know that
seems laughable, but I'm serious. It is what we need, and it works.
But it is a maintenance headache. So, I'm even more serious, as well
as hopeful, when I say crosscheck is a long way toward making my Rube
Goldberg testing harness obsolete. It only lacks a full-featured
DOM.

Regarding Browser differences, I agree that that issue has become
decreasingly important. IE7(IE8?) and FF3 is all we much care about
these days. Safari is showing a bit of ascendancy, but not really
something we need to take too seriously yet.

On May 19, 7:30 am, Jeff Hemminger <jeff.hemmin...@gmail.com> wrote:
> I'm nearing the end of  a client engagement; a major webapp rewrite using a
> Java backend delivering restful services, and a sophisticated client written
> in ExtJs.
> What I need from a headless client testing framework are two things:
>
> 1) Support a full-featured DOM
> 2) Allow me to debug my JavaScript in my IDE.
>
> I honestly don't care about the browser differences. Many of the javascript
> frameworks mitigate those pretty well. The volume of javascript code I'm
> writing now is where real troubles lie- I need a way to write effective unit
> tests, and tests those within my ide.
> I realize this may be a very different goal than the original crosscheck's,
> but these are the problems that drove me to sign up for the crosscheck
> mailing list, and this is what I'm still looking for.
>
> thanks,
> Jeff
>

Darrin Edelman

unread,
May 20, 2009, 5:29:47 PM5/20/09
to cross...@googlegroups.com
All -

I want to start by saying that from the day I first ran across crosscheck I thought it had some great vision and ideas.   Thanks very much for putting it together and I'm thrilled that it's seeing an uptick in momentum.  Now... back to the future... (couldn't help myself).

I have to agree whole heartedly with Jeff's points - I want the ability to use html complete with javascript include statements and the whole lot the same way I would for my browser - either by passing in a dynamically generated page via an API, by a filename (or maybe even a reference URL).   This would make it FAR easier to write tests and use crosscheck... We need to allow people to focus almost entirely on validation rather than constructing a parallel model of the conditions on our websites... This would allow us to generate a model of the DOM statically and then write tests to show how we manipulate the DOM with much less effort... As it stands we need to write alot of code just to get to the point where we have a model constructed to test - which means I'm doing much less testing that I probably should or I'm finding other ways to do things with less Javascript.  If we can minimize the amount of additional work you have to do to create and maintain tests I think we've got something of real value.

So - I want to add this one in:
1.5) Support unit tests ala xUnit directly in my IDE and as part of my maven/ant builds

I had this working (in a cludgy way) about 9 months ago... and it was sort of sweet if fragile (and based upon other people's work). No doubt improvements could be made.  I had something that would run the tests as a suite of Junit tests... which is nice because it just works with everything that Junit works with.  In theory this would have allowed us to do Test First Development and Test Driven Design - which I've not really been able to do easily in any existing environment. Crosscheck as it stands helps to move us in that direction - but that it is still to inconvenient to be practical.  It should be plug and play... e.g. provide a jar that can provide a superclass (for example) that can be referenced by a single test... it scans one or more directories and builds out one or more test suites and runs them providing output in Junit form... add the ability to read html and provide for a fully-featured DOM and I think we'd have something very nice.


To Jeff's point #2  'debug my JavaScript in my IDE' (which is Intellij by the way)... that would be pretty darn awesome perhaps even the holy grail.  BUT I think we would get a ton of use out of having 1.5 complete above - and I imagine we can get there along the way. In my organization we've moved a lot of code out of javascript because it is simply easier to test and maintain in Java... and we were driven to finding ways to get similar interaction benefits with less Javascript.

As for the notion of architectural revamps and rewrites in Java... you guys are better plugged into the need for this but it wouldn't be my driving concern as an end user - I just don't care.  As a developer a well defined set of Java classes should be fine and certainly easier to find errors in and debug than javascript (remember why crosscheck exists).  I don't know enough to judge but it seems like a rewrite in Javascript may not be the fastest way to address the biggest needs.  I imagine we've already polled the community to find the largest cross-section of people who would definitely contribute and what languages they are conversant in?


As for browser compatibility... I don't think we need broad support to start rejuvinating cross-check.  Why not focus on Firefox 3 or IE 7... and provide a great set of abstractions to ensure the core is browser independent (dare I say complete with tests for the core to ensure that we test the way it can flex).  Maybe kick this off with a poll of what the user community uses and wants to see most from a browswer support perspective?

Just some thoughts...

-Darrin

Antony Lees

unread,
May 20, 2009, 4:27:24 PM5/20/09
to cross...@googlegroups.com
I agree with the posts so far too.  I spent hours looking into the various unit testing frameworks and there are basically only 3 headless javascript frameworks that I can find - Crosscheck, Rhinounit and DOH.  I had already decided that Crosscheck was the closest to what I wanted which is:

1) Headless javascript unit testing
2) xUnit style unit tests I can write in java
3) A full DOM
4) to a lesser extent - cross js engine testing
5) I hadn't considered it before, but javascript debugging would be pretty cool
6) Javascript mocking

What most frameworks lack, for me, is a java interface as I'm not a proficient javascripter.  So I started writing my own using Java 6's new scripting engine support.   However, Crosscheck is already closer to what I envisage a full-featured framework to be.  I'd still like the cross-browser support, but I agree it's less important than getting it up and running.  Other browsers can always be rolled out afterwards

If you decide to continue with Crosscheck, I would definitely be interested in providing a JUnit-style, annotation-driven java interface for it, as I've already started one

Antony

Charles Lowell

unread,
May 24, 2009, 4:46:21 PM5/24/09
to cross...@googlegroups.com
Ok, it seems like people are relatively in agreement about the features that are needed, which are, to summarize 

1) A fully functional DOM implementation, capable of handling real world DHTML applications.
2) Simple Test Running 
3) IDE integration
4) Unit Test Debugging
5) Java Integration
6) Cross browser unit testing


Please add anything to that list which I missed. I'm going to attempt to flesh out what each point means, and ask for comments about what is missing from each one.

1) A full DOM implementation
 
There's a lot wrapped up in that bullet point, and so I'd like to get a brief laundry list about what exactly this means.  Leaving crossbrowser stuff out of the mix for the moment, there are still the following things to consider/prioritize:

--DOM Core.. but which versions are a priority? (0,1,2,3)? 
--DOM events(load, mutation, etc..) 
--HTML Events(mouse, key, etc...)
--Different DOM types HTML4/HTML5/XHTML1.0, XHTML 1.1
--HTML extensions cssText, innerHTML
--Script injection (inline and loaded from a URL).
--Stylesheet parsing, rule application
--IFrames, Frames, <objects>, i.e. nested DOMs
--XmlHttpRequest, etc...
--XPath

These are the major features that I can think of, but obviously this is not a complete list.

I'm thinking this should be treated almost as a completely separate subproject so that it is not coupled either to the assertion syntax, the test runner or anything else. It seems like a handy standalone tool to begin with, and furthermore, if we concentrate on this, then people can pick it up, experiment with it, and start using it however they want without being bound to any other constraints. It could, for example, be used to develop a totally separate js unit testing framework, or even a java web browser.

What's missing here? What should the priorities be?

2) Simple Test Running

I agree with the point that running tests with crosscheck should be as easy as running an executable against a directory. The area that needs some thought and consensus is how to group, organize and annotate your tests. Since crosscheck first started out, we've seen lots of different unit testing styles emerge. I'm partial to the rspec BDD style myself, so I'd like to see this part be pluggable. In other words, if you want to use xUnit style assertions, then you can do that. If you want to use BDD or some other yet to be determined style, then that should be possible too. 

But in their essence, runners are just ways to collect and then execute some scripts inside the context of a DOM.

What kind of syntax would you like to see the crosscheck runner(s) support that will make it as simple as possible to organize your tests? What testing style would you like to see first?

3) IDE Integration

The primary IDEs that I use are IntelliJ and TextMate, but I have zero experience integrating with an IDE, so my hope would be that if we have smart answers to (1) and (2), then this would be a natural contribution from the community. We'll have to see though.

It seems to me that something we would need to do here is have some interface for running focused unit tests, and some way to pass crosscheck the source file and have it determine what test the cursor is currently in. How the tests are structured will definitely effect this.

Does anybody have experience with this? What should the first priority be?

4) Debugging of unit tests.

I actually don't see this as being all that difficult. The rhino debugger interface is pretty sweet, and gives you access to everything you could possibly need. The trickiest part would probably be handling the stepping in and out of crosscheck code so that in most cases, it would be hidden from the user. Not only in control flow, but also in stack traces. We would need to take a survey of the different IDEs (including a commandline), and what interface they need to tie in to their debugging mechanisms.

5) Java Integration.

Also not that difficult if there is a high quality (1) and (2). For the java users, what would this look like?

6) Cross browser checking

This boils down to changing the behavior of the DOM implementation at runtime. If we're conscientious of how (1) is designed, we should be able to get plenty of re-use of components. Is there really anything beyond just W3C and IE?

cheers,
Charles

Antony Lees

unread,
May 24, 2009, 5:13:34 PM5/24/09
to cross...@googlegroups.com
Regarding points 2, I have a good idea of what I'd like it to look like.  An annotation driven xUnit-style testing layer that wraps JUnit would be good.  I already started thinking about this before, and imagine something like:
- re-use of the existing @RunsWith annotation eg
  @RunsWith(Crosscheck.class)
- method annotation to define a crosscheck test eg
   @CrossCheckTest
- class annotation to specify the javascript engine eg
  @RunWithEngine(Engine.RHINO) or @RunWithEngine(Engine.ALL)
- class annotation to specify the source file(s) or the source directly eg
  @ScriptSource(source="test.js") or @ScriptSource(source="some javascript")
- possibly another (optional) method annotation to define the function to be called eg
  @ScriptFunction(function="test")

Then wrap the JUnit assertion methods to call the javascript.  This would make a test look something like:

@RunWith(Crosscheck.class)
@RunWithEngine(Engine.RHINO)
@ScriptSource(source="test.js")
public class Testing {

    @CrosscheckTest
    @ScriptFunction(function="test")
    public void test() {
        Assert.assertEquals("hello", );
    }

Antony

Antony Lees

unread,
May 24, 2009, 5:17:36 PM5/24/09
to cross...@googlegroups.com
Then regarding point 5 - java 6 already provides java integration for the Rhino engine using the ScriptEngine class.  My original idea was to create new engine integration for other engines by creating new script engines

As I thought Crosshcheck might be dead, I already started this on Sourceforge
http://sourceforge.net/projects/jsengineunit4j/
I could probably, instead, rework this as a wrapper to the new Crosscheck?

Antony

Antony Lees

unread,
Jul 1, 2009, 6:27:43 PM7/1/09
to cross...@googlegroups.com
Hi all

It's all gone quiet here. Is there any decision on where to go with crosscheck?

Antony

Charles Lowell

unread,
Jul 3, 2009, 2:09:07 AM7/3/09
to cross...@googlegroups.com
Antony,

I've been slowly plodding at getting the version described by the folks
on this list out there. I only work on it tiny bursts.

I've been pushing all this incremental work to github.

you can see it at:

http://github.com/cowboyd/crosscheck/tree/master

Right now, the only thing that is implemented is dom level 1. There is
no assertion framework, no suite builder framework.... it's just a bare
DOM. To test it, I have been using the junit 4.5 RunWith and a bootstrap
test runner. My plan is to defer the actual testing part until there is
a better functioning dom, which means

I'm not going to focus on the runner until it's running jquery (what I'm
currently using)

Antony Lees

unread,
Jul 8, 2009, 9:22:27 AM7/8/09
to cross...@googlegroups.com
Looking good!

You don't need a hand then?

Antony

Charles Lowell

unread,
Jul 8, 2009, 10:28:57 AM7/8/09
to cross...@googlegroups.com
I totally need a hand :) I definitely didn't want to give off that
impression.

Maybe we can try and go through it at some point and see how we might be
able to make it useful for you?

cheers,
Charles

Antony Lees

unread,
Jul 9, 2009, 8:57:18 PM7/9/09
to cross...@googlegroups.com
Yep, sounds good. I'm not a javascript expert, but I am pretty good
with Java, so if you need a hand, just let know

Antony

Antony Lees

unread,
Sep 18, 2009, 9:42:30 AM9/18/09
to cross...@googlegroups.com
Hi again

How's the rewrite going?

Antony

ali65

unread,
Sep 30, 2009, 1:10:41 PM9/30/09
to crosscheck
Have you guys noticed that Gnome 3.0 has put JavaScript to the center?
I wonder what they are using if not crosscheck.
If crosscheck wins over those developers, I guess "Lack of an active
Community" problem is solved.


On Sep 18, 8:42 am, Antony Lees <antonyl...@gmail.com> wrote:
> Hi again
>
> How's the rewrite going?
>
> Antony
>
> On Fri, Jul 10, 2009 at 1:57 AM, Antony Lees <antonyl...@gmail.com> wrote:
> > Yep, sounds good.  I'm not a javascript expert, but I am pretty good
> > with Java, so if you need a hand, just let know
>
> > Antony
>
> > On Wed, Jul 8, 2009 at 3:28 PM, Charles Lowell<cowb...@thefrontside.net>
> > wrote:
>
> > > I totally need a hand :) I definitely didn't want to give off that
> > > impression.
>
> > > Maybe we can try and go through it at some point and see how we might be
> > > able to make it useful for you?
>
> > > cheers,
> > > Charles
>
> > > Antony Lees wrote:
> > >> Looking good!
>
> > >> You don't need a hand then?
>
> > >> Antony
>
> > >> On Fri, Jul 3, 2009 at 7:09 AM, Charles Lowell<cowb...@thefrontside.net>
> > wrote:
>
> > >>> Antony,
>
> > >>> I've been slowly plodding at getting the version described by the folks
> > >>> on this list out there. I only work on it tiny bursts.
>
> > >>> I've been pushing all this incremental work to github.
>
> > >>> you can see it at:
>
> > >>>http://github.com/cowboyd/crosscheck/tree/master
>
> > >>> Right now, the only thing that is implemented is dom level 1. There is
> > >>> no assertion framework, no suite builder framework.... it's just a bare
> > >>> DOM. To test it, I have been using the junit 4.5 RunWith and a
> > bootstrap
> > >>> test runner. My plan is to defer the actual testing part until there is
> > >>> a better functioning dom, which means
>
> > >>> I'm not going to focus on the runner until it's running jquery (what
> > I'm
> > >>> currently using)
>
> > >>> Antony Lees wrote:
>
> > >>>> Hi all
>
> > >>>> It's all gone quiet here.  Is there any decision on where to go with
> > crosscheck?
>
> > >>>> Antony
>
> > >>>> On Sun, May 24, 2009 at 10:17 PM, Antony Lees<antonyl...@gmail.com>
> > wrote:
>
> > >>>>> Then regarding point 5 - java 6 already provides java integration for
> > the
> > >>>>> Rhino engine using the ScriptEngine class.  My original idea was to
> > create
> > >>>>> new engine integration for other engines by creating new script
> > engines
>
> > >>>>> As I thought Crosshcheck might be dead, I already started this on
> > >>>>> Sourceforge
> > >>>>>http://sourceforge.net/projects/jsengineunit4j/
> > >>>>> I could probably, instead, rework this as a wrapper to the new
> > Crosscheck?
>
> > >>>>> Antony
>
> > >>>>> On Sun, May 24, 2009 at 9:46 PM, Charles Lowell <
> > cowb...@thefrontside.net>
> > jeff.hemmin...@gmail.com>
> > >>>>>> wrote:
>
> > >>>>>>> I'm nearing the end of  a client engagement; a major webapp rewrite
> > using
> > >>>>>>> a Java backend delivering restful services, and a sophisticated
> > client
> > >>>>>>> written in ExtJs.
> > >>>>>>> What I need from a headless client testing framework are two
> > things:
>
> > >>>>>>> 1) Support a full-featured DOM
> > >>>>>>> 2) Allow me to debug my JavaScript in my IDE.
>
> > >>>>>>> I honestly don't care about the browser differences. Many of the
> > >>>>>>> javascript frameworks mitigate those pretty well. The volume of
> > javascript
>
> ...
>
> read more »

Charles Lowell

unread,
Oct 1, 2009, 12:39:51 AM10/1/09
to cross...@googlegroups.com
Honestly, it has not progressed much beyond where it was this summer. It's become clear that between all my other obligations, I don't really have the time to make this a priority.

I wish I had a better answer, but right now it's just not on my list of active projects.

Charles Lowell

unread,
Oct 1, 2009, 12:46:53 AM10/1/09
to cross...@googlegroups.com
Crosscheck as it stands is a dom implementation written in javascript
with a test runner and operating on the Rhino interpreter (Java). If
the gnome folks were wanting to unit test their app code, they'd need
a test runner that embedded a javascript interpreter that could be
accessed from either C or Mono (probably SpiderMonkey), no?

cheers,
Charles

visola

unread,
Oct 26, 2009, 2:05:23 PM10/26/09
to crosscheck
Hi all. I've been thinking about Javascript unit testing for a few
weeks now and started to gather a few tools and projects that could
help to get a whole browser environment estabilished. I use Eclipse
for java development and Aptana for web development, which means that
all my thoughts were aligned with Eclipse/OSGI plugins/bundles.

So I finally stumbled with Crosscheck (didn't have time to test it
yet). The topics discussed here are just the things that have been on
my mind lately and the research I've been working on may help to
contribute to this project.

To start, I saw that no one mentioned the great project by John Resig
(JQuery author): http://www.envjs.com/
It's seems to take a more specific path, creating the whole
environment using Javascript only, if it is not what you guys have
been looking for, it should be pretty close. A lot of thing could be
used in javascript or maybe rewritten in Java, if necessary, but
surely, there is a lot to learn and colaborate with that project to.

And about the DOM implementation, there is already a started (and very
mature) project named Lobo (http://lobobrowser.org/) that is a Java
Browser, with DOM, CSS and rendering toolkit implemented, that surely
can be useful.

I'm here to know what are the odds of making all these communities to
colaborate with each other to create a pure Java implementation of a
Javascript unit testing tool that could have multiple plugin
implementations for each one's favorite IDE? And what do you guys
think about uniting all these to make this project grow faster and
stronger?

Best regards.
> ...
>
> mais »

Darrin Edelman

unread,
Oct 28, 2009, 10:14:18 PM10/28/09
to cross...@googlegroups.com
Amen -

I'd love to see it happen.  More/less this is the type of thing a number of us have been talking about and in fact why we're all interested in Cross-check to begin with.  We need someone with time to drive this type of vision forward and shake things up.

-Darrin
Reply all
Reply to author
Forward
0 new messages