Mechanize for Node.JS?

[AJ ONeal]

Has anyone written a browser in Node yet?

Is there some npm-installable goodness that will handle session / cookie data automatically?

AJ ONeal

[mscdex]

Define 'browser.' Do you mean something like a lynx/links clone or
something non-visual that simply uses jsdom and such?

[James Carr]

I think he means the latter. I would love to see something like
HTMLUnit (a headless browser that has great javascript support).

Thanks,
James

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

[AJ ONeal]

Particularly, I meant like Mechanize - the Perl/Python/Ruby/Insert-language-here module for a headless browser.

There's an API I'd like to use, but it has a cookie-based session rather than an API key.
... I suppose it's not as much an API as simply a well designed site that has the misfortune
of lacking a slightly simpler authentication mechanism... it also happens to be dreadfully slow.


But on the idea of a browser: if you had bindings for webkit for Node I suppose you could write a browser "fairly easily".
That would be very interesting.

AJ ONeal

[AJ ONeal]

I chanced upon a "test framework" which, as it happens, appears to be the Mechanize of Node.JS:

http://zombie.labnotes.org/

var zombie = require("zombie");
var assert = require("assert");

// Load the page from localhost
zombie.visit("http://localhost:3000/", function (err, browser, status) {

  // Fill email, password and submit form
  browser.
    fill("email", "zom...@underworld.dead").
    fill("password", "eat-the-living").
    pressButton("Sign Me Up!", function(err, browser, status) {

      // Form submitted, new page loaded.
      assert.equal(browser.text("title"), "Welcome To Brains Depot");

    })

});

AJ ONeal

[Douglas Drouillard]

Zombie.js is meant for testing sites for which you have access to the source. The parser is pretty fragile, and will fail even for mainstream sites like google/yahoo/etc. It is a testing tool, not a scraping library. 

[Josh Adell]

You don't need the source of the site you're testing with Zombie, that
would defeat the whole purpose. The parser it uses (html5parser and
jsdom) is a little unstable. That said, it's getting better all the
time. Depending on what you're trying to do, I would recommend it as
a good general purpose headless browser. If you find a site it
doesn't work on, create a bug report; that's the only way things will
get better.

[Douglas Drouillard]

Really? It would defeat its purpose as a web scraper, not as a testing tool. 

Not saying it won't get better, but referring to Zombie as the Mechanize of Node.js at this time is not correct.

https://groups.google.com/d/msg/zombie-js/DmDQluYhiaY/-MdweipRHnsJ

From Asaaf

One. Zombie was conceived and developed for testing Web applications. It's great that you can use it for other purposes, kick ass that it's good for more than one thing. But all the design decisions are biased towards testing. Reason is ...

Two. The people who contribute their time and energy to improve Zombie are passionate about testing. Naturally, we invest our time and attention in making Zombie the best test tool possible. As is the case with open source, that which is interesting to contributors gets fixed. So ...

Three. Until you or anyone else starts investing in fixing issues, adding features, answering questions, writing documentation, etc that relate to Web automation, the scope will remain limited to what existing contributors are interested in contributing to. That is why ...

Four. Zombie does not advertise itself as a tool for Web automation. I'd hate for it to promise on something it can't deliver. In the meanwhile ...

[Josh Adell]

So in order for a QA tester to test an app using Zombie, they need
access to the source code of that app? No, what they need is a host
that is serving the app at which they can point Zombie. I should be
able to automate end-user testing of a site without having any
knowledge of the underlying technology used to generate that site.

Is Zombie testing that the application generates valid HTML, or that
it works from an end-user interaction standpoint? If it's the first,
Assaf and the rest should just give up; there are already HTML
validators out there. But if it's the second, then Zombie will start
moving towards being a general purpose browser out of necessity. Or
it will be eclipsed by something that is a general purpose browser.

Right now, Zombie is the best thing out there for what it does. And
the more people use it, the better it will get, even if they're not
using it for its "intended" purpose.