I stumbled on a previous thread in this group:
http://groups-beta.google.com/group/netscape.public.mozilla.dom/brows...
which claims that it is impossible to create a new HTML DOM document
from a Firefox script without displaying it in a new window. This means
that HTML screenscraping using XMLHttpRequest is not possible.

In a fit of pique I ranted about this on my blog
(http://www.allpeers.com/blog/?p=136). I was trying to be funny, but
the issue is serious. I'm probably missing something, but can someone
explain to me why the appropriate interfaces are not exposed to
scripters using XPCOM?

Cheers,
Matt

Because HTML content model construction is so tied to having a window.  As one
simple example, it assumes the existence of a window it can reload to handle
charset autodetection and <meta> charset declarations that are not in the first
chunk of data we get from the document.  For XML this is not an issue, of
course, since the problem simply cannot arise.

There are other issues; for example the HTML parser needs the window to find out
whether scripts and frames are enabled (for parsing <noscript> and <noframes>
tags).  This is not an issue in XML, again, because the _parsing_ doesn't depend
on anything.  In HTML it does.  And since frames and scripts can be
enabled/disabled on a per-window basis, this is a bit of a problem.

Some work has been done to make the parsing not require a window, but a lot more
needs to be done, especially if people want it to work like it would with a
window around.

-Boris

Boris,

Many thanks for the reply. I understand the issue much better now. Two
more questions:

1) You mention that "a lot more needs to be done." Is there an active
effort to break the remaining dependence of the HTML parser on the
existence of a window?
2) Wouldn't it be a viable workaround, in the meantime, to associate an
HTML document retrieved with XMLHttpRequest with an invisible window?

Matt

Not very active right now, since Gecko is in 1.8 freeze, more or less.  There
may be more work on it in the 1.9 cycle.

That would execute scripts in the document in question, load stylesheets, etc,
etc.  That seems undesirable (especially executing scripts).

-Boris

Ok. Personally I think this functionality is important enough to merit
a short-term workaround involving an invisible window with scripts
disabled. I can't believe that this would be a huge programming effort.
At the same time, I can see how this approach could meet with
resistance since it's obviously a hack and might have other unintended
side effects.

Is there a Bugzilla report related to this that you know of? I had a
look around but couldn't find anything.

Cheers,
Matt

Patches accepted....

Not that I know of, no.

-Boris

Fair enough, I'll take a crack at it.

Cheers,
Matt