Embedding Gecko/XULRunner

[Peter]

I am developing an application that needs to embed mozilla's browsing
engine and control this engine programatically to navigate through
webpages and scrape content. The alterantive is to use libcurl, but it
seems too primitive for the rich and dynamic content found on the web
(especially that the final content on a webpage is often determined
after applying javascript stuff and css).

So far, I ve build the XULRunner SDK from source on Ubuntu (10.10
server 64bit). I am able to register the XRE (or GRE or whatever u
call it these days) so that example apps like TestGtkEmbed launch
properly.

However, the documentation on embedding is a bit fuzzy and all over
the place. I am not even completely sure with the terminology as it
keeps changing and is mixed up in the docs.

Question0: How is embedding Gecko different from embedding XULRunner?
Which of the two have I actually done above?

Question1a: My understanding is that I can build upon GtkEmbedMoz
(provides a gtk widget for the browser content pane - I think) and
TestGtkEmbed (provides the actual window in which GtkEmbedMoz is
displayed - I think) instead of working directly with the embedding
API (too much work). Is My understanding correct? If not, please
correct it.

Question1b: TestGtkEmbed was compiled as part of XULRunner SDK
compilation. If I am to build upon it, I would obviously have to
compile it as part of my app. However, a lot of stuff goes in
TestGtkEmbed, and I cannot find a good makefile or at least guidance
as to what flags, libs, and include paths to use. Any advice greatly
appreciated.

Question2: The ultimate goal would be using the mozilla engine in
headless mode - therefore I would eventually not need any of the gtk
stuff for display. In this case, am I doing an overkill by going
through GtkEmbedMoz? Would it be eventually simpler to use the
embedding API directly. Please comment.

[Benjamin Smedberg]

On 2/14/2011 3:05 PM, Peter wrote:
> However, the documentation on embedding is a bit fuzzy and all over
> the place. I am not even completely sure with the terminology as it
> keeps changing and is mixed up in the docs.

This is because there is little or no embedding community. The Mozilla
community is mainly focused on Firefox development, and embedding is not
a priority.

> Question0: How is embedding Gecko different from embedding XULRunner?
> Which of the two have I actually done above?

They are the same thing. XULRunner is the shipping "product" which
provides the Mozilla web engine, sometimes called gecko.

> Question1a: My understanding is that I can build upon GtkEmbedMoz
> (provides a gtk widget for the browser content pane - I think) and
> TestGtkEmbed (provides the actual window in which GtkEmbedMoz is
> displayed - I think) instead of working directly with the embedding
> API (too much work). Is My understanding correct? If not, please
> correct it.

TestGtkEmbed is example code for how you might use GtkEmbedMoz.

But in your case, I recommend that you *not* use GtkEmbedMoz except as
perhaps a template for embedding, and instead use the embedding API more
directly.

> Question1b: TestGtkEmbed was compiled as part of XULRunner SDK
> compilation. If I am to build upon it, I would obviously have to
> compile it as part of my app. However, a lot of stuff goes in
> TestGtkEmbed, and I cannot find a good makefile or at least guidance
> as to what flags, libs, and include paths to use. Any advice greatly
> appreciated.

https://developer.mozilla.org/en/XPCOM_Glue

> Question2: The ultimate goal would be using the mozilla engine in
> headless mode - therefore I would eventually not need any of the gtk
> stuff for display. In this case, am I doing an overkill by going
> through GtkEmbedMoz? Would it be eventually simpler to use the
> embedding API directly. Please comment.

The GtkEmbedMoz code will probably be removed in a future release of
Mozilla because it is buggy and unmaintained, and not the direction we
want for embedding in general. I would recommend using the low-level API
if possible. Also, you might want be aware of the gecko headless
rendering branch, which is rather old now but might be useful to you:

http://hg.mozilla.org/incubator/offscreen/

--BDS

[Benjamin Smedberg]

On 2/16/2011 11:28 PM, Peter Tsonev wrote:
> My understanding of the embedding process (one of the possible ways)
> is that 1) GRE is found,
> 2) XPCOMGlue is started up, 3) XUL functions are initialized, 4)
> XRE_InitEmbedding is called.
Yes. Note that I strongly recommend that you never try to use a "system
XULRunner". Always ship XULRunner with your code (e.g. in a xulrunner/
subdirectory) and look for it in that well-known location. DLL-hell will
result otherwise.

> Is this correct?
>
> I am able to execute all of the above correctly. Then I proceed to get
> nsIWebBrowser via do_CreateInstance and
> nsIWebNavigation via do_QueryInterface. Then I load a URL through the
> latter. It is this loading that fails.
> Please refer to the code below. Everything I described is attempted by
> the code.
You haven't initialized the webbrowser yet. I believe that requires
QIing it to nsIBaseWindow and calling create() (and maybe initWindow()
before that) on it.

--BDS

[moz_embedder]

I think am able to initialise the browser by QI-ing for nsBaseWindow
and then init and then create. I think it s initialised properly
because the return code on loading a URL through nsIWebNavigation is
OK.
I am not displaying anything since I am trying to run offscreen and I
only have the codes to work with. However, I cannot get to the
nsIDOMWindow (I get null) by doing:

nsCOMPtr<nsIDOMWindow> mDomWin;
mWebBrowser->GetContentDOMWindow(getter_AddRefs(mDomWin));

Any suggestions??
Here is a snippet with all the initialisations going on:

nsCOMPtr<nsIWebBrowser> mWebBrowser =
do_CreateInstance(NS_WEBBROWSER_CONTRACTID, &rv);
if (NS_FAILED(rv)) {
std::cout << "Couldn't create Web Browser" << std::endl;
exit(1);
}

nsCOMPtr<nsIWebNavigation> mWebNav =
do_QueryInterface(mWebBrowser, &rv);
if (NS_FAILED(rv)) {
std::cout << "Couldn't create Web Nav" << std::endl;
exit(1);
}

nsCOMPtr<nsIBaseWindow> baseWindow =
do_QueryInterface(mWebBrowser, &rv);
if (NS_FAILED(rv)) {
std::cout << "Couldn't QI base win" << std::endl;
exit(1);
}

WebBrowserChrome *cw = new WebBrowserChrome();
mWebBrowser->SetContainerWindow(static_cast<nsIWebBrowserChrome*>
(cw));

WebBrowserChromeUI *nw = new WebBrowserChromeUI(); //empty - I
dont want to display stuff
rv = baseWindow->InitWindow(nw, nsnull, 0, 0, 0, 0);
if (NS_FAILED(rv)) {
std::cout << "Couldn't init base win" << std::endl;
exit(1);
}

rv = baseWindow->Create();
if (NS_FAILED(rv)) {
std::cout << "Couldn't create base win" << std::endl;
exit(1);
}

rv = mWebNav->LoadURI(
NS_ConvertASCIItoUTF16("http://www.google.com/").get(),
nsIWebNavigation::LOAD_FLAGS_NONE,
nsnull,
nsnull,
nsnull);
if (NS_FAILED(rv)) {
std::cout << "Couldn't load URL: " << std::endl;
exit(1);
} else {
std::cout << "Maybe loaded url" << std::endl; //reaches
here :) :) :)
}

nsCOMPtr<nsIDOMDocument> mDomDoc;
nsCOMPtr<nsIDOMWindow> mDomWin;
rv = mWebBrowser->GetContentDOMWindow(getter_AddRefs(mDomWin));
if(mDomWin) { // mDomWin is NULL here :( :( :(
mDomWin->GetDocument(getter_AddRefs(mDomDoc));
if(!mDomDoc)
std::cout << "Cannot get DOM doc" << std::endl;
} else {
std::cout << "Cannot get dom win " << std::endl;
}

[Denis Lazarev]

Hello Peter,

Have you implemented nsIInterfaceRequestor for WebBrowserChrome class?

Best regards,
Denis Lazarev.