How can retrieved content be manipulated before being displayed by the browser?

[Steffen Heinzl]

Hi!

I'm trying to develop a Firefox (v2.0.0.6) extension, which should check
(and perhaps modify) the contents of a site having been retrieved,
before it is shown by firefox.

Up to now, I have registered an observer to the http-on-examine-response
event. The observer receives an HttpChannel as subject. I would like to
call subject.asyncOpen but I get an error that the method is unknown.

Error: Component returned failure code: 0x804b000f [nsIChannel.open] =
<unknown>

So, I copy the URI of the channel and have the IOservice create a new
channel. Before that, I have to remove the observer to prevent an
infinite loop since the creating a new channel will emit a new event.

Now I can call asyncOpen with a streamListener and read data from the
input stream. After modifying data, I could write them to disk and open
the file:// url in a new tab, but it would be better if the browser
showed the modified site automatically.

My main question is: How can I modify the data, so the modified web site
is displayed in the browser?

Another question is why I cannot use subject.asyncOpen. Instead I have
to open a new channel to the URI's address, and then call asyncOpen().

Best Regards and thanks in advance,
Steffen

P.S.: Up to now my implementation looks like this:

function Observer() {}

Observer.prototype = {
observe: function(subject,topic,data){
subject.QueryInterface(Components.interfaces.nsIChannel);
observerService.removeObserver(this,"http-on-examine-response");
dump(subject.URI.spec);
var ioService =
Components.classes["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);
var scriptableStream =
Components.classes["@mozilla.org/scriptableinputstream;1"].getService(Components.interfaces.nsIScriptableInputStream);
var channel = ioService.newChannel(subject.URI.spec,null,null);

channel.asyncOpen(streamListener, null);
// observerService.addObserver(this, "http-on-examine-response", false);
dump("opened channel"+"\n");
},
QueryInterface: function(iid) {
if (!iid.equals(Components.interfaces.nsIObserver))
throw Components.results.NS_ERROR_NO_INTERFACE;
dump("object of type nsIObserver");
return this;
}
}

var streamListener = {
data: "",
onDataAvailable: function( /*nsIRequest*/ request , /*nsISupports*/
context, /*nsIInputStream*/ inputStream , /*PRUint32*/ offset ,
/*PRUint32*/ count) {
var scriptableInputStream =
Components.classes["@mozilla.org/scriptableinputstream;1"].createInstance(Components.interfaces.nsIScriptableInputStream);
scriptableInputStream.init(inputStream);

this.data += scriptableInputStream.read(count);
dump("OnDataAvailable:"+ this.data);
this.data = this.data.replace("Erweiterte","Buh");
this.data = this.data.replace("Auf","Buh2");
dump("OnDataAvailable:"+ this.data);
},
onStartRequest: function(request, context){ dump("stepped into
onStartRequest" + "\n");},
onStopRequest: function(request, context, status)
{
if (Components.isSuccessCode(status))
{
dump("success\n");
}
var wm =
Components.classes["@mozilla.org/appshell/window-mediator;1"].getService(Components.interfaces.nsIWindowMediator);
var browserWindow = wm.getMostRecentWindow("navigator:browser");
// browserWindow.QueryInterface(Components.interfaces.nsIDOMJSWindow);
var browser = browserWindow.getBrowser();
browser.addTab("file:///E:/temp/test2.htm", null, null, null, null,
null);
//browser.removeCurrentTab();
},
QueryInterface: function(iid) {
if (!iid.equals(Components.interfaces.nsIStreamListener))
throw Components.results.NS_ERROR_NO_INTERFACE;
dump("object of type nsIStreamListener");
return this;
}
};

// initialization
function NSGetModule() {
observerService.addObserver(new Observer(),
http-on-examine-response", false);
dump("register observer\n");
dump("loaded grid services browser add-on\n");
return browserModule;
}

[Christian Biesinger]

Steffen Heinzl wrote:
> My main question is: How can I modify the data, so the modified web site
> is displayed in the browser?

It's probably easiest if you change the contentType property of the
channel and register a stream converter to convert from that to */*. The
stream converter can then do whatever it wants with the data.

> Another question is why I cannot use subject.asyncOpen. Instead I have
> to open a new channel to the URI's address, and then call asyncOpen().

Because the channel got already opened by the code that wanted the
data... You can't open a channel twice. Note that your approach of
opening a new channel (probably) causes another connection to the server

BTW, looking at your code:

var scriptableStream =
Components.classes["@mozilla.org/scriptableinputstream;1"].getService(Components.interfaces.nsIScriptableInputStream);

You shouldn't use getService for streams; use createInstance (your code
does that correct in another place)

-christian

[Steffen Heinzl]

Thank you very much. That really helped.

Now, I've got a question relating to the content type.
I have installed the wmlbrowser extension to my firefox and now try to
change the content type in the source code.
They define a content type for wml:

const WMLSTREAM_CONVERT_CONVERSION = "?from=text/vnd.wap.wml&to=*/*";

If I try to change it to one of the following versions nothing happens:
"?from=text/xml&to=text/html";
"?from=text/xml&to=*/*";
"?from=text/html&to=*/*";

The StreamConverter is not invoked. I wanted to react to wsdl documents
which are (in my case) of the content type text/xml.
The last (html) version is not invoked either when trying to obtain an
HTML site like google.com.

registerSelf contains the following code:

const WMLSTREAM_CONVERT_CONVERSION = "?from=text/xml&to=text/html";
//"?from=text/vnd.wap.wml&to=*/*";
const WMLSTREAM_CONVERTER_CONTRACTID =
"@mozilla.org/streamconv;1" + WMLSTREAM_CONVERT_CONVERSION;

const WMLSTREAM_CONVERTER_CID =
Components.ID("{51427b76-8626-4a72-bd5f-2ac0ce5d101a}");

WMLBrowserModule.registerSelf =
function (compMgr, fileSpec, location, type)
{
dump("registerSelf\n");
var compMgr =
compMgr.QueryInterface(Components.interfaces.nsIComponentRegistrar);

compMgr.registerFactoryLocation(WMLSTREAM_CONVERTER_CID,
"WML Stream Converter",
WMLSTREAM_CONVERTER_CONTRACTID,
fileSpec,
location,
type);

var catman = Components.classes["@mozilla.org/categorymanager;1"]
.getService(Components.interfaces.nsICategoryManager);
catman.addCategoryEntry("@mozilla.org/streamconv;1",
WMLSTREAM_CONVERT_CONVERSION,
"WML to HTML stream converter",
true, true);
};

Thanks again,
Steffen

[Steffen Heinzl]

A short addendum:
If I define a content type "?from=text/xml2&to=*/*"; and receive a
document with content-type text/xml2 everything works fine.

So, is seems that the content type is already registered (surprise!). Is
there a way to override the text/xml content-type handling of Firefox?

Thanks,
Steffen

[Christian Biesinger]

Steffen Heinzl wrote:
> So, is seems that the content type is already registered (surprise!). Is
> there a way to override the text/xml content-type handling of Firefox?

Not really... it's probably easiest to write an http-on-modify-request
observer and change the content type of the channel in it.

[Steffen Heinzl]

I thought http-on-modify-request was used to modify the HTTP message
sent to the HTTP server. Can I really use this to change the
content-type of the message I receive from the server? Or do I
misunderstand how this works?

Thanks again!

[Christian Biesinger]

Steffen Heinzl wrote:
> I thought http-on-modify-request was used to modify the HTTP message
> sent to the HTTP server. Can I really use this to change the
> content-type of the message I receive from the server? Or do I
> misunderstand how this works?

I'm sorry, you are correct. What I meant was http-on-examine-response.

[Steffen Heinzl]

I added the following code and everything seems to work fine.

//Observer-object
function Observer() {}

Observer.prototype = {
observe: function(subject,topic,data){
subject.QueryInterface(Components.interfaces.nsIChannel);

if (subject.contentType =="text/xml") subject.contentType = "text/xml2";

},
QueryInterface: function(iid) {
if (!iid.equals(Components.interfaces.nsIObserver))
throw Components.results.NS_ERROR_NO_INTERFACE;

return this;
}
}

Thank you very much. I really appreciate your help.
Steffen

[Steffen Heinzl]

Yes, I've noticed. Thanks again!

[Jan Wrobel]

Hi,

On Thu, 30 Aug 2007, Steffen Heinzl wrote:
> > [...]

> > I thought http-on-modify-request was used to modify the HTTP message
> > sent to the HTTP server. Can I really use this to change the
> > content-type of the message I receive from the server? Or do I
> > misunderstand how this works?
> >
> > Thanks again!
> I added the following code and everything seems to work fine.
>

I'm trying to do the same thing but from a C++ extension. My code
works fine only for requests for which URL is entered directly in the
address bar. For all other requests (ones that download images, css,
js files etc.) my "http-on-examine-response" is fired, it changes
ContentType, but converter is not invoked. Is javascript converter
invoked for all these additional requests? I wonder if I have to do
anything extra in C++ code to be able to capture all requests.

Regards,
Jan
http://firekeeper.mozdev.org

[Boris Zbarsky]

Jan Wrobel wrote:
> I'm trying to do the same thing but from a C++ extension. My code
> works fine only for requests for which URL is entered directly in the
> address bar. For all other requests (ones that download images, css,
> js files etc.) my "http-on-examine-response" is fired, it changes
> ContentType, but converter is not invoked.

Type converters are only automatically invoked for loads that are happening in a
docshell (so essentially toplevel load, frame, iframe).

-Boris

[Jan Wrobel]

On Tue, 16 Oct 2007, Boris Zbarsky wrote:

>[...]

> Type converters are only automatically invoked for loads that are happening in a
> docshell (so essentially toplevel load, frame, iframe).
>

So, currently there is no simple way to capture and filter all the
content browser gets? Is it right?

I would like to propose an improvement that would make it possible.
But I'm not aware if it isn't something developers are already working
on, I can't find any bug related to this issue but maybe I've missed
something.

My idea is to add following interface and make nsChannel implement it:

#include "nsIChannel.idl"
#include "nsIStreamListener.idl"

[scriptable, uuid(969cce1a-c045-4656-a0cb-93bf9a8ebb1a)]
interface nsITraceableChannel : nsIChannel
{
/*
* Channel's listener
*/
attribute nsIStreamListener listener;
};

This interface would allow to get and set StreamListener that was
passed in AsyncOpen call.
To make think simple, it would be sufficient to let the caller replace
listener only before its OnStartRequest() was called.

This solution would make possible replacing StreamListener in the
"on-examine-response" event with a new version that can pass content
to original listener after checking or manipulating it (something very
similar that converter do).

What do you think about this idea.
If it is accepted I can help to implement or test it.

Best regards,
Jan
http://firekeeper.mozdev.org

[Boris Zbarsky]

Jan Wrobel wrote:
> So, currently there is no simple way to capture and filter all the
> content browser gets? Is it right?

That's correct.

> But I'm not aware if it isn't something developers are already working
> on

I don't think anyone is working on this.

> My idea is to add following interface and make nsChannel implement it:

> This solution would make possible replacing StreamListener in the
> "on-examine-response" event with a new version that can pass content
> to original listener after checking or manipulating it (something very
> similar that converter do).

How would the interaction between different on-examine-response implementations
work?

-Boris

[Jan Wrobel]

On Sun, 21 Oct 2007, Boris Zbarsky wrote:

> Jan Wrobel wrote:
> > My idea is to add following interface and make nsChannel implement it:
> > This solution would make possible replacing StreamListener in the
> > "on-examine-response" event with a new version that can pass content
> > to original listener after checking or manipulating it (something very
> > similar that converter do).
>
> How would the interaction between different on-examine-response implementations
> work?

If few different "on-examine-response" observers replace channel's
StreamListener, a chain of listeners is created.

Lets say there are two observers: observer1 and observer2.

First, channel creator calls AsyncOpen(originalListener, ...)

Then observer1 receives "on-examine-response" notification. It gets
originalListener and replaces it with
observer1Listener. observer1Listener can pass OnStartRequest,
OnStopRequest, OnDataAvailable to the originalListener (it has
reference to it).

Next, observer2 receives "on-examine-response" notification. It gets
observer1Listener, replaces it with observer2Listener and can pass
calls to it.

So the call chain looks like this: nsHttpChannel calls
observer2Listener->OnStartRequest() it then calls
observer1Listener->OnStartRequest() and it calls
originalListener->OnStartRequest().
The same think happens for other calls.

If there are more observers, chain is further extended.

In this solution there is no way to control which observer is notified
first and where in the chain it is placed. Whether this control is
needed depends on the application. I think that for filters like
Firekeeper it isn't at all necessary.

Cheers,
Jan

[Jan Wrobel]

> > Jan Wrobel wrote:
> > > My idea is to add following interface and make nsChannel implement it:
> > > This solution would make possible replacing StreamListener in the
> > > "on-examine-response" event with a new version that can pass content
> > > to original listener after checking or manipulating it (something very
> > > similar that converter do).

> > >[...]

Getting back to the topic. What are your thoughts on this feature? Do you
think it can be accepted and added to Mozilla?

Cheers,
Jan

[Boris Zbarsky]

Jan Wrobel wrote:
> Getting back to the topic. What are your thoughts on this feature? Do you
> think it can be accepted and added to Mozilla?

My only concern with something like this would be the unpredictable behavior if
there are multiple consumers who all switch out the streamlistener.

Well, that and the problem we already have with stream converters: delaying
passing on OnStartRequest breaks things like "save as". This needs to be more
clearly documented somewhere...

-Boris

[Jan Wrobel]

On Sun, 28 Oct 2007, Boris Zbarsky wrote:

> Jan Wrobel wrote:
> > Getting back to the topic. What are your thoughts on this feature? Do you
> > think it can be accepted and added to Mozilla?
>
> My only concern with something like this would be the unpredictable behavior if
> there are multiple consumers who all switch out the streamlistener.

But idea is not to switch out streamlistener but to allow building
streamlisteners chain in which original stream listener is the last
one. The only unpredictable think here is an order of listeners in
this chains. But this is also unpredictable in other already existing
functionality. For example if you have multiple extensions that modify
request headers in "http-on-modify-request" headers that are sent can
be different depending on an order in which this event is passed to
extensions. And there is not mechanism to control this order, right?

> Well, that and the problem we already have with stream converters: delaying
> passing on OnStartRequest breaks things like "save as". This needs to be more
> clearly documented somewhere...

OK, so if there is a requirement (clean statement in the
documentation) that new listener has to pass OnStartRequest and
OnStopRequest to the original listener without any delay it would be
OK? For things like Firekeeper delaying OnStartRequest is not
necessary and I can't think of an application for which it is.
Firekeeper either passes OnStartRequest without any delay or cancels
request (doesn't pass OnStartRequest at all). Can such behaviour be
the source of some problems?

Cheers,
Jan