Tests with JIT disabled?
John J Barton
guess that Firebug somehow causes the JIT not to be running, so code
running while Firebug is active will be less tested than normal path
code unless the Firefox tests are run with JIT disabled as well as enabled.
jjb
Shawn Wilsher
Firebug would also disable JIT since attaching a debugger disables JIT.
Cheers,
Shawn
Boris Zbarsky
> I'm wondering if the Firefox tests are run with JIT enabled always? I
> guess that Firebug somehow causes the JIT not to be running, so code
> running while Firebug is active will be less tested than normal path
> code
This is the case no matter what. None of jsd, the debugger APIs it and
Firebug use, or the js code that gets run when there are jsd hooks set
up is "normal path code"....
In particular, I seem to recall the JS engine deoptimizing in ways other
than disabling jit when a debugger is attached.
-Boris
John J. Barton
> On 12/1/09 9:01 PM, John J Barton wrote:
>> I'm wondering if the Firefox tests are run with JIT enabled always? I
>> guess that Firebug somehow causes the JIT not to be running, so code
>> running while Firebug is active will be less tested than normal path
>> code
>
> This is the case no matter what. None of jsd, the debugger APIs it and
> Firebug use, or the js code that gets run when there are jsd hooks set
> up is "normal path code"....
Yes I know that part. Most of that code is probably not subject to
significant testing, a problem that is not easy to fix. But Firebug also
runs a lot of normal browser code and calls into a bunch of APIs all
with tracing off. In theory all of that code is tested with JIT on and
in theory all of it works just as well with JIT off. But I wonder if we
might just consider an occasional experiment to check the theory.
jjb
L. David Baron
> Yes I know that part. Most of that code is probably not subject to
> significant testing, a problem that is not easy to fix. But Firebug
> also runs a lot of normal browser code and calls into a bunch of
> APIs all with tracing off. In theory all of that code is tested with
> JIT on and in theory all of it works just as well with JIT off. But
> I wonder if we might just consider an occasional experiment to check
> the theory.
I think it would be very useful if we could run our test suites with
various extensions installed. I don't think it should be *too* hard
(though maybe I'm missing something) to hack the initializeProfile
function in build/automation.py.in to install an extension in the
profile and enable it (and configure it so it's on for all sites).
Then we could occasionally or regularly run our mochitests,
reftests, etc., with firebug (or other extensions) enabled.
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
Jesse Ruderman
> various extensions installed. I don't think it should be *too* hard
> (though maybe I'm missing something) to hack the initializeProfile
> function in build/automation.py.in to install an extension in the
> profile and enable it (and configure it so it's on for all sites).
> Then we could occasionally or regularly run our mochitests,
> reftests, etc., with firebug (or other extensions) enabled.
runreftest.py doesn't use the initializeProfile function in
build/automation.py.in. Instead, runreftest.py has its own
createReftestProfile function with a separate list of prefs. So you'd
have to add the new code in two places, or do a little refactoring
first.
Ted Mielczarek
> On Tuesday 2009-12-01 21:46 -0800, John J. Barton wrote:
> > Yes I know that part. Most of that code is probably not subject to
> > significant testing, a problem that is not easy to fix. But Firebug
> > also runs a lot of normal browser code and calls into a bunch of
> > APIs all with tracing off. In theory all of that code is tested with
> > JIT on and in theory all of it works just as well with JIT off. But
> > I wonder if we might just consider an occasional experiment to check
> > the theory.
>
> I think it would be very useful if we could run our test suites with
> various extensions installed. I don't think it should be *too* hard
> (though maybe I'm missing something) to hack the initializeProfile
> function in build/automation.py.in to install an extension in the
> profile and enable it (and configure it so it's on for all sites).
> Then we could occasionally or regularly run our mochitests,
> reftests, etc., with firebug (or other extensions) enabled.
>
This wouldn't be hard at all to set up, in fact runreftest.py already
installs the reftest harness as an extension (which is why it runs the app
twice). The problem is likely to just be the logistics of picking extensions
and running them and correlating the results.
-Ted
Boris Zbarsky
> I think it would be very useful if we could run our test suites with
> various extensions installed. I don't think it should be *too* hard
> (though maybe I'm missing something) to hack the initializeProfile
> function in build/automation.py.in to install an extension in the
> profile and enable it (and configure it so it's on for all sites).
> Then we could occasionally or regularly run our mochitests,
> reftests, etc., with firebug (or other extensions) enabled.
I think this would be a good idea. For one thing it would confirm or
deny my strong suspicion that Firebug leaks a bunch of stuff for the
process lifetime... (probably by holding on to it somewhere).
-Boris
Boris Zbarsky
> This wouldn't be hard at all to set up, in fact runreftest.py already
> installs the reftest harness as an extension (which is why it runs the app
> twice). The problem is likely to just be the logistics of picking extensions
> and running them and correlating the results.
Let's start with Firebug, since it's pretty common amongst the web
developer crowd and they form their impressions of our product based on
Firefox + Firebug, not just Firefox itself.
-Boris
Ted Mielczarek
> Let's start with Firebug, since it's pretty common amongst the web
> developer crowd and they form their impressions of our product based on
> Firefox + Firebug, not just Firefox itself.
>
I filed https://bugzilla.mozilla.org/show_bug.cgi?id=532445 on that.
-Ted
Serge Gautherie
> On 12/2/09 12:45 AM, L. David Baron wrote:
Fwiw,
>> I think it would be very useful if we could run our test suites with
>> various extensions installed.
SeaMonkey 2.0 is already running tests with CZ and DOMi enabled ...
> I think this would be a good idea. For one thing it would confirm or
> deny my strong suspicion that Firebug leaks a bunch of stuff for the
> process lifetime... (probably by holding on to it somewhere).
... but I couldn't enable Venkman because of
Bug 504730 - [SeaMonkey] mochitest-browser-chrome, test_idcheck.xul:
venkman.xul leaks 375 kB
Serge Gautherie
> ... but I couldn't enable Venkman because of
> Bug 504730 - [SeaMonkey] mochitest-browser-chrome, test_idcheck.xul:
> venkman.xul leaks 375 kB
Oh, I think the situatio is more like
"Venkman extension is enabled ... but loading its UI (in that test)
leaks"...
John J. Barton
I agree with the "Let's start" part, but Firebug is very different from
most addons. If this experiment is successful, I hope you will consider
one additional test: a Top 20 (or 50 or whatever) addons test. One third
of Firefox users run Firefox with an addon so one third of users run
Firefox in a way that is not routinely tested. A Top 20 test would
probably drop that number to few percent. Yes many addons have
enablement settings that may make these test have low coverage, but its
a start. I believe you can create a social network of addon developers
around such a test to increase addon developer engagement and success.
This is the path we are taking with Firebug extension developers.
jjb
Ted Mielczarek
> wrote:
> I agree with the "Let's start" part, but Firebug is very different from
> most addons. If this experiment is successful, I hope you will consider one
> additional test: a Top 20 (or 50 or whatever) addons test. One third of
> Firefox users run Firefox with an addon so one third of users run Firefox in
> a way that is not routinely tested. A Top 20 test would probably drop that
> number to few percent. Yes many addons have enablement settings that may
> make these test have low coverage, but its a start. I believe you can create
> a social network of addon developers around such a test to increase addon
> developer engagement and success. This is the path we are taking with
> Firebug extension developers.
>
Baby steps! :) I don't think anyone disagrees with you, it's just a matter
of the time required to get things going. We'll get something working with
Firebug, and take it from there.
-Ted