Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Is "Gecko" 1.9 only Firefox?

20 views
Skip to first unread message

Samuel Sidler

unread,
Mar 1, 2008, 7:40:15 PM3/1/08
to
(Cross-posting to dev-planning. Follow-ups to dev-platform.)

As we get closer to release, I'm seeing more and more core bugs
minused for "blocking1.9". Often time, this is good as we've plussed
many things that aren't nearly major enough to fix. However, many
times, these are clear regressions that simply don't affect Firefox.
As a Camino contributor in my off time, this brings up a pretty
critical question to me, personally: Are we considering Gecko 1.9 a
Firefox-only release?

Previous conversations between Damon and other Camino contributors (on
IRC a couple/few months ago) implied that we're indeed considering
Gecko 1.9 a proper Gecko release and giving other mozilla.org apps [1]
that rely on our codebase the same amount of core support they've
received in the past. I.e., not shipping with regressions that were
caused from changes made for Firefox and/or providing a clear way for
them to move forward to a new, supported way of implementing their
same features without hacking around bad changes.

However, I've seen drivers minus bugs and leave comments that counter
that idea. For example: "We certainly aren't going to block the
Firefox release on an issue that doesn't affect Firefox." [2] Fixing
such bugs later, on the actual timeline of another product's release
isn't always possible (changing major core components to fix other
products doesn't fly in point releases like 1.9.0.x given that those
changes usually can affect Firefox) and/or ends up creating huge hacks
in the codebase.

In the past, we've blocked on bugs in Gecko releases that didn't
affect Firefox but were clearly regressions from Firefox-focused
checkins. If this has changed, I think it needs to be announced
somewhere, very publicly, with a clear statement as to why we're
making that change in policy (and if it wasn't a policy, feel free to
substitute "behavior" there).

I understand our need to release Firefox RSN, but not fixing core bugs
we've caused simply because they don't affect Firefox doesn't seem
like a good thing for our developer community and only perpetuates
ideas that Mozilla doesn't care about other projects.

(At no point do I want to demean the work some developers have done to
care about regressions from their checkins that only happen in other
products. There are several developers who not only care, but go out
of their way to fix those regressions. I also am well aware of non-
Firefox bugs getting fixed in the core and approved for landing. I'm
specifically talking about regressions that have been minussed where
we've broken other products and caused regressions in our platform.)

If I'm way off base, please let me know, but this is the general
sentiment I've been getting from the bugs I watch.

-Sam

[1] http://www.mozilla.org/projects/#applications
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=382138#c30

Philip Chee

unread,
Mar 1, 2008, 11:14:37 PM3/1/08
to
On Sat, 1 Mar 2008 16:40:15 -0800 (PST), Samuel Sidler wrote:

> However, I've seen drivers minus bugs and leave comments that counter
> that idea. For example: "We certainly aren't going to block the
> Firefox release on an issue that doesn't affect Firefox." [2] Fixing
> such bugs later, on the actual timeline of another product's release
> isn't always possible (changing major core components to fix other
> products doesn't fly in point releases like 1.9.0.x given that those
> changes usually can affect Firefox) and/or ends up creating huge hacks
> in the codebase.

> In the past, we've blocked on bugs in Gecko releases that didn't
> affect Firefox but were clearly regressions from Firefox-focused
> checkins. If this has changed, I think it needs to be announced
> somewhere, very publicly, with a clear statement as to why we're
> making that change in policy (and if it wasn't a policy, feel free to
> substitute "behavior" there).

I guess this is part of the reason KaiRo has been thinking of proposing
a Gecko 1.9.1 branch for landing bugfixes needed by SeaMonkey,
Thunderbird, Sunbird and other projects that don't have a release cycle
synchronized to Firefox's.

Phil

--
Philip Chee <phi...@aleytys.pc.my>, <phili...@gmail.com>
http://flashblock.mozdev.org/ http://xsidebar.mozdev.org
Guard us from the she-wolf and the wolf, and guard us from the thief,
oh Night, and so be good for us to pass.
[ ]Quick! Close your mind!! Something might get in.
* TagZilla 0.066.6

Boris Zbarsky

unread,
Mar 1, 2008, 11:41:16 PM3/1/08
to
Philip Chee wrote:
> I guess this is part of the reason KaiRo has been thinking of proposing
> a Gecko 1.9.1 branch

Note that this would double the amount of work needed for security
releases, since at that point Firefox and the other projects would be on
different Gecko release branches.

Given the difficulty we've had with getting things fixed at all on the
1.8 branch, this is something to avoid if possible.

There's also the problem that this undercuts the "Gecko is Gecko" claim
we make.

For what it's worth, it looks to me like bug 382138 was misdiagnosed,
and then mistriaged based on this misdiagnosis. If it doesn't make 1.9,
you probably will need a 1.9.1 for the non-Firefox apps, if it gets
fixed at all, because it's not the sort of change we should be making on
a stable branch.

-Boris

sayrer

unread,
Mar 2, 2008, 12:49:42 AM3/2/08
to
On Mar 1, 7:40 pm, Samuel Sidler <samuel.sid...@gmail.com> wrote:
> As a Camino contributor in my off time, this brings up a pretty
> critical question to me, personally: Are we considering Gecko 1.9 a
> Firefox-only release?

Serious question: is there a Gecko 1.9 release, or are there multiple
product releases using most of the same Gecko code?

Trying to fix bugs in/for every Gecko product at once seems foolish to
me.

- Rob

Andrew Schultz

unread,
Mar 2, 2008, 1:10:23 AM3/2/08
to
sayrer wrote:
> Serious question: is there a Gecko 1.9 release, or are there multiple
> product releases using most of the same Gecko code?

All of the same Gecko code. We're all pulling CVS. There are no app
ifdefs in Gecko. Gecko is Gecko.

> Trying to fix bugs in/for every Gecko product at once seems foolish to
> me.

Are you suggesting each app fork Gecko? In the case at hand, I suspect
it wouldn't really matter unless and until the regression actually has a
fix, which (based on comments in the bug) won't be happening anytime soon.

--
Andrew Schultz
ajsc...@verizon.net
http://www.sens.buffalo.edu/~ajs42/

sayrer

unread,
Mar 2, 2008, 1:58:36 AM3/2/08
to
On Mar 2, 1:10 am, Andrew Schultz <ajsch...@verizon.net> wrote:
> sayrer wrote:
> > Serious question: is there a Gecko 1.9 release, or are there multiple
> > product releases using most of the same Gecko code?
>
> All of the same Gecko code.  We're all pulling CVS.  There are no app
> ifdefs in Gecko.  Gecko is Gecko.

This response skirts my original question. Is there a Gecko 1.9
release? I am not aware of one being planned, so I am not sure which
release Sam is referring to in his message.

- Rob

Andrew Schultz

unread,
Mar 2, 2008, 2:34:57 AM3/2/08
to
sayrer wrote:
> This response skirts my original question. Is there a Gecko 1.9
> release? I am not aware of one being planned, so I am not sure which
> release Sam is referring to in his message.

Your question was really, "what did Sam mean by 'release'". I don't
know exactly what he meant, but I got a pretty good idea from the
context ("the Gecko that's included with Firefox 3.0" would probably
suffice) and I really doubt a solid definition is needed to address the
issues he's bringing up.

sayrer

unread,
Mar 2, 2008, 3:01:40 AM3/2/08
to
On Mar 2, 2:34 am, Andrew Schultz <ajsch...@verizon.net> wrote:
> sayrer wrote:
> > This response skirts my original question. Is there a Gecko 1.9
> > release? I am not aware of one being planned, so I am not sure which
> > release Sam is referring to in his message.
>
> Your question was really, "what did Sam mean by 'release'".  I don't
> know exactly what he meant, but I got a pretty good idea from the
> context ("the Gecko that's included with Firefox 3.0" would probably
> suffice) and I really doubt a solid definition is needed to address the
> issues he's bringing up.

So the claim is that all Gecko bugs for Seamonkey and Thunderbird and
Camino (but not Komodo, Flock, and Songbird) must be fixed before
shipping Firefox 3? That is really stupid.

Another theory about one bug in question is that it's actually /really
bad/. That has nothing to do with project coordination, so I'm not
sure why the two issues are being conflated. If the patch shouldn't be
in the tree, take it up with the module owner and superreviewer, don't
splash project politics all over bugzilla.

Either way, I think this thread is making a big issue out of a
pedestrian problem, and I can't bring myself to care about it
anymore.

- Rob

bre...@mozilla.org

unread,
Mar 2, 2008, 4:42:21 AM3/2/08
to
On Mar 2, 12:01 am, sayrer <say...@gmail.com> wrote:
> On Mar 2, 2:34 am, Andrew Schultz <ajsch...@verizon.net> wrote:
>
> > sayrer wrote:
> > > This response skirts my original question. Is there a Gecko 1.9
> > > release? I am not aware of one being planned, so I am not sure which
> > > release Sam is referring to in his message.
>
> > Your question was really, "what did Sam mean by 'release'".  I don't
> > know exactly what he meant, but I got a pretty good idea from the
> > context ("the Gecko that's included with Firefox 3.0" would probably
> > suffice) and I really doubt a solid definition is needed to address the
> > issues he's bringing up.
>
> So the claim is that all Gecko bugs for Seamonkey and Thunderbird and
> Camino (but not Komodo, Flock, and Songbird) must be fixed before
> shipping Firefox 3? That is really stupid.

There are lots of platforms on top of platforms in the world; some
apps even act as platforms. Obviously, not all bugs are fixed at once.
On the other hand, Gecko has integrity as a platform beyond Firefox.
It's in the User-Agent. It has to have sane semantics and be a black
box as much as we can stand to engineer it that way, in memory-unsafe
programming languages and with time-to-market in mind, especially
Firefox market.

Making this a black/white, your way of the highway thing is false.
Obviously the bug in question arose because of Thunderbird and other
apps behaving differently. That's something to talk about, without
making false dilemmas between impossible coordination vs. Firefox uber
alles. For one thing, it's not clear to everyone what the right answer
is, so interacting with other apps' owners and peers makes sense. If
this doesn't fit in the schedule, so be it, but we're not done yet.

It should not be cut on the Procrustean bed of your (or my)
impatience. There are many hands at work in the community. The
particulars of this bug bear on XUL and its platform quality, so XUL
owners (you know who you are) need to be involved.

> Another theory about one bug in question is that it's actually /really
> bad/. That has nothing to do with project coordination, so I'm not
> sure why the two issues are being conflated.

They're not. They are both valid issues to bring up for a 1.9
milestone.

> If the patch shouldn't be
> in the tree, take it up with the module owner and superreviewer, don't
> splash project politics all over bugzilla.

This dodges the crucial point that module owner and reviewers are
obligated to look beyond one app when considering platform quality.

Politics is about competing ideas of the common good. Complaining
about it splashing on the bug after you've made your splash as
blocking1.9 rejecter is not playing fair. We have to argue about what
XUL menus should do in the disabled state, should they be cross-
platform. Etc. Meta-attitude against the other guy's "politics" really
doesn't help.

> Either way, I think this thread is making a big issue out of a
> pedestrian problem, and I can't bring myself to care about it
> anymore.

Maybe it's like the WK* API thing. :-/

/be

Michael Lefevre

unread,
Mar 2, 2008, 8:23:33 AM3/2/08
to
On 2008-03-02, sayrer <say...@gmail.com> wrote:
[snip]

>
> So the claim is that all Gecko bugs for Seamonkey and Thunderbird and
> Camino (but not Komodo, Flock, and Songbird) must be fixed before
> shipping Firefox 3? That is really stupid.

Why do you want to exclude the others? Would actually be good if core
bugs needed by Flock, Songbird and Komodo could be fixed too. ;)

Seems obvious that the "claim" that Gecko release is the same as the
Firefox release is a statement of fact, given that Firefox won't allow any
changes in Gecko after Firefox is released. That is the way it's been in
the past. The alternative, I guess, is forking Gecko and maintaining
parallel versions until Mozilla 2 - that also sounds stupid.

> Either way, I think this thread is making a big issue out of a
> pedestrian problem, and I can't bring myself to care about it
> anymore.

The particular bug in question seems pretty pedestrian. The crappy
attitude of Moz Corp developers isn't a pedestrian issue. If you don't
care about other Mozilla apps, then why on earth are you in charge of
managing a Mozilla release? Mozilla spends huge amounts of time trying to
say that it's not just about Firefox and how Mozilla benefits the
community and blah blah blah, and then people come along and go "Firefox 3
is supreme. I don't care about anyone else. Screw open source and
community - look at all our money and market share. Muahahaha!" (ok, so
maybe you didn't exactly say that, but that's how I read it)

However big the initial issue was or wasn't, you saying that you don't
care about it is an issue in itself. Please bring yourself to care about
it or go work on some other project... I think Flock are still hiring.

--
Michael

Robert Kaiser

unread,
Mar 2, 2008, 8:31:28 AM3/2/08
to
Boris Zbarsky schrieb:

> Philip Chee wrote:
>> I guess this is part of the reason KaiRo has been thinking of proposing
>> a Gecko 1.9.1 branch
>
> Note that this would double the amount of work needed for security
> releases, since at that point Firefox and the other projects would be on
> different Gecko release branches.

Right, I'm fully without you on that I don't like deviation of core
between the applications.

> For what it's worth, it looks to me like bug 382138 was misdiagnosed,
> and then mistriaged based on this misdiagnosis. If it doesn't make 1.9,
> you probably will need a 1.9.1 for the non-Firefox apps, if it gets
> fixed at all, because it's not the sort of change we should be making on
> a stable branch.

The problem is larger than one bug, actually:
https://bugzilla.mozilla.org/show_bug.cgi?id=398751 is a list of things
we know for now that are core bugs that would be marked SeaMonkey
blockers if they were in the SeaMonkey realm. And the one you mention
isn't even among those.

So, as much as I dislike having two branches and know the problems with
this, I currently fear we need to go that way, making 1.9.1 a 1.9 plus
core fixes that are too risky to take for FF at the current stage, and
maybe once it's been tested enough, do a FF 3.1 based on it, with a
direct update without overlap to 3.0.x to merge back to a single branch.
This might be a way that works for all, but we need to keep tight
control over what goes into core in 1.9.1 if we want to do it that way.

Robert Kaiser

Robert Kaiser

unread,
Mar 2, 2008, 8:43:58 AM3/2/08
to
sayrer schrieb:

I think there's a XULRunner release supposed to happen some time, that
is supposed to have the same core featureset and functionality as FF3
and to be able to be used by any XUL application - including products
from TomTom and others.

And then, we are supposed to have the same core code for all our apps
like FF3, SeaMonkey 2, Thunderbird 3, Sunbird, etc. so that security
updates of that code/branch apply to all Gecko apps and we don't
knowingly leave Mozilla applications vulnerable and damage the Mozilla
image of being very secure.

Robert Kaiser

Serge Gautherie

unread,
Mar 2, 2008, 9:17:30 AM3/2/08
to
Samuel Sidler wrote:

> Fixing
> such bugs later, on the actual timeline of another product's release
> isn't always possible (changing major core components to fix other
> products doesn't fly in point releases like 1.9.0.x given that those
> changes usually can affect Firefox) and/or ends up creating huge hacks
> in the codebase.

(Not reading the thread, so this may have already been said.)

From where we are now, a solution would be:
first, release Gecko 1.9 + Firefox 3;
then, do a Gecko 1.9.1 + Firefox 3.1 + other apps.

I don't know how doable this would be, but it's how I feel.

Mike Shaver

unread,
Mar 2, 2008, 9:48:36 AM3/2/08
to Samuel Sidler, dev-pl...@lists.mozilla.org
On Sat, Mar 1, 2008 at 7:40 PM, Samuel Sidler <samuel...@gmail.com> wrote:
> In the past, we've blocked on bugs in Gecko releases that didn't
> affect Firefox but were clearly regressions from Firefox-focused
> checkins. If this has changed, I think it needs to be announced
> somewhere, very publicly, with a clear statement as to why we're
> making that change in policy (and if it wasn't a policy, feel free to
> substitute "behavior" there).

We now have much richer test suite infrastructure than we've ever had
before; do you think it's reasonable to ask that dependent projects,
in cases where they aren't able or willing to fix the core bugs that
affect their apps themselves, contribute test cases that protect the
behaviour they need, which may not be exercised by Firefox as primary
test vehicle? In post-facto cases we'd probably need to commit them
as TODOs with a bug to track their enabling, though in such post-facto
cases where I have module authority I'd be more tempted to back out
the offending patch and commit the test cases in support of whatever
the subsequent fix is. It does mean that app owners need to take an
active role in helping make Gecko a better platform for their charges,
and can't rely simply on negotiating for blocking status to get
someone else to carry the costs of fixes, but I don't think the burden
is . And in some cases, as with Cocoa widgets, we would start to get
some badly-needed test coverage -- something I would imagine the
Camino folks would really shine at, given their historical expertise
in blending Gecko with Cocoa.

Is that something that non-Firefox app owners are up for? Are there
already test cases for the various SM2-blocking bugs to let people
working on the core know when they've fixed them? Do we have test
cases that will let Gecko folk know when they've fixed whatever is
ailing Camino or Sunbird or Komodo?

I agree with Rob in this narrow way at least: the bug in question is
just a crappy bug, which shouldn't have been "fixed" the way it was,
and I think it's a (happily rare) failing of reviewership and
stewardship that we're in some danger of shipping it in that state.
As someone who's often a reviewer and steward, I feel bad that we got
here as well, and I'll do what I can to make things right. Hard cases
make strange bedfellows, or something, but I don't think that this
outlier ("what we did works for Firefox[*], module owner doesn't want
to make a better change, we're not yet ready to make it an
ownership-confidence issue") is or should be taken as a trend for
Firefox-driver decisions.

[*] Except where it doesn't, by breaking reasonable cross-platform
assumptions for extension authors. This thing has enough strikes to
end a few innings by itself, I think.

We shouldn't block Firefox for things that don't affect Firefox, as a
tautology, but Gecko's effect on Firefox is deeper than as a shared
library to which it links, and if we're saying that we can't take
incompatible fixes on the maintenance branch -- sound policy, in
general! -- then we need to look a few moves ahead when locking out
fixes until the next feature release cycle. Firefox benefits from
Gecko's role as a shared technology, and we should try to avoid taking
unnecessary regressions in sister products. I don't know how many of
the SM2 bugs are "it would be good if this capability were more
general" rather than regressions in the 1.8->1.9 cycle, but the latter
should be triaged very differently, by my lights.

I'll say again, though: all apps, Firefox included, should be helping
Gecko help them by ladling on the test cases, and we should be
requiring tests for anything of significance that hits our trunk.
We're paying the cost every day for not having tests on key areas of
code, and the price only goes up as we get farther into the release.

I'm out of commas and synthetic em dashes now so I guess I'm done!

Mike

sayrer

unread,
Mar 2, 2008, 10:37:55 AM3/2/08
to
On Mar 2, 4:42 am, "bren...@mozilla.org" <bren...@mozilla.org> wrote:
>
> Maybe it's like the WK* API thing. :-/

On Mar 2, 8:23 am, Michael Lefevre <news+07.nos...@michaellefevre.com>
wrote:


> If you don't
> care about other Mozilla apps, then why on earth are you in charge of
> managing a Mozilla release?  

Good question. I'm interested in managing a Firefox release, but not a
Mozilla release.

So I think I'll make myself not a 1.9-driver anymore.

> Please bring yourself to care about
> it or go work on some other project... I think Flock are still hiring.

...

- Rob

Robert Kaiser

unread,
Mar 2, 2008, 12:11:09 PM3/2/08
to
Mike Shaver schrieb:

> We now have much richer test suite infrastructure than we've ever had
> before; do you think it's reasonable to ask that dependent projects,
> in cases where they aren't able or willing to fix the core bugs that
> affect their apps themselves, contribute test cases that protect the
> behaviour they need, which may not be exercised by Firefox as primary
> test vehicle?

Why should this be of any help when those who contribute patches have a
hard enough time getting them into core that forking the code tends to
sound better than improving the common platform?

I'm saying this as someone who is coordinating a in-CVS product that
tries very hard not to fork stuff, but from our experiences I easily see
why products maintained outside CVS tend to not fight for inclusion of
their improvements into core code but just keep them in their forked
repository.

Robert Kaiser

Boris Zbarsky

unread,
Mar 2, 2008, 12:40:59 PM3/2/08
to
Robert Kaiser wrote:
> Mike Shaver schrieb:
>> We now have much richer test suite infrastructure than we've ever had
>> before; do you think it's reasonable to ask that dependent projects,
>> in cases where they aren't able or willing to fix the core bugs that
>> affect their apps themselves, contribute test cases
>
> Why should this be of any help when those who contribute patches have a
> hard enough time getting them into core

I'm not sure what that has to do with shaver's proposal. I'll be honest: if
there were better test coverage of docshell and session history, it would be a
lot easier to accept patches for them (a point that I know is being a problem
right now). More tests directly translates into a much easier review and more
module owner confidence. Tests attached to a bug with no patch directly
translates into it being easier to fix the bug for someone else.

Just to put this in perspective, for the typical docshell bug that I fix and
write regression tests for writing the tests takes somewhere between 75% and 95%
of the total time I spend.

-Boris

Boris Zbarsky

unread,
Mar 2, 2008, 12:42:33 PM3/2/08
to
Mike Shaver wrote:
> do you think it's reasonable to ask that dependent projects,
> in cases where they aren't able or willing to fix the core bugs that
> affect their apps themselves, contribute test cases that protect the
> behaviour they need, which may not be exercised by Firefox as primary
> test vehicle?

That seems perfectly reasonable to me, for what it's worth. I'd be very much in
favor of it, as a Core developer. I can't speak for any of the non-Firefox
project owners, of course.

-Boris

Boris Zbarsky

unread,
Mar 2, 2008, 12:56:07 PM3/2/08
to
sayrer wrote:
> Serious question: is there a Gecko 1.9 release, or are there multiple
> product releases using most of the same Gecko code?

I think this is the wrong question to ask, because "Gecko 1.9" is not a static
thing. The right question is: "is there a single Gecko branch that is used for
all the product releases of the various products, or are here multiple Gecko
branches, one per product, with changes being ported between them on a piecemeal
basis?"

Historically, we have aimed to have a single release branch of the core code.
Part of the reasoning for this was the "Gecko is Gecko" idea (which has its own
reasons, both philosophical and practical). But another important part of the
reasoning is that maintaining multiple release branches is a waste of time.
It's particularly a waste of time if whoever is doing the porting could better
spend the time contributing to Core development. Put another way, Neil
Rashbrool and I, say, would much rather be fixing branch security bugs than
porting branch security bug fixes from the Firefox branch to the Seamonkey
branch. We'd much rather fix the bug once rather than fix it on the Seamonkey
branch and then have to port to the Firefox branch (assuming we have to, of
course. Then again, would we even need to backport to the Firefox branch? Or
just let someone who cares about Firefox handle it, somehow? The whole thing is
a recipe for inefficiency and bad feelings, in my opinion, and would hurt the
Firefox release branch, as well as the others. From the point of view of a
Firefox driver this hurt should be weighed against the pain of fixing the bugs
that could cause this situation before ship. I can even understand that after
said weighing a decision would be made to ship without fixing such a bug, but
what I find unacceptable is refusing to actually do said weighing.

Note that no one is saying that all Gecko bugs that affect non-Firefox apps need
to be fixed by the time Firefox 3 ships. In particular, if a bug can be fixed
in a point release, it makes all the sense in the world to do so, since the
other apps will ship later than Firefox. But for bugs that wouldn't be able to
go into a point release, I think we should have very very strong reasons if
we're not going to fix them.

If we do plan to do a Firefox 3.1 off Gecko 1.9.1, which will be allowed to
break compat to some extent, then we can push off some bugs to this. Last I
heard, brendan was strongly opposed to this.

-Boris

bre...@mozilla.org

unread,
Mar 2, 2008, 1:54:04 PM3/2/08
to
On Mar 2, 5:23 am, Michael Lefevre <news+07.nos...@michaellefevre.com>
wrote:

> The particular bug in question seems pretty pedestrian.  The crappy
> attitude of Moz Corp developers isn't a pedestrian issue.

Hey, knock that off. Collective blame is never justified for so
individual a set of actors and so fluid a situation. Mano, who
objected strenuously in the bug, is a "Moz Corp developer". So am I.

Why don't you stick to the substance of the argument, instead of
generalizing from Rob -- or attack Rob if you must, but cut the anti-
corporate groupthink.

/be

bre...@mozilla.org

unread,
Mar 2, 2008, 2:02:00 PM3/2/08
to
On Mar 2, 6:48 am, "Mike Shaver" <mike.sha...@gmail.com> wrote:
> ...

Mano thinks that pedestrian bug will break a bunch of add-ons. This is
not just an SM2 or Flock issue, it's a Firefox 3 question. It needs to
be answered before the bug can be blocking1.9-, with comity, clean
hands, and composure.

Drivers make the wrong call on blocking negations sometimes, but
instead of reconsidering the technical facts, this bug blew up into
something pretty ugly. Obviously there is a political (as in, what's
good for the _polis_, the common engine that's shared among apps, and
more important, the project that's made up of people sharing the code)
problem. Let's have it out, here, but in the mean time that bug needs
technical work and a blocking1.9 reconsideration.

/be

bre...@mozilla.org

unread,
Mar 2, 2008, 2:07:17 PM3/2/08
to
On Mar 2, 9:56 am, Boris Zbarsky <bzbar...@mit.edu> wrote:
> Note that no one is saying that all Gecko bugs that affect non-Firefox apps need
> to be fixed by the time Firefox 3 ships.  In particular, if a bug can be fixed
> in a point release, it makes all the sense in the world to do so, since the
> other apps will ship later than Firefox.

That's right, and that is exactly what we've done in the past.

> But for bugs that wouldn't be able to
> go into a point release, I think we should have very very strong reasons if
> we're not going to fix them.

We don't break web or platform API (not just nsIFoo but also XUL and
XBL) compatibility if we can help it on a minor (1.9.1) release. That
includes unfrozen APIs, wherefore the nsIBar_MOZILLA_1_8_BRANCH
interfaces. We burned plugin vendors badly in the past by doing
otherwise, so this is (again) not mindless "religion" or philosophy.

> If we do plan to do a Firefox 3.1 off Gecko 1.9.1, which will be allowed to
> break compat to some extent, then we can push off some bugs to this.  Last I
> heard, brendan was strongly opposed to this.

I'm in favor for 3.1 -- your pronoun "this" is confusing me. Do you
mean compatibility changes in a minor release? Those are still trouble
as far as I am concerned, based on long experience.

/be

Mark Banner

unread,
Mar 2, 2008, 2:11:00 PM3/2/08
to
Boris Zbarsky wrote:
> Note that no one is saying that all Gecko bugs that affect non-Firefox
> apps need to be fixed by the time Firefox 3 ships. In particular, if a
> bug can be fixed in a point release, it makes all the sense in the world
> to do so, since the other apps will ship later than Firefox. But for
> bugs that wouldn't be able to go into a point release, I think we should
> have very very strong reasons if we're not going to fix them.
>
> If we do plan to do a Firefox 3.1 off Gecko 1.9.1, which will be allowed
> to break compat to some extent, then we can push off some bugs to this.

I think these are important points. However, there's several things I
think that are also worth consideration:

a) Where non-Firefox apps don't have developers experience in certain
areas it is potentially harder for them to get bugs fixed by core
developers because the focus will have shifted to the next release.
b) If xulrunner was being shipped as a true platform, then these bugs
should be fixed anyway, as any potential xulrunner app might run across
them. This is probably something that needs to be thought more about for
Moz2, but I think worth bringing up here.
c) Is it always clear if a bug will need changes that will break compat
to fix them?

Standard8

Boris Zbarsky

unread,
Mar 2, 2008, 2:19:14 PM3/2/08
to
bre...@mozilla.org wrote:
>> If we do plan to do a Firefox 3.1 off Gecko 1.9.1, which will be allowed to
>> break compat to some extent, then we can push off some bugs to this. Last I
>> heard, brendan was strongly opposed to this.
>
> I'm in favor for 3.1 -- your pronoun "this" is confusing me. Do you
> mean compatibility changes in a minor release? Those are still trouble
> as far as I am concerned, based on long experience.

"this" was referring to a Gecko 1.9.1 with feature additions to the core (e.g.
adding new CSS selectors).

-Boris

Boris Zbarsky

unread,
Mar 2, 2008, 2:19:46 PM3/2/08
to
Mark Banner wrote:
> c) Is it always clear if a bug will need changes that will break compat
> to fix them?

Not always, no. But most of the time, I think it is.

-Boris

Message has been deleted
Message has been deleted

Samuel Sidler

unread,
Mar 2, 2008, 4:48:31 PM3/2/08
to
On Mar 1, 8:41 pm, Boris Zbarsky <bzbar...@mit.edu> wrote:
> Philip Chee wrote:
> > I guess this is part of the reason KaiRo has been thinking of proposing
> > a Gecko 1.9.1 branch
>
> Note that this would double the amount of work needed for security
> releases, since at that point Firefox and the other projects would be on
> different Gecko release branches.
>
> Given the difficulty we've had with getting things fixed at all on the
> 1.8 branch, this is something to avoid if possible.

I completely agree with this. This isn't the way to go.

> For what it's worth, it looks to me like bug 382138 was misdiagnosed,
> and then mistriaged based on this misdiagnosis. If it doesn't make 1.9,
> you probably will need a 1.9.1 for the non-Firefox apps, if it gets
> fixed at all, because it's not the sort of change we should be making on
> a stable branch.

This is one of many bugs. KaiRo links to more in the next message, but
there are even more than that. I simply linked to that one because
I've been flamed for not providing proper references in the past (and
that happens to be the bug that pushed me over the top).

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 4:50:40 PM3/2/08
to
On Mar 2, 5:31 am, Robert Kaiser <ka...@kairo.at> wrote:
> The problem is larger than one bug, actually:https://bugzilla.mozilla.org/show_bug.cgi?id=398751is a list of things

> we know for now that are core bugs that would be marked SeaMonkey
> blockers if they were in the SeaMonkey realm. And the one you mention
> isn't even among those.

Much larger. There are bugs I'm sure you're missing in your list as
well. Several Camino contributors gave up long ago and assumed nothing
would really get fixed.

> So, as much as I dislike having two branches and know the problems with
> this, I currently fear we need to go that way, making 1.9.1 a 1.9 plus
> core fixes that are too risky to take for FF at the current stage, and
> maybe once it's been tested enough, do a FF 3.1 based on it, with a
> direct update without overlap to 3.0.x to merge back to a single branch.
> This might be a way that works for all, but we need to keep tight
> control over what goes into core in 1.9.1 if we want to do it that way.

I don't think it's reasonable to do this and it would end up hurting
the platform in the long run. Not maintaining the platform we create
is shitty all around. And there's really been no guarantee that a
1.9.1 would be any better.

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 4:52:20 PM3/2/08
to
On Mar 1, 9:49 pm, sayrer <say...@gmail.com> wrote:
> On Mar 1, 7:40 pm, Samuel Sidler <samuel.sid...@gmail.com> wrote:
>
> > As a Camino contributor in my off time, this brings up a pretty
> > critical question to me, personally: Are we considering Gecko 1.9 a
> > Firefox-only release?

>
> Serious question: is there a Gecko 1.9 release, or are there multiple
> product releases using most of the same Gecko code?

Brendan answered this, but I'll reiterate: the Gecko 1.9 "release" is
that wonderful revision we add to our UA string that all other Gecko/
Mozilla-based apps ship with.

> Trying to fix bugs in/for every Gecko product at once seems foolish to
> me.

That's not what I was asking for. You know that, so let's not get off
track here.

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 4:56:03 PM3/2/08
to
On Mar 2, 12:01 am, sayrer <say...@gmail.com> wrote:
> > Your question was really, "what did Sam mean by 'release'". I don't
> > know exactly what he meant, but I got a pretty good idea from the
> > context ("the Gecko that's included with Firefox 3.0" would probably
> > suffice) and I really doubt a solid definition is needed to address the
> > issues he's bringing up.
>
> So the claim is that all Gecko bugs for Seamonkey and Thunderbird and
> Camino (but not Komodo, Flock, and Songbird) must be fixed before
> shipping Firefox 3? That is really stupid.

Um, no. The claim is exactly what it is and nothing more, like you
seem to make it out to be. "Fix the problems in the platform we've
created or provide supported workarounds." I thought I was fairly
clear about that in my initial message.

> Another theory about one bug in question is that it's actually /really
> bad/. That has nothing to do with project coordination, so I'm not
> sure why the two issues are being conflated. If the patch shouldn't be
> in the tree, take it up with the module owner and superreviewer, don't
> splash project politics all over bugzilla.

Please. You told me to move that bug to m.d.platform. I realized that
it was part of a larger problem and decided to address that problem.
The fact that I pointed a specific bug is a Good Thing. If I hadn't,
you would have demanded examples as you have from me before. Now that
I have, you've decided to focus on that one bug and issue and ignore
the overall problem.

> Either way, I think this thread is making a big issue out of a
> pedestrian problem, and I can't bring myself to care about it
> anymore.

Then don't.

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 5:01:01 PM3/2/08
to
On Mar 2, 5:23 am, Michael Lefevre <news+07.nos...@michaellefevre.com>
wrote:
> On 2008-03-02, sayrer <say...@gmail.com> wrote:
> > So the claim is that all Gecko bugs for Seamonkey and Thunderbird and
> > Camino (but not Komodo, Flock, and Songbird) must be fixed before
> > shipping Firefox 3? That is really stupid.
>
> Why do you want to exclude the others? Would actually be good if core
> bugs needed by Flock, Songbird and Komodo could be fixed too. ;)

I'm the one who (intentionally) excluded others in my initial post.

Flock, Songbird, and Komodo (and others, really) aren't under the
auspices of "mozilla.org" and, specifically, aren't in our codebase or
supported applications of the Mozilla Foundation.

What's more, those three in particular all have major forms of funding
that most of the other projects don't have (Thunderbird excluded).

> The particular bug in question seems pretty pedestrian. The crappy
> attitude of Moz Corp developers isn't a pedestrian issue. If you don't
> care about other Mozilla apps, then why on earth are you in charge of
> managing a Mozilla release? Mozilla spends huge amounts of time trying to
> say that it's not just about Firefox and how Mozilla benefits the
> community and blah blah blah, and then people come along and go "Firefox 3
> is supreme. I don't care about anyone else. Screw open source and
> community - look at all our money and market share. Muahahaha!" (ok, so
> maybe you didn't exactly say that, but that's how I read it)

Let's not get off-topic. I work for the Mozilla Corporation, so please
don't talk about crappy attitudes of people there. This is exactly the
idea I don't want to perpetuate. People care or we wouldn't be talking
about it. See my initial post.

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 5:02:44 PM3/2/08
to
On Mar 2, 7:37 am, sayrer <say...@gmail.com> wrote:
> > If you don't
> > care about other Mozilla apps, then why on earth are you in charge of
> > managing a Mozilla release?
>
> Good question. I'm interested in managing a Firefox release, but not a
> Mozilla release.
>
> So I think I'll make myself not a 1.9-driver anymore.

That sounds like a good idea to me. If you don't care about the
release you're working on (which, to be clear, isn't just Firefox but
includes Gecko/Mozilla 1.9), then you shouldn't be driving it.

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 5:04:19 PM3/2/08
to
On Mar 2, 5:43 am, Robert Kaiser <ka...@kairo.at> wrote:
> I think there's a XULRunner release supposed to happen some time, that
> is supposed to have the same core featureset and functionality as FF3
> and to be able to be used by any XUL application - including products
> from TomTom and others.

I'm not sure if there's an official XULRunner release planned for
Gecko 1.9, but if there is then yes. :)

> And then, we are supposed to have the same core code for all our apps
> like FF3, SeaMonkey 2, Thunderbird 3, Sunbird, etc. so that security
> updates of that code/branch apply to all Gecko apps and we don't
> knowingly leave Mozilla applications vulnerable and damage the Mozilla
> image of being very secure.

Don't forget Camino 2! ;)

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 5:06:23 PM3/2/08
to
On Mar 2, 9:56 am, Boris Zbarsky <bzbar...@mit.edu> wrote:
> I can even understand that after
> said weighing a decision would be made to ship without fixing such a bug, but
> what I find unacceptable is refusing to actually do said weighing.

Exactly.

> Note that no one is saying that all Gecko bugs that affect non-Firefox apps need
> to be fixed by the time Firefox 3 ships. In particular, if a bug can be fixed
> in a point release, it makes all the sense in the world to do so, since the
> other apps will ship later than Firefox. But for bugs that wouldn't be able to
> go into a point release, I think we should have very very strong reasons if
> we're not going to fix them.

I completely agree with this.

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 5:09:10 PM3/2/08
to
On Mar 2, 11:11 am, Mark Banner <bugzi...@invalid.standard8.plus.com>
wrote:

> a) Where non-Firefox apps don't have developers experience in certain
> areas it is potentially harder for them to get bugs fixed by core
> developers because the focus will have shifted to the next release.

I can say this is a huge problem for Camino where most of our
developers don't really ever deal with Gecko. They all speak Obj-C and
simply use the wrappers around Gecko we've had for a while to access
it.

> b) If xulrunner was being shipped as a true platform, then these bugs
> should be fixed anyway, as any potential xulrunner app might run across
> them. This is probably something that needs to be thought more about for
> Moz2, but I think worth bringing up here.

Yeah, this will be an issue more with Mozilla 2. Since XULRunner-as-
platform isn't yet, I'm not sure we can focus on that right now.

-Sam

Samuel Sidler

unread,
Mar 2, 2008, 5:11:23 PM3/2/08
to
On Mar 2, 6:17 am, Serge Gautherie <sgautherie...@free.fr> wrote:
> From where we are now, a solution would be:
> first, release Gecko 1.9 + Firefox 3;
> then, do a Gecko 1.9.1 + Firefox 3.1 + other apps.
>
> I don't know how doable this would be, but it's how I feel.

The problem with this is that then "other apps" can't release until
Gecko 1.9.1 is done. (Asking a question I know no one can answer...)
How long is that? A year? Six months? I don't know the release
schedules of other apps, but what if one wants to release their next
major version in July? Keeping the platform usable by other apps is a
Good Thing.

-Sam

Michael Lefevre

unread,
Mar 2, 2008, 5:15:48 PM3/2/08
to
On 2008-03-02, bre...@mozilla.org <bre...@mozilla.org> wrote:
> On Mar 2, 5:23 am, Michael Lefevre <news+07.nos...@michaellefevre.com>
> wrote:
>> The particular bug in question seems pretty pedestrian.  The crappy
>> attitude of Moz Corp developers isn't a pedestrian issue.
>
> Hey, knock that off. Collective blame is never justified for so
> individual a set of actors and so fluid a situation. Mano, who
> objected strenuously in the bug, is a "Moz Corp developer". So am I.

That part was badly worded - I actually meant "That any MoCo developer has
a crappy attitude isn't a pedestrian issue". But, that aside, the whole
tone of my post was wrong, as was the attack on Rob personally rather than
his statement(s). Other people are better placed to make the argument in a
sensible way, and they (starting with you) have. My apologies to Rob and
everyone else for my previous dumb post...

--
Michael

Samuel Sidler

unread,
Mar 2, 2008, 5:22:42 PM3/2/08
to
On Mar 2, 6:48 am, "Mike Shaver" <mike.sha...@gmail.com> wrote:
> We now have much richer test suite infrastructure than we've ever had
> before; do you think it's reasonable to ask that dependent projects,
> in cases where they aren't able or willing to fix the core bugs that
> affect their apps themselves, contribute test cases that protect the
> behaviour they need, which may not be exercised by Firefox as primary
> test vehicle?

I think, in theory, it's reasonable. In practice, I'm not so sure, but
I definitely think it's worth trying and pushing for.

I'll talk about Camino since I know it best...

Most Camino contributors are Obj-C writers and don't speak Gecko at
all (or very little). This is a bad thing when you think about
Mozilla, but because of the way Camino's written, it's a good thing
for attracting more Mac developers to the project.

We're small. Really small. Writing testcases will benefit the platform
and us in the long run, but it takes time. Time that most developers
would rather spend fixing bugs and working on new features. That being
said, many contributors have done an incredible job of linking to
appropriate testcases in bugs throughout this development cycle. Those
weren't automated testcases, but making them automated isn't nearly as
hard when manual ones exist. I kind of think this is something the
patch writers of various bugs should be working on, when given manual
testcases.

> It does mean that app owners need to take an
> active role in helping make Gecko a better platform for their charges,
> and can't rely simply on negotiating for blocking status to get
> someone else to carry the costs of fixes, but I don't think the burden
> is . And in some cases, as with Cocoa widgets, we would start to get
> some badly-needed test coverage -- something I would imagine the
> Camino folks would really shine at, given their historical expertise
> in blending Gecko with Cocoa.

I agree with this, but I do question how much burden it would really
be, as I've explained above.

> Is that something that non-Firefox app owners are up for? Are there
> already test cases for the various SM2-blocking bugs to let people
> working on the core know when they've fixed them? Do we have test
> cases that will let Gecko folk know when they've fixed whatever is
> ailing Camino or Sunbird or Komodo?

I don't have answers for these questions, but I definitely think we
should strive to write more tests to protect against regressions.
There's no way a reasonable person can argue with that.

> I agree with Rob in this narrow way at least: the bug in question is
> just a crappy bug, which shouldn't have been "fixed" the way it was,
> and I think it's a (happily rare) failing of reviewership and
> stewardship that we're in some danger of shipping it in that state.
> As someone who's often a reviewer and steward, I feel bad that we got
> here as well, and I'll do what I can to make things right. Hard cases
> make strange bedfellows, or something, but I don't think that this
> outlier ("what we did works for Firefox[*], module owner doesn't want
> to make a better change, we're not yet ready to make it an
> ownership-confidence issue") is or should be taken as a trend for
> Firefox-driver decisions.

I don't want to get caught up on one bug. That happens to be the bug
that pushed me over the edge, but there are others as KaiRo linked to
above (and more that aren't in tracking bugs that I just don't have
time to push back on). A few bugs does not a trend make, necessarily,
but we need to stop, look around, and make sure we're not making
things horrible overall.

> We shouldn't block Firefox for things that don't affect Firefox, as a
> tautology, but Gecko's effect on Firefox is deeper than as a shared
> library to which it links, and if we're saying that we can't take
> incompatible fixes on the maintenance branch -- sound policy, in
> general! -- then we need to look a few moves ahead when locking out
> fixes until the next feature release cycle. Firefox benefits from
> Gecko's role as a shared technology, and we should try to avoid taking
> unnecessary regressions in sister products.

Wholeheartedly agree.

> I'll say again, though: all apps, Firefox included, should be helping
> Gecko help them by ladling on the test cases, and we should be
> requiring tests for anything of significance that hits our trunk.
> We're paying the cost every day for not having tests on key areas of
> code, and the price only goes up as we get farther into the release.

Agreed again.

Overall, you're calling for more test cases and I agree with that.
Part of me wants to move to "no checkins without tests" for Core
checkins as a rule, with few exceptions. I think that would mitigate a
lot of this. (Although it'd probably have a negative impact on the
tinderbox page which would only get wider. We'll need to find a better
way to display tinderbox. ;) )

-Sam

bre...@mozilla.org

unread,
Mar 2, 2008, 5:23:52 PM3/2/08
to
On Mar 2, 11:19 am, Boris Zbarsky <bzbar...@mit.edu> wrote:

I'm in favor provided the feature additions can be feature-tested or
object-detected without version-locking the only way. The 1.9.1 (1.x.y
instead of 1.x.0.z in general) is novel too. In the face of increasing
competition and accelerating browser evolution, doing minor *additive*
releases in addition to the major milestones and the patch releases
makes sense. Not doing these, waiting 1-2 years between releases, is
no good for users or developers.

/be

Samuel Sidler

unread,
Mar 2, 2008, 5:32:05 PM3/2/08
to
On Mar 2, 11:02 am, "bren...@mozilla.org" <bren...@mozilla.org> wrote:
> Drivers make the wrong call on blocking negations sometimes, but
> instead of reconsidering the technical facts, this bug blew up into
> something pretty ugly. Obviously there is a political (as in, what's
> good for the _polis_, the common engine that's shared among apps, and
> more important, the project that's made up of people sharing the code)
> problem. Let's have it out, here, but in the mean time that bug needs
> technical work and a blocking1.9 reconsideration.

Yeah... My concern here wasn't one bug, but an overall feeling for how
drivers have been triaging bugs that don't specifically affect Firefox
(or don't appear to, anyway).

It'd help immensely if everyone (drivers, developers, other projects,
etc) was one the same page as to what "Gecko 1.9" (or Mozilla 1.9,
since you seem to prefer that ;) ) is and if the blocking flag should
represent it or Firefox 3. My impression was that it represents the
core technology; one that will soon "freeze" and only accept
reasonably small-impact patches or security/stability fixes. If that's
true, and if drivers are thinking the same thing, then we need to take
a strong look at these patches.

If that's not true, what *is* true needs to be broadcast throughout
the community.

But I think, based on the replies in this thread, we're on the same
page.

-Sam

Karsten Düsterloh

unread,
Mar 2, 2008, 5:32:36 PM3/2/08
to
Mike Shaver aber hob zu reden an und schrieb:

> do you think it's reasonable to ask that dependent projects,
> in cases where they aren't able or willing to fix the core bugs that
> affect their apps themselves, contribute test cases that protect the
> behaviour they need, which may not be exercised by Firefox as primary
> test vehicle?

Yes.
And as the rather lesser imp whose thoughts upon bug vs. feature kind of
caused the escalation here - if I had known that it'd create such
uproar, I'd happily have written a mochitest(?) back then (which I have
never done so far)...

> Is that something that non-Firefox app owners are up for? Are there
> already test cases for the various SM2-blocking bugs to let people
> working on the core know when they've fixed them? Do we have test
> cases that will let Gecko folk know when they've fixed whatever is
> ailing Camino or Sunbird or Komodo?

Not that I know of, especially mailnews has almost no testcases outside
of addressbook, and even those are more prechecking interfaces or patch
regressions.

> I'll say again, though: all apps, Firefox included, should be helping
> Gecko help them by ladling on the test cases, and we should be
> requiring tests for anything of significance that hits our trunk.
> We're paying the cost every day for not having tests on key areas of
> code, and the price only goes up as we get farther into the release.

Yep.


Karsten
--
Feel free to correct my English. :)

Samuel Sidler

unread,
Mar 2, 2008, 5:38:00 PM3/2/08
to
On Mar 2, 11:02 am, "bren...@mozilla.org" <bren...@mozilla.org> wrote:
> Drivers make the wrong call on blocking negations sometimes, but
> instead of reconsidering the technical facts, this bug blew up into
> something pretty ugly. Obviously there is a political (as in, what's
> good for the _polis_, the common engine that's shared among apps, and
> more important, the project that's made up of people sharing the code)
> problem. Let's have it out, here, but in the mean time that bug needs
> technical work and a blocking1.9 reconsideration.

Yeah... My concern here wasn't one bug, but an overall feeling for how

Message has been deleted
Message has been deleted

Robert Kaiser

unread,
Mar 2, 2008, 6:34:39 PM3/2/08
to
Samuel Sidler wrote:
> On Mar 2, 5:43 am, Robert Kaiser<ka...@kairo.at> wrote:
>> I think there's a XULRunner release supposed to happen some time, that
>> is supposed to have the same core featureset and functionality as FF3
>> and to be able to be used by any XUL application - including products
>> from TomTom and others.
>
> I'm not sure if there's an official XULRunner release planned for
> Gecko 1.9, but if there is then yes. :)

I'm not sure what's official in XULRunner terms at all, but I know that
a good number of projects is already basing their current development on
1.9something code, even with the bugs we are talking about, and I'm
pretty sure a number of those projects have their own workarounds for
such issues in place.

That said, in some cases, like e.g. for customizable toolbars in
SeaMonkey, we probably will just fork the toolkit code that isn't up to
our standards. Things like that are harder when it comes to crashes in
bfcache, for example (all those have patches that for some reason have
been waiting for toolkit reviews a long time - you know, non-Firefox
code just seems to have no priority for a lot of core developers).

>> And then, we are supposed to have the same core code for all our apps
>> like FF3, SeaMonkey 2, Thunderbird 3, Sunbird, etc. so that security
>> updates of that code/branch apply to all Gecko apps and we don't
>> knowingly leave Mozilla applications vulnerable and damage the Mozilla
>> image of being very secure.
>
> Don't forget Camino 2! ;)

Oh, the one that gets rid of xpfe? :p

Robert Kaiser

Boris Zbarsky

unread,
Mar 2, 2008, 6:36:36 PM3/2/08
to
Samuel Sidler wrote:
> Most Camino contributors are Obj-C writers and don't speak Gecko at
> all (or very little).

Note that for most test-writing you only need to speak JS, HTML, and whatever
you're testing.

> Writing testcases will benefit the platform
> and us in the long run, but it takes time. Time that most developers
> would rather spend fixing bugs and working on new features.

Yes, this is the case for platform developers too. ;)

> Those weren't automated testcases, but making them automated isn't nearly as
> hard when manual ones exist.

Sadly, this is not always true. Just to reiterate: for a typical docshell bug,
the tests take somewhere between 75% and 95% of the total time I spend on the
bug. That's starting with a manual testcase in a bug. Automating anything that
involves a good bit of user interaction in the original testcase is always a pain.

Of course automating a layout bug with reftest is often easy. But anything
involving the UI is a huge pain to automate. I suspect more of the app-parity
bugs are in this latter category.

> I kind of think this is something the
> patch writers of various bugs should be working on, when given manual
> testcases.

That's where we are right now. The result is spotty test coverage in a lot of
cases, and more time spent writing tests than fixing bugs in others....

-Boris

Boris Zbarsky

unread,
Mar 2, 2008, 6:37:54 PM3/2/08
to
bre...@mozilla.org wrote:
> I'm in favor provided the feature additions can be feature-tested or
> object-detected without version-locking the only way.

Or are dependent on CSS forward-compatible parsing, I assume?

> In the face of increasing
> competition and accelerating browser evolution, doing minor *additive*
> releases in addition to the major milestones and the patch releases
> makes sense. Not doing these, waiting 1-2 years between releases, is
> no good for users or developers.

OK. Something's clearly changed since the last time I heard your views on this.
;) That approach makes a lot of sense to me.

-Boris

Boris Zbarsky

unread,
Mar 2, 2008, 6:43:47 PM3/2/08
to
Robert Kaiser wrote:
> Things like that are harder when it comes to crashes in
> bfcache, for example (all those have patches that for some reason have
> been waiting for toolkit reviews a long time

They've been waiting on core, not toolkit reviews. And the reason is simple:
reviewing them will just take (me, at least) a lot of time. I just haven't had
that sort of time available for anything Mozilla-related in many months.

-Boris

Robert Kaiser

unread,
Mar 2, 2008, 6:53:11 PM3/2/08
to
Mike Shaver wrote:
> Is that something that non-Firefox app owners are up for? Are there
> already test cases for the various SM2-blocking bugs to let people
> working on the core know when they've fixed them?

I can't answer that, but I agree those would be good to have. I for
myself am trying to do project coordination and product management, but
am not too skilled in working on the code itself. Additionally, things
like that bfcache crash we have in our "Undo Closed Tabs" implementation
(bug 358599, the right fix is probably bug 396519 if I read things
correctly) have patches and even tests but wait eternally for reviews :(

Robert Kaiser

John J. Barton

unread,
Mar 2, 2008, 8:34:20 PM3/2/08
to
Boris Zbarsky wrote:
>
>... Just to reiterate: for a typical
> docshell bug, the tests take somewhere between 75% and 95% of the total
> time I spend on the bug. That's starting with a manual testcase in a
> bug. Automating anything that involves a good bit of user interaction
> in the original testcase is always a pain.
>
> Of course automating a layout bug with reftest is often easy. But
> anything involving the UI is a huge pain to automate. I suspect more of
> the app-parity bugs are in this latter category.

Is there somewhere that explains what automation mechanism/UI testing
mechanism you/Mozilla uses?

If this tool could be made end-user friendly then at least some
reporters could propose better tests.

John

bre...@mozilla.org

unread,
Mar 2, 2008, 8:43:20 PM3/2/08
to
On Mar 2, 3:37 pm, Boris Zbarsky <bzbar...@mit.edu> wrote:

> bren...@mozilla.org wrote:
> > I'm in favor provided the feature additions can be feature-tested or
> > object-detected without version-locking the only way.
>
> Or are dependent on CSS forward-compatible parsing, I assume?

Of course -- nice if your web language has such syntactic future-
proofing.

> > In the face of increasing
> > competition and accelerating browser evolution, doing minor *additive*
> > releases in addition to the major milestones and the patch releases
> > makes sense. Not doing these, waiting 1-2 years between releases, is
> > no good for users or developers.
>
> OK.  Something's clearly changed since the last time I heard your views on this.
>   ;)  That approach makes a lot of sense to me.

Not sure when that was, but I've been promoting the 3.x for focused
and compatibly added .x, without turning into a big all-hands-on-deck
massive trunk continuous integration exercise, for a while.

/be

bre...@mozilla.org

unread,
Mar 2, 2008, 9:38:17 PM3/2/08
to
On Mar 2, 2:32 pm, Samuel Sidler <samuel.sid...@gmail.com> wrote:
> On Mar 2, 11:02 am, "bren...@mozilla.org" <bren...@mozilla.org> wrote:
> Yeah... My concern here wasn't one bug, but an overall feeling for how
> drivers have been triaging bugs that don't specifically affect Firefox
> (or don't appear to, anyway).

I'm not going to generalize from this bug. I've seen too few cases of
controversy like this, but it could be that other apps or platform
consumers (extensions) are not being squeaky wheels.

OTOH, if SM2 or other apps aren't ready and can't realistically patch
the core to do what they want, then they are out of luck for now, and
there's no way Firefox 3 should block. We've done this many times
over, with other apps tracking the first patch release, if not the
second or later.

Again the issue in this bug is concerning to me because it makes for a
platform disparity in XUL that might bite enough add-ons that we'd
make it a blocker. If it has no effect on Firefox 3, only on other XUL
apps, then it can be blocking1.9.0.1 or wanted-next or whatever the
urgent but post-Firefox-3 flag is.

> It'd help immensely if everyone (drivers, developers, other projects,
> etc) was one the same page as to what "Gecko 1.9" (or Mozilla 1.9,
> since you seem to prefer that ;) ) is and if the blocking flag should
> represent it or Firefox 3. My impression was that it represents the
> core technology; one that will soon "freeze" and only accept
> reasonably small-impact patches or security/stability fixes. If that's
> true, and if drivers are thinking the same thing, then we need to take
> a strong look at these patches.

It should be the case that core platform bugs are evaluated by drivers
based on owners and others (notably, consumers) making the case for a
fix, based on the platform's own design (e.g. XUL menus including
platform parity), Firefox's requirements, and other apps if they fit
in the time and risk tolerance remaining.

There's no iron law that says Firefox correctness is the only
criterion. As I wrote already, for one thing it's hard to know that we
aren't breaking an add-on or ten in addition to another XUL app. Very
close to the end of the release, it would be foolish to take any
platform change that could be worked around at low cost, but again
there's no reductive rule.

That's why since I seeded drivers in 1998, we've had front-, back-,
and pan-codebase top people on it. If we were only an app, without
deep extensibility, this wouldn't be necessary.

But then we wouldn't be who we are, and we'd be more deeply beholden
to some other platform or platforms.

This is part of the problem for downstream XUL apps whose hackers are
mostly or all front-end folks. Their ability to hack on the platform
is limited, and they would want to avoid a long-lived fork if they did
so hack.

Gathering other apps' requirements, patches, and tests over more time
than we can afford to take on Firefox 3 + our best shot at Gecko for
Mozilla 1.9 (the name for the whole enchilada) and feeding it early
into the next major milestone is the best we can do.

This still won't necessarily give bz more free time to review bfcache
patches on a schedule he didn't agree to, but it will give him more
aggregate free time. He'll get to it, or we will replace the bad old
code based on requirements and tests anyway (a lot of this code causes
Firefox stress too).

/be

Mark Finkle

unread,
Mar 2, 2008, 9:47:53 PM3/2/08
to
On Mar 2, 8:34 pm, "John J. Barton" <johnjbar...@johnjbarton.com>
wrote:

A good starting point:
http://developer.mozilla.org/en/docs/Mozilla_automated_testing

Justin Wood (Callek)

unread,
Mar 2, 2008, 10:17:16 PM3/2/08
to
Boris Zbarsky wrote:

> Mike Shaver wrote:
>> do you think it's reasonable to ask that dependent projects,
>> in cases where they aren't able or willing to fix the core bugs that
>> affect their apps themselves, contribute test cases that protect the
>> behaviour they need, which may not be exercised by Firefox as primary
>> test vehicle?
>
> That seems perfectly reasonable to me, for what it's worth. I'd be very
> much in favor of it, as a Core developer. I can't speak for any of the
> non-Firefox project owners, of course.

In most cases, I would personally be glad to do this as well (especially
if I discover/file the bug). In some cases, in order to write the test,
I may need to gather some guidance from people like you (bz, shaver,
etc.) for areas of code I am shallowly familiar with. But if taking
time to write a test(s) greatly improves chances to get these types of
bugs fixed, it is definately worth the time, imo.

--
~Justin Wood (Callek)

sayrer

unread,
Mar 2, 2008, 10:50:52 PM3/2/08
to
On Mar 2, 2:02 pm, Samuel Sidler <samuel.sid...@gmail.com> wrote:
>
> That sounds like a good idea to me.

I don't care what you think, and no else does either.

- Rob

David E. Ross

unread,
Mar 3, 2008, 1:43:10 AM3/3/08
to
On 3/1/2008 4:40 PM, Samuel Sidler wrote:
> (Cross-posting to dev-planning. Follow-ups to dev-platform.)
>
> As we get closer to release, I'm seeing more and more core bugs
> minused for "blocking1.9". Often time, this is good as we've plussed
> many things that aren't nearly major enough to fix. However, many
> times, these are clear regressions that simply don't affect Firefox.
> As a Camino contributor in my off time, this brings up a pretty
> critical question to me, personally: Are we considering Gecko 1.9 a
> Firefox-only release?
>
> Previous conversations between Damon and other Camino contributors (on
> IRC a couple/few months ago) implied that we're indeed considering
> Gecko 1.9 a proper Gecko release and giving other mozilla.org apps [1]
> that rely on our codebase the same amount of core support they've
> received in the past. I.e., not shipping with regressions that were
> caused from changes made for Firefox and/or providing a clear way for
> them to move forward to a new, supported way of implementing their
> same features without hacking around bad changes.
>
> However, I've seen drivers minus bugs and leave comments that counter
> that idea. For example: "We certainly aren't going to block the
> Firefox release on an issue that doesn't affect Firefox." [2] Fixing
> such bugs later, on the actual timeline of another product's release
> isn't always possible (changing major core components to fix other
> products doesn't fly in point releases like 1.9.0.x given that those
> changes usually can affect Firefox) and/or ends up creating huge hacks
> in the codebase.
>
> In the past, we've blocked on bugs in Gecko releases that didn't
> affect Firefox but were clearly regressions from Firefox-focused
> checkins. If this has changed, I think it needs to be announced
> somewhere, very publicly, with a clear statement as to why we're
> making that change in policy (and if it wasn't a policy, feel free to
> substitute "behavior" there).
>
> I understand our need to release Firefox RSN, but not fixing core bugs
> we've caused simply because they don't affect Firefox doesn't seem
> like a good thing for our developer community and only perpetuates
> ideas that Mozilla doesn't care about other projects.
>
> (At no point do I want to demean the work some developers have done to
> care about regressions from their checkins that only happen in other
> products. There are several developers who not only care, but go out
> of their way to fix those regressions. I also am well aware of non-
> Firefox bugs getting fixed in the core and approved for landing. I'm
> specifically talking about regressions that have been minussed where
> we've broken other products and caused regressions in our platform.)
>
> If I'm way off base, please let me know, but this is the general
> sentiment I've been getting from the bugs I watch.
>
> -Sam
>
> [1] http://www.mozilla.org/projects/#applications
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=382138#c30

I reviewed the comments in Reference [2] and also the thread "Regression
handling" in this newsgroup. The real question is: "Which is more
important, the product schedule or the product quality?"

I am a retired software test engineer and configuration manager, who
worked for an independent test company, quite separate from the
companies that developed software. I experienced numerous developers
who insisted the schedule was all-important. They often felt that their
employment depended on meeting the schedule. Thus, they too often
delivered garbage to me. Several times, I sent the product back to the
developer for more work and also informed the end-user of the problem.
In this case, the software operated military space satellites; and the
end-user -- the customer -- was the U.S. Air Force, which was paying for
both the software and our testing. Both the USAF and my company
recognized that product quality had priority over product schedule.
After all, we could not afford to lose a multi-million dollar satellite
as the result of a software bug. In a few cases, the launches of new
satellites were delayed to ensure critical bugs were corrected,
schedules notwithstanding. Contractors that developed software quickly
learned that, if they wanted contracts for additional software, quality
superseded schedule.

No, Firefox is not a key component in operating space satellites. But
it is a key component in the continued existence of the Mozilla
Foundation and Mozilla Corporation. I would thus hope that regression
bugs are indeed corrected and not redesignated as "features" or swept
under the rug merely to meet a schedule. The latter course could put
the existence of the Mozilla organizations at risk. I would also hope
that the concept "Gecko is Gecko" is not put at risk by concentrating
only on Firefox when judging the severity of Gecko bugs.

--
David E. Ross
<http://www.rossde.com/>

Natural foods can be harmful: Look at all the
people who die of natural causes.

Ted Mielczarek

unread,
Mar 3, 2008, 7:08:30 AM3/3/08
to
On Mar 2, 8:34 pm, "John J. Barton" <johnjbar...@johnjbarton.com>
wrote:
> Is there somewhere that explains what automation mechanism/UI testing
> mechanism you/Mozilla uses?
>
> If this tool could be made end-user friendly then at least some
> reporters could propose better tests.

http://developer.mozilla.org/en/docs/Mozilla_automated_testing

We have several automated test suites. The biggest problem right now
is that you pretty much need to have a full source tree to develop
tests for any of them. I proposed a Seneca student project (based on
Ray's post here: http://xoatlicue.blogspot.com/2008/01/standalone-test-product.html)
to develop standalone packages for them to make test development
easier:
http://zenit.senecac.on.ca/wiki/index.php/Potential_Projects#Standalone_Test_Harnesses

Since then, I did in fact package reftest as an extension:
http://ted.mielczarek.org/code/mozilla/reftest.xpi (no real
documentation to speak of, see the reftest docs), but MochiTest and
everything else are still tough to develop tests for.

-Ted

Mike Shaver

unread,
Mar 3, 2008, 10:45:24 AM3/3/08
to Samuel Sidler, dev-pl...@lists.mozilla.org
On Sun, Mar 2, 2008 at 5:22 PM, Samuel Sidler <samuel...@gmail.com> wrote:
> > We now have much richer test suite infrastructure than we've ever had
> > before; do you think it's reasonable to ask that dependent projects,
> > in cases where they aren't able or willing to fix the core bugs that
> > affect their apps themselves, contribute test cases that protect the
> > behaviour they need, which may not be exercised by Firefox as primary
> > test vehicle?
>
> Most Camino contributors are Obj-C writers and don't speak Gecko at
> all (or very little). This is a bad thing when you think about
> Mozilla, but because of the way Camino's written, it's a good thing
> for attracting more Mac developers to the project.
>
> We're small. Really small. Writing testcases will benefit the platform
> and us in the long run, but it takes time. Time that most developers
> would rather spend fixing bugs and working on new features.

Sure, everything takes time, and some developers would rather spend
their time working on bugs that block Fx3 rather than ones that only
affect other projects. My argument, and yours too I think, is that
people should care about the shared core beyond the effect on their
pet project, especially where the effects of that caring actually
_helps_ their pet project in the (not very) long run.

Very many of our test-lacking Cocoa widget bugs are Obj-C bugs, so I
don't see why a skilled Camino Cocoa hacker couldn't give us good unit
tests or a unit test framework for it.

> That being
> said, many contributors have done an incredible job of linking to
> appropriate testcases in bugs throughout this development cycle. Those
> weren't automated testcases, but making them automated isn't nearly as
> hard when manual ones exist.

Then why not go the rest of the way?

> I kind of think this is something the
> patch writers of various bugs should be working on, when given manual
> testcases.

I kind of think a lot of things, but I'm pointing out how I think
Camino, as an example, can help themselves and everyone else here,
rather than just playing us-vs-them with the people who maintain the
core code on which they depend. I don't think Camino _owes_ Gecko
tests, but I also don't think that Gecko _owes_ Camino fixes for these
regressions. I think there's

"I shouldn't have to do that" isn't even very satisfying to read, and
we get a ton of mileage from people who run tests and write tests and
review patches and triage bugs and fix bugs and document changes and
help people even when others "should be" doing those things. What
_should_ be, foremost, is that the important behaviours of our code
are captured in tests, so that people don't have to remember or rely
on manual test protocols when they make changes that could affect
those behaviours. (Which, in Gecko's Labyrinthine halls, could be
quite a distance away indeed.) If you can help with that (you
specifically, Sam Sidler, Camino contributor), then why don't you?
"I'd rather work on something else" isn't sustainable if everyone does
it, obviously.

The need to contribute to, and care for, the overall health of the
Mozilla project's assets isn't unique to Firefox, or to MoCo
employees. MoCo's resources -- as derived largely from Firefox's
success, so be careful when complaining about the care shown for the
golden goose -- outstrip those of Camino, and those resources are
necessarily brought to bear on problems that would likely remain
utterly unsolved for all projects otherwise. But helping projects
that aren't willing to help themselves or contribute as they can to
the shared commons is bad economics, and I hope that we continue to
press for more involvement than just "consume Gecko" from other
projects that wish to be treated as part of the family when their pet
bugs are concerned. I'm not accusing Camino of this, to be very
clear, but I do think that "well, we _shouldn't have to_ write
automated tests, and we'd rather work on things in _Camino_, see" is
exactly the sort of position that people are saying is evil when held
by Firefox-focused contributors or release drivers, so I'm happy in my
belief that you're not actually espousing it. :)

Mike

Robert Kaiser

unread,
Mar 3, 2008, 11:18:25 AM3/3/08
to
Mike Shaver wrote:
> Sure, everything takes time, and some developers would rather spend
> their time working on bugs that block Fx3 rather than ones that only
> affect other projects. My argument, and yours too I think, is that
> people should care about the shared core beyond the effect on their
> pet project, especially where the effects of that caring actually
> _helps_ their pet project in the (not very) long run.

Good to see major people in the project who are on the same side on
that. Our platform is cool because so many products can base on it.
Actually, for Firefox alone, the platform would be inefficient because
it supports doing much more than what FF needs. :)

> The need to contribute to, and care for, the overall health of the
> Mozilla project's assets isn't unique to Firefox, or to MoCo
> employees.

Right. Maybe it's just the fact that patches that Firefox needs are much
easier to push into the platform than patches which seem to be only
needed by other apps that repeatedly frustrates devs of other applications.

Robert Kaiser

bre...@mozilla.org

unread,
Mar 3, 2008, 1:55:54 PM3/3/08
to
On Mar 3, 8:18 am, Robert Kaiser <ka...@kairo.at> wrote:
> Good to see major people in the project who are on the same side on
> that. Our platform is cool because so many products can base on it.
> Actually, for Firefox alone, the platform would be inefficient because
> it supports doing much more than what FF needs. :)

But not than what we need next year, or an add-on needed last year.
Not more than what we needed but didn't know we needed. This is non-
trivial. If we ran a Firefox-only platform we would be a lot more like
certain browsers I don't think anyone wants to resemble (I don't mean
IE here).

> Right. Maybe it's just the fact that patches that Firefox needs are much
> easier to push into the platform than patches which seem to be only
> needed by other apps that repeatedly frustrates devs of other applications.

No free lunch. Step up.

/be

Robert Kaiser

unread,
Mar 3, 2008, 2:21:32 PM3/3/08
to
bre...@mozilla.org wrote:
>> Right. Maybe it's just the fact that patches that Firefox needs are much
>> easier to push into the platform than patches which seem to be only
>> needed by other apps that repeatedly frustrates devs of other applications.
>
> No free lunch. Step up.

How is it "free lunch" if we do patches and try to push them into the
platform, and see we have a much harder time to do that than people who
do similar fixes but magically need them for Firefox?

Robert Kaiser

bre...@mozilla.org

unread,
Mar 3, 2008, 3:43:14 PM3/3/08
to
On Mar 2, 5:43 pm, "bren...@mozilla.org" <bren...@mozilla.org> wrote:
> Not sure when that was, but I've been promoting the 3.x for focused
> and compatibly added .x, without turning into a big all-hands-on-deck
> massive trunk continuous integration exercise, for a while.

That was it. A 1.10 milestone after 1.9 was what I opposed, and still
do. I also am in favor of getting to Mozilla 2 as quickly as possible,
by not freezing APIs again, simply breaking C++ API/ABI compat and
fixing XPCOM's memory management for good (XPCOMGC). Whatever we do,
another 1.9-like project-wide major cycle in the old style, on
cvs.mozilla.org, without working smarter not harder on core
architecture problems that get in our way, is a bad idea.

/be

bre...@mozilla.org

unread,
Mar 3, 2008, 3:46:59 PM3/3/08
to
On Mar 3, 11:21 am, Robert Kaiser <ka...@kairo.at> wrote:

Boris already replied here

http://groups.google.com/group/mozilla.dev.platform/msg/949dfd21e982bd32?dmode=source

and here

http://groups.google.com/group/mozilla.dev.platform/msg/72b01a486ad94f1d

(probably elsewhere, but I can't find the message I'm thinking of)
suggest to me that you are not facing unfair delays in review. The
code that I understand is involved is a fragile, ugly mess (docshell/
shist/bfcache). To expect faster reviews from bz, probably the only
person who can review patches for this code, is unreasonable. It's not
like Firefox has tried to take a bath messing with this code since
1.5, modulo fixes, and we almost drowned then.

A free lunch is when you think a patch should go in without sufficient
accompanying tests and code review. If we've erred in the past here
(bfcache itself landing was a case of deferred review, I approved it,
and I've learned my lesson), that's no reason to keep on doing the
same wrong thing (the definition of neurosis, btw).

If you find other reviewers are unaccountably slow, please mail me.

/be

Al Billings

unread,
Mar 3, 2008, 4:17:05 PM3/3/08
to

Is this kind of commentary either constructive or necessary? What does
it add to this conversation (it *is* a conversation in this thread).

You've already made clear that you don't highly regard Sam's opinions.
This kind of thing just seems petty to me and beneath people here.

Al
abil...@mozilla.com

Dan Mosedale

unread,
Mar 3, 2008, 7:14:56 PM3/3/08
to
Mike Shaver wrote:
>
> Sure, everything takes time, and some developers would rather spend
> their time working on bugs that block Fx3 rather than ones that only
> affect other projects. My argument, and yours too I think, is that
> people should care about the shared core beyond the effect on their
> pet project, especially where the effects of that caring actually
> _helps_ their pet project in the (not very) long run.

Agreed; the economics of this are pretty clear cut, it seems to me.
We would probably do well to at least roll some verbiage into our
"getting involved" and "hacking" documentation discussing that this is a
worthwhile tack if a bug you care about doesn't seem to be getting
enough traction.

Dan

Robert Kaiser

unread,
Mar 3, 2008, 9:28:00 PM3/3/08
to

I think we probably can all agree on that. What I wanted to propose with
1.9.1 is a very much controlled update to 1.9, which could take a small
collection of patches that are probably too risky to put into a Firefox
that is frozen in a state for release as it is now but are a big help
for other apps - but still take those in a controlled manner, with
thorough automated and manual testing.
In that way, Firefox can do a number of the usual security and stability
releases before we are in a state with that a-little-more-risky 1.9.1
tree and then switch over to it for the rest of the security/stability
update cycle, shipping the same platform code as the other apps.
We might apply bsmedberg's rules for that 1.9.1 tree, so that we don't
break the usual XUL-only extensions when FF switches over to that tree,
and we could disentangle the stability phases for FF and other apps to a
certain part.

If we would all be perfectly XULRunner-based and maybe XULRunner would
even be the perfect cross-app platform we'd wish it to be, this could
all be easier, but we are not there yet, so we should try to find some
reasonable compromise we all can live well enough with.

Robert Kaiser

Dan Mosedale

unread,
Mar 3, 2008, 10:01:32 PM3/3/08
to
Karsten Düsterloh wrote:
>
>> Is that something that non-Firefox app owners are up for? Are there
>> already test cases for the various SM2-blocking bugs to let people
>> working on the core know when they've fixed them? Do we have test
>> cases that will let Gecko folk know when they've fixed whatever is
>> ailing Camino or Sunbird or Komodo?
>
> Not that I know of, especially mailnews has almost no testcases outside
> of addressbook, and even those are more prechecking interfaces or patch
> regressions.

However, a number of us working in the mailnews area are pushing hard to
change that.

Dan

Karsten Düsterloh

unread,
Mar 4, 2008, 1:35:05 PM3/4/08
to
Dan Mosedale aber hob zu reden an und schrieb:

>> Not that I know of, especially mailnews has almost no testcases outside
>> of addressbook, and even those are more prechecking interfaces or patch
>> regressions.
>
> However, a number of us working in the mailnews area are pushing hard to
> change that.

Yeah, I know, and I'm really fond of it. :)


Karsten
--
Feel free to correct my English. :)

0 new messages