Bugzilla Locations for test infrastructure bugs
Mark Banner
improving existing test infrastructure. These are currently tending to
hang around 'Mailnews Core: Build Config' and 'Thunderbird: Build Config'.
I'd like to propose that we get some defined locations for these bugs.
There is a Testing product already defined in bugzilla, but I'm not
convinced we'd fit well in there. For instance, they already have a
'Testing: Mozmill' component, so we'd need to have 'Testing: Thunderbird
Mozmill' or something.
I'm therefore proposing that we create two new components in our "own"
areas:
MailNews Core: Testing
Thunderbird: Testing
So for example, the MailNews Core component would currently deal with
fake servers, xpcshell tests in mailnews/, bloat tests. The Thunderbird
component would currently deal with mozmill and xpcshell tests in mail/.
These are components are intended for infrastructure bugs and issues
with the tests we're currently running. Bugs in the tools we're using
(e.g. mozmill) would still be located in their appropriate
product/components.
I think this will centralise our test infrastructure bugs better and
allow us to manage them more easily.
Thoughts?
Standard8
Ben Bucksch
> I'm therefore proposing that we create two new components in our "own"
> areas:
>
> MailNews Core: Testing
> Thunderbird: Testing
Agreed, exactly like that. Very needed.
Ben
Dan Mosedale
> I'm therefore proposing that we create two new components in our "own"
> areas:
>
> MailNews Core: Testing
> Thunderbird: Testing
Sounds sensible to me.
Dan
Robert Kaiser
> MailNews Core: Testing
> Thunderbird: Testing
Sounds good to me, though I wonder if we perhaps should use "Testing
Infrastructure" or such to not make people file bugs in tests themselves
there. I also wonder if SeaMonkey should have such a component as well.
Robert Kaiser
Ludovic Hirlimann
> We've got a number of bugs for new test infrastructure or bugs on
> improving existing test infrastructure. These are currently tending to
> hang around 'Mailnews Core: Build Config' and 'Thunderbird: Build Config'.
> MailNews Core: Testing
> Thunderbird: Testing
Should we also move bugs that are related to adding new test fixing
tests there ?
Ludo
--
Ludovic Hirlimann MozillaMessaging QA lead
http://www.spreadthunderbird.com/aff/79/2
Ludovic Hirlimann
Why restrict a component just to the infrastructure ? That sounds a
small component over time, while testing won't ....
Ben Bucksch
> Mark Banner wrote:
>> MailNews Core: Testing
>> Thunderbird: Testing
>
> Sounds good to me, though I wonder if we perhaps should use "Testing
> Infrastructure" or such to not make people file bugs in tests
> themselves there.
I think tests should no in there, too (unless they are created as part
of bugs of course). In many cases, the boundary is blurry, too. A lot of
tests suffer from copypasteritis, and solving that and bugs in the
copied code is a mix between bugs in tests and testsuite. So, the
boundary is blurry.
Ben
Ben Bucksch
> I think tests should no in there, too
"go", not "no", sorry.
Mark Banner
I don't see an issue with having bugs on tests in those components. I
think it could be argued either way for some bugs - whether a bug on a
test should be in the component it relates to or not.
I could certainly see intermittent failures being in those components
(due to not knowing what is the exact cause - the infrastructure, test
or code), and there's possibly some bugs where we say we need a test for
foo, but we don't know how to do that yet.
Standard8
Serge Gautherie
> I also wonder if SeaMonkey should have such a component as well.
It can't hurt to create it, even if it's less used than the two other
ones: I do vote for having it too ;-)
Dan Mosedale
And Ludo wrote:
> Why restrict a component just to the infrastructure ? That sounds a
> small component over time, while testing won't ....
So this really goes back to the higher level question: why create
components at all. Presumably, the answer is that they allow us to, in
combination with other attributes, bucket bugs into searches that
actually end up being a manageable size for one person to look at and
triage/address somewhow. Mark, Ludo, does this sound like a good
summary to you?
If that is a good summary, the next question that comes to mind is who
do we expect will or could do that management of the individual lists?
For me, the general case of who is responsible for which bugs has always
felt very ill-defined. For this particular specific case, though, it's
easy enough to ask: Mark, did you have someone or some specific process
in mind?
Dan
Ludovic Hirlimann
> So this really goes back to the higher level question: why create
> components at all. Presumably, the answer is that they allow us to, in
> combination with other attributes, bucket bugs into searches that
> actually end up being a manageable size for one person to look at and
> triage/address somewhow. Mark, Ludo, does this sound like a good
> summary to you?
Yeah I agree. It's buckets. Most of the time in the end what counts is
getting the attention and interest of the developers so the issue get's
fixed. Buckets are sometimes easier to search in especially small and
new buckets - so I see value in that.
Ludo
Mark Banner
> So this really goes back to the higher level question: why create
> components at all. Presumably, the answer is that they allow us to, in
> combination with other attributes, bucket bugs into searches that
> actually end up being a manageable size for one person to look at and
> triage/address somewhow. Mark, Ludo, does this sound like a good summary
> to you?
I think so. In this case we have various test issues that are dotted
around relating to creating test harnesses, upgrading harnesses, etc.
What I always find hard is determining where a bug should go - if I want
to file one on adding xxx capability to mozmill tests, where do I file
it? Build Config isn't necessarily quite right (though most frequently
used), and none of our other components fit into it. If I then want to
point someone to all our test harness bugs, I need to either tag them
all, or have a tracking bug.
> If that is a good summary, the next question that comes to mind is who
> do we expect will or could do that management of the individual lists?
> For me, the general case of who is responsible for which bugs has always
> felt very ill-defined. For this particular specific case, though, it's
> easy enough to ask: Mark, did you have someone or some specific process
> in mind?
In the general case, I'd expect module-owners to manage buckets (to some
extent) - but that's a different discussion.
For the Testing bucket, up until now I guess I've been leading most of
the testing work. I'd like to encourage more work on test harness via
student-project or other means. I could see a role for a dedicated test
engineer (volunteer, whatever) but for now I'd be happy to keep an eye
on that component (I'll even do some work on it when I get time).
Mark.
Dan Mosedale
> For the Testing bucket, up until now I guess I've been leading most of
> the testing work. I'd like to encourage more work on test harness via
> student-project or other means. I could see a role for a dedicated
> test engineer (volunteer, whatever) but for now I'd be happy to keep
> an eye on that component (I'll even do some work on it when I get time).
high-order bit here, whereas making it easier to file these sorts of
bugs doesn't seem quite as important. So if you, as effective module
owner here, believe adding the component it will be significantly
helpful to you in keeping them triaged/driven/managed, I'd say go for it.
Dan
Phil Ringnalda
> So this really goes back to the higher level question: why create
> components at all. Presumably, the answer is that they allow us to, in
> combination with other attributes, bucket bugs into searches that
> actually end up being a manageable size for one person to look at and
> triage/address somewhow. Mark, Ludo, does this sound like a good summary
> to you?
There were times in my career when I cared about components for
searches, but I've pretty much gotten over that. The only time I think
about them for searches (other than "I'm assigned to triage noms in this
set of bugs" searches) is when I'm totally desperate to find something
that I know exists, that no keywords will find, by looking at every
single open bug in the component I'm sure it must be in.
The one thing that matters to me about a component is that it produces a
watchable qa contact. If there are, or even are likely to be, people who
want to watch all Foo bugs, but don't want to watch all of Thunderbird
and Mailnews Core to get at all of them, then it's a good component, and
in the other direction, if leaving the Foo bugs where they are will
drive someone away from watching that (say, a core build-config person
like Ted who might be willing to watch Tb: Build Config if it was build
config but not if it was build config plus fakeserver plus MozMill, or
someone who watches Compose because they're interesting in editing, but
not if they have to watch us fumble to get MozMill working with
compose), then it's also a good component. Where it becomes a bad
component is if you have bugs about testing IMAP, that would benefit
from people who watch the IMAP component because they know about the
protocol, hidden away in a testing component. That, I think, is the
reason and the benefit to Core/Fx/Toolkit having testing components, but
having bugs in particular tests go in components for the code that is
being tested, rather than in testing components.
Robert Kaiser
Mark, when you request those new components, can you please request that
one as well?
Thanks
Robert Kaiser
Robert Kaiser
OK, understood.
I think bugs on testing specific functionality should be in the
component for whatever it's actually testing, but I see that sometimes
borders are blurry there...
Robert Kaiser
Mark Banner
> I'm therefore proposing that we create two new components in our "own"
> areas:
>
> MailNews Core: Testing
> Thunderbird: Testing
I'm just revisiting this before filing a bug. I think Phil's and
Robert's comments are especially significant - bugs with specific tests
should be filed in the components they are relevant to. Therefore I'm
going to go with Robert's suggestion of:
MailNews Core: Testing Infrastructure
Thunderbird: Testing Infrastructure
SeaMonkey: Testing Infrastructure
The main aim of these being to make it easier to collate and manage bugs
on our test infrastructure. Typical examples: improvements to mozmill
tests, improvements to fake servers, new testing infrastructure (e.g.
performance tests).
I'll file a bug sometime in the next day or so to get these created.
Standard8