Location of new crash-test tests
Jonas Sicking
At the last leak meeting we discussed where to put the new crash-test
tests. See bug 397725 for more detailed description of the tests.
The only thing blocking us right now is deciding on a place to put these
tests. The following alternatives were suggested:
A) All tests in a single directory, for example
mozilla/testing/crashtest
B) All tests in a single directory tree, for example
mozilla/testing/crashtest/dom
mozilla/testing/crashtest/xbl
mozilla/testing/crashtest/floats
C) Where we put mochitests, so for example code in
mozilla/content/base/src would get tests in
mozilla/content/base/test
D) In directory parallel to mochitests, for example
mozilla/content/base/src would get tests in
mozilla/content/base/crashtest
E) In directory under mochitests, for example
mozilla/content/base/src would get tests in
mozilla/content/base/test/crashtest
Let the bikeshedding begin! :)
/ Jonas
Justin Dolske
> The only thing blocking us right now is deciding on a place to put these
> tests. The following alternatives were suggested:
>
> A) All tests in a single directory, for example
> mozilla/testing/crashtest
> B) All tests in a single directory tree, for example
> mozilla/testing/crashtest/dom
> mozilla/testing/crashtest/xbl
> mozilla/testing/crashtest/floats
Huh, I didn't even realize we had actual tests under /testing. I thought
the usual rule was to put tests near the code they're testing. [ie, any
of the following suggestions]
> C) Where we put mochitests, so for example code in
> mozilla/content/base/src would get tests in
> mozilla/content/base/test
> D) In directory parallel to mochitests, for example
> mozilla/content/base/src would get tests in
> mozilla/content/base/crashtest
> E) In directory under mochitests, for example
> mozilla/content/base/src would get tests in
> mozilla/content/base/test/crashtest
I'd say (D) is undesirable; better to stick testing-related things under
foo/test, lest we end up with foo/test, foo/crasttest, foo/othertest,
foo/moretest, foo/spamtest, foo/footest, foo/getmypointtest.
Either (C) or (E) would work for me.
> Let the bikeshedding begin! :)
Blue!!!
Justin
L. David Baron
> C) Where we put mochitests, so for example code in
> mozilla/content/base/src would get tests in
> mozilla/content/base/test
I'm actually leaning towards this one (and starting to think that
not doing it for reftests was a mistake).
Perhaps we should do this, along with a file-naming convention? By
a file-naming convention, I mean something like calling tests for
crashes crashtest_bug12345.html just like we call mochitests
test_bug12345.html . (We could also then use
leaktest_bug12345.html, asserttest_bug12345.html, etc., and
potentially switch over to putting reftests in
reftest_bug12345.html.)
-David
--
L. David Baron http://dbaron.org/
Mozilla Corporation http://www.mozilla.com/
Dan Mosedale
> I'm actually leaning towards this one (and starting to think that
> not doing it for reftests was a mistake).
>
> Perhaps we should do this, along with a file-naming convention? By
> a file-naming convention, I mean something like calling tests for
> crashes crashtest_bug12345.html just like we call mochitests
> test_bug12345.html.
That sounds like a good plan to me.
Dan
Robert O'Callahan
> On Friday 2007-10-19 18:24 -0700, Jonas Sicking wrote:
>
> > C) Where we put mochitests, so for example code in
> > mozilla/content/base/src would get tests in
> > mozilla/content/base/test
>
> I'm actually leaning towards this one (and starting to think that
> not doing it for reftests was a mistake).
>
> Perhaps we should do this, along with a file-naming convention? By
> a file-naming convention, I mean something like calling tests for
> crashes crashtest_bug12345.html just like we call mochitests
> test_bug12345.html . (We could also then use
> leaktest_bug12345.html, asserttest_bug12345.html, etc., and
> potentially switch over to putting reftests in
> reftest_bug12345.html.)
Sounds good to me.
Rob
Jonathan Watt
> On Friday 2007-10-19 18:24 -0700, Jonas Sicking wrote:
>> C) Where we put mochitests, so for example code in
>> mozilla/content/base/src would get tests in
>> mozilla/content/base/test
>
> I'm actually leaning towards this one (and starting to think that
> not doing it for reftests was a mistake).
>
> Perhaps we should do this, along with a file-naming convention? By
> a file-naming convention, I mean something like calling tests for
> crashes crashtest_bug12345.html just like we call mochitests
> test_bug12345.html . (We could also then use
> leaktest_bug12345.html, asserttest_bug12345.html, etc., and
> potentially switch over to putting reftests in
> reftest_bug12345.html.)
If all tests are going to be prefixed with a strings like "leaktest_",
"asserttest_", etc., you might as well create subdirectories in the "test"
directory called "leaktest/", "asserttest/", etc. That's neater IMO. A lot of
tests (e.g. the replaced element tests I've been writing) already have names
that are quite long.
Jonathan
Jeff Walden
> L. David Baron wrote:
>> On Friday 2007-10-19 18:24 -0700, Jonas Sicking wrote:
>>> C) Where we put mochitests, so for example code in
>>> mozilla/content/base/src would get tests in
>>> mozilla/content/base/test
>>
>
> "asserttest_", etc., you might as well create subdirectories in the "test"
> directory called "leaktest/", "asserttest/", etc. That's neater IMO.
Decentralizing is good, so I prefer not keeping all tests in the same directory. I'd also go for subdirectories over prefixes or a naming convention -- less visual clutter when reading file listings, and easier to type as well. reftest got this right with test manifests, and I sort of think Mochitest should switch to doing that as well.
> A lot of tests (e.g. the replaced element tests I've been writing)
> already have names that are quite long.
Yeah, I think the reftest setup encourages long names to make what's tested more obvious (ignoring the case of fixing an isolated regression/bug -- I'm thinking more of testing a specific feature in many different ways, for example). More descriptive names make it easier to find a specific test, and they make it more obvious what's tested and what's not tested.
Jeff
Robert Sayre
> Jonathan Watt wrote:
>> L. David Baron wrote:
>>> On Friday 2007-10-19 18:24 -0700, Jonas Sicking wrote:
>>>> C) Where we put mochitests, so for example code in
>>>> mozilla/content/base/src would get tests in
>>>> mozilla/content/base/test
>>>
>>> Perhaps we should do this, along with a file-naming convention?
>>
>> If all tests are going to be prefixed with a strings like "leaktest_",
>> "asserttest_", etc., you might as well create subdirectories in the
>> "test"
>> directory called "leaktest/", "asserttest/", etc. That's neater IMO.
>
> Decentralizing is good, so I prefer not keeping all tests in the same
> directory. I'd also go for subdirectories over prefixes or a naming
> convention -- less visual clutter when reading file listings, and easier
> to type as well. reftest got this right with test manifests, and I sort
> of think Mochitest should switch to doing that as well.
Looks like we're using a reftest-based harness, so we'll need a root
manifest. Regarding the rightness of manifests in general, I don't think
it matters. Either you use a naming convention, or you slowly duplicate
the functionality of make.
I don't care which system we use.
- Rob
Jeff Walden
> Looks like we're using a reftest-based harness, so we'll need a root
> manifest. Regarding the rightness of manifests in general, I don't think
> it matters. Either you use a naming convention, or you slowly duplicate
> the functionality of make.
Getting a little off-topic, but I think my beef is that Mochitest is doing both -- copying rules in Makefiles are effectively the manifest, and prefixes are the naming convention. In any case, there are lots more interesting things to care about doing, so I doubt this will change any time soon.
Jeff