Selenium Chapter Listings
Scott
One of my current side projects is updating the Selenium driver
listings found here:
http://code.google.com/p/ubiquity-xforms/source/browse/trunk/testsuite/W3C-XForms-1.1/Edition1/driverPages/SeleniumTests/
These pages are what you see in the left most column when executing
tests with the manual Selenium interface. The problem is that while
the test suite has been continuously updated, these pages have fallen
out of date. So with that said, I propose amending the updating
process to include regenerating these files. While doing this in
itself is a trivial process, the real issue is that some of the tests
have been manually flagged within the various listings. These flags
include the unless="true" attribute on <tr> tags and a red styling on
some of the links. From what I understand, the red style is supposed
to mean that the test is incapable of being automated, while the
unless="true" attribute removes the test from the queue. Could someone
explain why there are two systems at play? I see no reason not to
amalgamate the two under the unless="true" flag, as if the test can't
be automated then there is no sense in having it executed by Selenium.
Furthermore, another problem is that the individual chapter listings
and the all encompassing listing are not in sync; the same test is
marked differently in the chapter listing as the total listing. What
I'd like to do is abstract this information to a separate XML file to
make maintenance easier, however this begs the question, which
listing's flags are correct? I'd like to avoid manually verifying
these flags if possible as there are 60-70 tests that would need to be
reviewed.
So in short, I suppose my query comes down to this: should I preserve
the red styling separate from the unless="true" flag, and secondly,
should I use the flags in the chapter listings, or the
W3CTestSuite.html file, or some combination of the two? Any guidance
on these issues would be appreciated.
Thanks,
Scott
Mark Birbeck
> One of my current side projects is updating the Selenium driver
> listings found here:
> http://code.google.com/p/ubiquity-xforms/source/browse/trunk/testsuite/W3C-XForms-1.1/Edition1/driverPages/SeleniumTests/
Great! :)
> These pages are what you see in the left most column when executing
> tests with the manual Selenium interface. The problem is that while
> the test suite has been continuously updated, these pages have fallen
> out of date. So with that said, I propose amending the updating
> process to include regenerating these files.
Excellent.
If you haven't seen it already, can I direct you to a new build.xml
file that Phil has created in the root directory? The plan is to have
all tools, processes, conversions, etc., driven by this one file. At
the moment we have a number of them scattered around various
directories, but it makes it difficult to share paths and other
information between them (not to mention difficult to find them). So
we're moving them up to the top, and deleting as we go.
Also, if the associated XSLT files can go in the new 'tools' directory.
> While doing this in
> itself is a trivial process, the real issue is that some of the tests
> have been manually flagged within the various listings. These flags
> include the unless="true" attribute on <tr> tags and a red styling on
> some of the links. From what I understand, the red style is supposed
> to mean that the test is incapable of being automated, while the
> unless="true" attribute removes the test from the queue. Could someone
> explain why there are two systems at play? I see no reason not to
> amalgamate the two under the unless="true" flag, as if the test can't
> be automated then there is no sense in having it executed by Selenium.
Difficult to answer this politely. :)
I think the person who added the red styling only ever saw the tests
as being run manually. (The fact that many tests rely on messages is a
similar reflection of the same thing.) But you are right that the
styling can be dropped, and the @unless="true" can be used instead.
I'm sure it's obvious, but there are some tests that will be never be
'automatable' -- the email tests, for example, open local machine's
email application, which cannot be tested for in Selenium -- and then
there are tests that are not currently automated, but could be.
Obviously Selenium won't distinguish the two, and they'll both have
@unless="true", but your XSLT will no doubt be able to tell the
difference.
One last thing, again you probably know this, but if not, the string
in @unless is executed as JavaScript. So if you wanted to have a test
that checked the browser type or something, you can.
(Alternatively, we could generate a set of tests for each browser if
we wanted to, with different tests marked for skipping. That's
something to ponder.)
> Furthermore, another problem is that the individual chapter listings
> and the all encompassing listing are not in sync; the same test is
> marked differently in the chapter listing as the total listing. What
> I'd like to do is abstract this information to a separate XML file to
> make maintenance easier, however this begs the question, which
> listing's flags are correct? I'd like to avoid manually verifying
> these flags if possible as there are 60-70 tests that would need to be
> reviewed.
We also arrived at this point a few months ago. The problem is that
the W3C test suite has been issued as an amalgam of test descriptions
and individual processor results. There is actually a W3C test format
that simply describes a test with its expected result, but it hasn't
been adopted here.
> So in short, I suppose my query comes down to this: should I preserve
> the red styling separate from the unless="true" flag...
I agree with you that it doesn't add anything, so feel free to remove
the styling.
> ... and secondly,
> should I use the flags in the chapter listings, or the
> W3CTestSuite.html file, or some combination of the two? Any guidance
> on these issues would be appreciated.
On that I'm afraid I'm not sure...sorry. It probably doesn't matter
that much, but I'd prefer John to decide that one, being as he's the
chair of the W3C working group, and all that. ;)
Regards,
Mark
--
Mark Birbeck, webBackplane
mark.b...@webBackplane.com
http://webBackplane.com/mark-birbeck
webBackplane is a trading name of Backplane Ltd. (company number
05972288, registered office: 2nd Floor, 69/85 Tabernacle Street,
London, EC2A 4RR)
John Boyer
Hi Mark and Scott,
Regarding the last question, I think it is not a W3C group question. I believe Scott is indicating that the "unless" flag settings are different between the chapter-by-chapter files versus the full test suite file.
It is exactly because this confusion has arisen that we need a more descriptive name than unless="true". This flag seems to indicate that the W3C test should not be run by the automated framework. There are at least two possible reasons for setting this flag, which is why, I think, the settings got out o synch. First, there are those tests that will always have to be run manually, i.e. there is no way to automate. Second, there are those tests that were removed at times because some fault in ubiquity prevented the automated system from making a complete run. We don't know how many of these there are, but they should be indicated differently so that we can eventually get to fixing whatever is preventing their automation. A third setting, which perhaps has not arisen yet, is the set of tests which must be manual now but which could be automated if we used a different test of the same feature. This is different from the second case because it claims that we can automate in the future, but only by proposing a revised test rather than by revising the ubiquity implementation (or our own test automation scripts).
So, could consider changing the attribute name and/or attribute values to reflect these possibilities? This could be something like manual="always|revise-test|fix-bug". Then, could you or Phil provide info on which tests should receive the manual="always" setting? Finally, for every test marked manual with a setting other than "always", we need an issue filed so that someone will eventually get around to revising the implementation or the test.
John M. Boyer, Ph.D.
STSM, Interactive Documents and Web 2.0 Applications
Chair, W3C Forms Working Group
Workplace, Portal and Collaboration Software
IBM Victoria Software Lab
E-Mail: boy...@ca.ibm.com
Blog: http://www.ibm.com/developerworks/blogs/page/JohnBoyer
Blog RSS feed: http://www.ibm.com/developerworks/blogs/rss/JohnBoyer?flavor=rssdw
| From: | Mark Birbeck <mark.b...@webbackplane.com> |
| To: | ubiquity-...@googlegroups.com |
| Date: | 06/29/2009 03:22 AM |
| Subject: | Re: Selenium Chapter Listings |
Mark Birbeck
Ah...sorry. I took this bit to be more important than it was:
>> Furthermore, another problem is that the individual chapter listings
>> and the all encompassing listing are not in sync...
I thought Scott was implying that the titles, descriptions, and so on,
were out of sync.
Sorry. :)
So on Scott's question, I hope that the individual chapters will be
more up-to-date -- they are certainly the only ones that I have
touched when adding @unless values.
But if anyone else has added @unless values in the couple of months
since our code sprint, they could quite possibly have added them to
the old 'monolithic' file. I can only suggest checking with SVN
history on that, I'm afraid.
Phil Booth
I've got couple of thoughts on this thread, not sure what others make
of them. In the past, when adding @unless="true", I've also raised a
bug in the issue tracker detailing what has been done and why (issue
503 [1], which I raised earlier today, is an example of this). I think
this is a better place to record that kind of information than
encoding it in the attribute value, it also gives scope for discussion
to take place about individual test cases.
Phil.
[1] http://code.google.com/p/ubiquity-xforms/issues/detail?id=503
2009/6/29 Mark Birbeck <mark.b...@webbackplane.com>:
Phil Booth, webBackplane
phil....@webbackplane.com
http://webbackplane.com/phil-booth
John Boyer
Hi Phil,
A benefit to encoding the different possibilities in the attributes is that it is more clearly documented. Right now, with issues, the information flow is only one way, from the issue list projected onto the attributes. So there is no way to know whether each setting of unless="true" was considered permanent or temporary, and if temporary what will allow it to change to unless="false" (implementation change, script creation or change, test suite revision).
Having the additional states amounts to the attributes projecting information to the issue list. Specifically, it would fix the problem of having to "hope" that a manual test with no corresponding issue meant a permanently manual test. If I see a test with manual="revise-test" or some such setting, and I found no issue, then I would know there is an omission from the issue list. If I see unless="true" and no issue, I'm not sure if there is a gap in the issue list or not.
John M. Boyer, Ph.D.
STSM, Interactive Documents and Web 2.0 Applications
Chair, W3C Forms Working Group
Workplace, Portal and Collaboration Software
IBM Victoria Software Lab
E-Mail: boy...@ca.ibm.com
Blog: http://www.ibm.com/developerworks/blogs/page/JohnBoyer
Blog RSS feed: http://www.ibm.com/developerworks/blogs/rss/JohnBoyer?flavor=rssdw
| From: | Phil Booth <phil....@webbackplane.com> |
| To: | ubiquity-...@googlegroups.com |
| Date: | 06/29/2009 07:56 AM |
| Subject: | Re: Selenium Chapter Listings |
Mark Birbeck
I'm not sure whether we're all talking across each other here. :)
So, first, are you talking about the @unless attribute that is used in
the driver files? If so, I think we should leave the name of that
attribute as it is, as well as its usage -- a simple JavaScript
expression that evaluates to true or false.
However, I do think that these files should be generated using XSLT,
and then the value of @unless could be set based on some other
criteria, stored elsewhere.
This is the approach I was talking about a few months ago on a
telecon; the idea was to be able to generate different driver pages
for different circumstances.
For example, a test of UXF conformance requires us to run every test,
whilst a test for 'breaking the build' would only require you to run
those tests that have been known to pass at some point.
XSLT could give you a handful of separate files for each of these
conditions; for conformance it could generate two lots of tests, one
for Selenium (that includes all tests that can be run unsupervised),
and one listing tests that require manual intervention. For Buildbot
it could just give a list of files that have previously passed and are
known to be automated.
At the time of that conversation, you were primarily concerned to get
the conformance results generated, and asked that we didn't take this
approach. But if Scott is working on this, I'd suggest that he does
move in this direction.
So, how might this be done?
Well, at the moment we have the XML files that are actually only used
to drive the reports for each chapter, but they could be used to drive
the tests, too. Each file has multiple entries, each of which looks
like this:
<testCase>
<testCaseSection>3.3.1</testCaseSection>
<testCaseName>3.3.1.d1</testCaseName>
<testCaseLink>../../Chapt03/3.3/3.3.1/3.3.1.d1.xhtml</testCaseLink>
<testCaseDescription>model element now supports a version
attribute</testCaseDescription>
<testCaseSpecLinkName>3.3.1</testCaseSpecLinkName>
<testCaseSpecLink>http://www.w3.org/TR/xforms11/#structure-model</testCaseSpecLink>
<testCaseBasic>true</testCaseBasic>
<testCaseNormative>true</testCaseNormative>
<testCaseStatus>Passed</testCaseStatus>
<testCaseDate>2008-12-16</testCaseDate>
<testCaseRequired>true</testCaseRequired>
<testCaseNote/>
</testCase>
(Let's ignore the large amount of duplicated information in here, for now.)
We could put the 'test or skip' information in here, but I think we
should go one better than that. First, I'd suggest we factor out the
test description to a separate file (it's this information that I
thought Scott was saying was out of sync):
<testCase>
<testCaseSection>3.3.1</testCaseSection>
<testCaseName>3.3.1.d1</testCaseName>
<testCaseLink>../../Chapt03/3.3/3.3.1/3.3.1.d1.xhtml</testCaseLink>
<testCaseDescription>model element now supports a version
attribute</testCaseDescription>
<testCaseSpecLinkName>3.3.1</testCaseSpecLinkName>
<testCaseSpecLink>http://www.w3.org/TR/xforms11/#structure-model</testCaseSpecLink>
<testCaseBasic>true</testCaseBasic>
<testCaseNormative>true</testCaseNormative>
<!--
REMOVED STATUS
-->
<testCaseRequired>true</testCaseRequired>
<testCaseNote/>
</testCase>
Then we would have separate status files for each browser, looking
something like this:
<testCase>
<testCaseName>3.3.1.d1</testCaseName>
<testCaseStatus>Passed</testCaseStatus>
<testCaseDate>2008-12-16</testCaseDate>
<testCaseNote/>
</testCase>
The list of possible 'testCaseStatus' values might be expanded to
include a value that represents whether the test can be run
automatically. Or we might go the other way, and assume that 'Passed'
means it can be automated, and then add something like @mode to flag
up those tests that need manual intervention.
Either way, by combining the test description file (the first one),
and this browser-specific results file, we can generate a
browser-specific driver page. These pages could then either include
@unless="true"/"false", based on the values in these files, or we
might even decide to not bother having entries at all, for those tests
that we aren't going to run.
Having a driver page for each browser gives us much more control over
which tests should be run and which ones skipped, for a particular
browser. We can also have a driver page for those tests that require
manual supervision, as well as automatically generate driver pages for
the regression tests that Phil set up.
Also, using XSLT to read this information and generate the driver
pages, gives us more control over the evaluation of conditions that
determine whether a particular test will be run on a particular driver
page.
Regards,
Mark
John Boyer
Hi Mark,
I don't disagree with anything you said.
In particular, I quite agree that there is a longer term issue that the test suite could be both revised and reorganized for a number of reasons. Shorter term, though, I hope we can make some use of Scott's time to get better at measuring our conformance in an automated way.
The immediate issue for Scott, though, is regardless of how the "unless" information is retained going forward, there is a disconnect between the "unless" setting currently in the chapter-by-chapter files versus the "unless" settings in the full test suite file. Scott is asking which set should be used as the authoritative setting of which tests are supposed to be excluded from automatic testing. Your response indicated you believed this was a W3C working group issue and therefore needed my response. My response was that I thought Scott's question had been misunderstood because our decision to automate or not automate a particular test is not, strictly speaking, a W3C working group issue.
Regardless of how we store them, what should the current manually run tests be in order to produce an updated implementation report using the automated test framework? And are there any manual tests that *should* be automatic except that they impede completion of an automated test?
Thanks,
STSM, Interactive Documents and Web 2.0 Applications
Chair, W3C Forms Working Group
Workplace, Portal and Collaboration Software
IBM Victoria Software Lab
E-Mail: boy...@ca.ibm.com
Blog: http://www.ibm.com/developerworks/blogs/page/JohnBoyer
Blog RSS feed: http://www.ibm.com/developerworks/blogs/rss/JohnBoyer?flavor=rssdw
Mark Birbeck
Sorry...I thought I'd clarified this bit:
> The immediate issue for Scott, though, is regardless of how the "unless"
> information is retained going forward, there is a disconnect between the
> "unless" setting currently in the chapter-by-chapter files versus the
> "unless" settings in the full test suite file. Scott is asking which set
> should be used as the authoritative setting of which tests are supposed to
> be excluded from automatic testing.
I'm assuming we're talking about driver pages here, and I think the
chapter-by-chapter files are the more up-to-date. However, I think
it's worth checking via SVN that no-one has modified the monolithic
files.
> Your response indicated you believed
> this was a W3C working group issue and therefore needed my response. My
> response was that I thought Scott's question had been misunderstood because
> our decision to automate or not automate a particular test is not, strictly
> speaking, a W3C working group issue.
Right...as I said, that's because I thought Scott was talking about
the test titles/descriptions, etc. We're on the same page now, and
yes, it's nothing to do with the W3C.
> Regardless of how we store them, what should the current manually run tests
> be in order to produce an updated implementation report using the automated
> test framework?
The only tests I know of that cannot be automated in Selenium are the
email tests I did.
That doesn't mean there aren't others, but I don't know of them.
> And are there any manual tests that *should* be automatic
> except that they impede completion of an automated test?
I believe this is work that everyone is doing, not just in our team,
so it does mean that no one person has the full answer to this.
The easiest way to find out would be from issue 337, but it looks like
this hasn't been kept up to date. It wouldn't take long to sort that
out, and then you'd have an up-to-date picture, so perhaps people can
link their issues to the chapter-specific issues that come off of 337.
(I.e., by using blockers.)