What is the report syntax?

[David Abrahams]

http://bitten.edgewall.org/wiki/DataStorage has some examples, but
they're not enough to infer everything.

Among other things, it's not clear which fields are required. Some
examples on the page noted above have a <generator> and some don't.

--
Dave Abrahams
Boost Consulting
http://www.boost-consulting.com

[Jay]

I've just ran thru the exercise myself. The only soln I found was to
query on the underlying db itself (trac.db) for the bitten_report and
bitten_report_item tables. The latter is just a K/V pair, so you can
put any fields you want. Not really the way I would have done it, but
extensible certainly.

Perhaps you can help me. Even though I now save a rich set of details
on the tests, the bitten build test-view tab only returns minimal
information on how the tests went. Am I missing something?

Thanks,
-Jay

On Oct 27, 11:11 pm, David Abrahams <d...@boost-consulting.com> wrote:
> http://bitten.edgewall.org/wiki/DataStoragehas some examples, but

[David Abrahams]

on Mon Oct 29 2007, Jay <jaydamask-AT-gmail.com> wrote:

> I've just ran thru the exercise myself. The only soln I found was to
> query on the underlying db itself (trac.db) for the bitten_report and
> bitten_report_item tables. The latter is just a K/V pair, so you can
> put any fields you want. Not really the way I would have done it, but
> extensible certainly.

AFAICT, the database schema doesn't tell me anything about the
required syntax of XML that is sent to the server as a report. Am I
missing something?

> Perhaps you can help me. Even though I now save a rich set of details
> on the tests, the bitten build test-view tab only returns minimal
> information on how the tests went. Am I missing something?

I don't know; I'm only just getting my feet wet here. I do wish
cmlenz would weigh in, since he knows the answers.

[Allen Bierbaum]

This does need better documentation.  I recently had to go through this and what I ended up doing was reverse engineering the required information from the code in bitten that is used to process and display the information.  To the best of my knowledge the structure is...

The base element of the report must be "report" and have an attribute "category" that is one of "test" or "coverage".  Ex:

<report category="test|coverage">
</report>

Inside the report there must be elements for each report type.  The way the data is captured is pretty flexible because it can either be in attributes or in child elements. 

Test reports must have sub-elements of report of type "test".  These elements can have any of these attributes (or subelements with contained cdata)

duration: float duration of test
status: string "success" or "failure" or "error"
name: string name of the test
fixture: string name of the test fixture
file: path to test file relative to the base path for the build configuration
stdout: The output from the test
traceback: The traceback from any error or failure.

Example:

<report category="test">
   <test duration=" 0.073" status="success"
            fixture="bitten.tests.model.BuildConfigTestCase"
       name="test_config_update_name" file="bitten/tests/model.py"
           stdout="Renaming build configuration"
           traceback="None"
   </test>
   <test>
         <duration>0.073 </duration>
        <status>success</status>

     <fixture>bitten.tests.model.BuildConfigTestCase
</fixture>
     <name>test_config_update_name</name>
     <file>bitten/tests/model.py</file>

     <stdout>Renaming build configuration</stdout>
   </test>

</report>


Similarly, coverage reports must be inside "coverage" elements.  The allowable attributes are:

name: The name of the module being tested for coverage
file: The name of the file relative to the base path in the build configuration
percentage: The percentage of lines in that file covered
lines: The number of statements covered

Ex:

<test category="coverage">
   <coverage name="my_module"
                    file="my_module.py"
                    percentage="23.4"
                    lines="450"
    </coverage>
</test>

This is the format as to the best of my knowledge.  If anyone else has any more insights, please let me know so I can try to use them. :)

-Allen

[Christopher Lenz]

Am 30.10.2007 um 17:01 schrieb David Abrahams:
> on Mon Oct 29 2007, Jay <jaydamask-AT-gmail.com> wrote:
>> I've just ran thru the exercise myself. The only soln I found was to
>> query on the underlying db itself (trac.db) for the bitten_report and
>> bitten_report_item tables. The latter is just a K/V pair, so you can
>> put any fields you want. Not really the way I would have done it, but
>> extensible certainly.
>
> AFAICT, the database schema doesn't tell me anything about the
> required syntax of XML that is sent to the server as a report. Am I
> missing something?

Actually, the database schema maps pretty directly to the XML. The
mapping is described here:

<http://bitten.edgewall.org/wiki/DataStorage#MappingXMLReportstoDatabaseTables
>

Now, the information you need for *specific* reports is a different
question, and isn't documented at all outside of the code. It really
needs to be documented, and help with that is greatly appreciated.

Cheers,
Chris
--
Christopher Lenz
cmlenz at gmx.de
http://www.cmlenz.net/

[Christopher Lenz]

Am 30.10.2007 um 17:26 schrieb Allen Bierbaum:
> This does need better documentation.

Agreed 100%. You've put together a great starting point (all correct
AFAICT) here, can you add it to the Wiki ("ReportFormats" or something)?

Thanks,

[Allen Bierbaum]

I would, but I don't have a login to the trac/bitten wiki so I can't edit the page.  If you can tell me how to get an account I would be more then happy to add this as a starting point for further documentation.

-Allen

[Christopher Lenz]

Am 30.10.2007 um 19:14 schrieb Allen Bierbaum:
> I would, but I don't have a login to the trac/bitten wiki so I can't
> edit the page. If you can tell me how to get an account I would be
> more then happy to add this as a starting point for further
> documentation.

Which page? AFAIK most pages aren't locked down, and you can create
new pages without having an account.

Cheers,

[Allen Bierbaum]

Ah.  I was on the front page for Bitten.  The documentation page appears to be editable.  Where would you like me to put this description and where would you like me to link it from?

-Allen

On 10/30/07, Christopher Lenz <cml...@gmx.de> wrote:

[Jay]

I've now gone thru this all and build my unit tests using CxxTest.
imho the Bitten build_report_item is very sparing and we're going to
add three separate tables, one for a 'suite' level, a 'test' level,
and an 'assert' level. I've modified XmlFormatter.h (against CxxTest)
from the web to generate output attr's that are interesting (not hard)
and now have an xml format that I can load into these tables. I intend
to bypass the Bitten as-is tables. This is not a criticism but rather
to add the notion that what's in Bitten is not set in stone.

I will say that all this is a weak point of Bitten (which I otherwise
love). There should be a more rich way to import rich test results and
display them on the build page.

Cheers,
-Jay

[Christopher Lenz]

On Oct 31, 2007, at 1:58 PM, Jay wrote:
> I've now gone thru this all and build my unit tests using CxxTest.
> imho the Bitten build_report_item is very sparing and we're going to
> add three separate tables, one for a 'suite' level, a 'test' level,
> and an 'assert' level. I've modified XmlFormatter.h (against CxxTest)
>> from the web to generate output attr's that are interesting (not
>> hard)
> and now have an xml format that I can load into these tables. I intend
> to bypass the Bitten as-is tables. This is not a criticism but rather
> to add the notion that what's in Bitten is not set in stone.
>
> I will say that all this is a weak point of Bitten (which I otherwise
> love). There should be a more rich way to import rich test results and
> display them on the build page.

I agree this aspect is problematic. After I finally (cry) get 0.6 out
of the door, I'd like to revisit this whole story. The plan is vague,
but something along the lines of an added IReportDataExtractor
extension point, which would allow you to make a plugin on the Trac
side that could set up its own database tables, and store the
extracted data in those tables, as well as query those tables in the
summarizers and chart generators.

This extension point should also allow merging the data from builds of
the same revision for different target platforms. An important example
for this is the code coverage statistics when you have a code base
that follows different paths depending on platform. That's a case
that's currently handled rather badly.

It should also be possible to "compact" the database somehow, removing
detail information for old builds, to avoid the database growing ad
infinitum. So the report data extraction would probably differentiate
between data that is interesting in the long term (for example code
coverage percentage per module), and data that can safely be purged
(for example, per-line/statement coverage info).