msbuild.xml
potter
using a custom logger in msbuild.exe to generate an XML log file and
then using the x:transform command to transform this file using
msbuild.xml to another format before typing this file.
Questions:
* What does all of this buy me?
* From what I can tell it just makes things better formatted. Is
this true?
* Where can I find msbuild.xml?
* This page says it is a modified version of msbuild2ccnet.xml from
the referenced page, but how has it been modified?
* When building under a high warning level, the warning messages can
be considered "lint" warnings. Has anyone done any work at
transforming the msbuild log file into a bitten lint report?
* Has anyone done any "lint" report work for VC++ with bitten?
potter
3 or later installed.We I run a build recipe that contains the
x:transform I get "No usable XSLT implementation found" error.
According to my control panel I have "MSXML 6.0 Parser (KB933579)"
installed. What am I missing or is that the correct package?
[1] http://bitten.edgewall.org/wiki/Documentation/commands.html#x-transform
potter
Simon Cross
in question is:
<step id="Build">
<sh:exec file="echo" args="Build start"/>
<sh:exec file="msbuild.exe" args="project.sln
@C:\\Builds\\utils\\logger\\msbuild.rsp"/>
<x:transform src="msbuild-output.xml" dest="msbuild-result.xml"
stylesheet="..\\utils\\logger\\msbuild.xsl" />
<sh:exec file="type" args="msbuild-result.xml"/>
<sh:exec file="echo" args="Build complete"/>
</step>
On Thu, Feb 11, 2010 at 7:35 PM, potter <mpo...@catcomtec.com> wrote:
> Questions:
> * What does all of this buy me?
> * From what I can tell it just makes things better formatted. Is
> this true?
I don't use Windows for development if I can at all help it so I can't
really say anything conclusive. The resulting .xml file doesn't appear
to have a <report .../> command that uploads it to the Bitten master
so I guess you're correct that it's just to neaten up the output
(which the linked page suggests is somewhat less than usable to start
with). If one did want to upload the result XML to Bitten one would
then need to transform it into XML structured the way Bitten expects
(as is done in the next step "Smoke Test").
> * When building under a high warning level, the warning messages can
> be considered "lint" warnings. Has anyone done any work at
> transforming the msbuild log file into a bitten lint report?
> * Has anyone done any "lint" report work for VC++ with bitten?
Not sure.
> On Feb 4, 2:01 pm, potter <mpot...@catcomtec.com> wrote:
>> According to [1] to use x:transform on Windows one needs MSXML version
>> 3 or later installed.We I run a build recipe that contains the
>> x:transform I get "No usable XSLT implementation found" error.
>> According to my control panel I have "MSXML 6.0 Parser (KB933579)"
>> installed. What am I missing or is that the correct package?
>>
>> [1] http://bitten.edgewall.org/wiki/Documentation/commands.html#x-transform
Do you have XSLT and its Python bindings installed?
Schiavo
Simon
potter
> For those (like me) who are too lazy to go look at the wiki, the step
> in question is:
>
> <step id="Build">
> <sh:exec file="echo" args="Build start"/>
> <sh:exec file="msbuild.exe" args="project.sln
> @C:\\Builds\\utils\\logger\\msbuild.rsp"/>
> <x:transform src="msbuild-output.xml" dest="msbuild-result.xml"
> stylesheet="..\\utils\\logger\\msbuild.xsl" />
> <sh:exec file="type" args="msbuild-result.xml"/>
> <sh:exec file="echo" args="Build complete"/>
> </step>
>
> > Questions:
> > * What does all of this buy me?
> > * From what I can tell it just makes things better formatted. Is
> > this true?
Since posting these questions, I took a closer look at the
msbuild2ccnet.xml file. All it appears to do is format the log
messages into an HTML table with sections for each project. Not a
great improvement, but does point towards things that could be done
(e.g. color highlighting).
However I am surprised that one can issue HTML to standard out and
have it interpreted as such in the bitten log pane. I could be wrong
though since I have not yet seen this work because of x:transform not
working for me yet (below).
> > * When building under a high warning level, the warning messages can
> > be considered "lint" warnings. Has anyone done any work at
> > transforming the msbuild log file into a bitten lint report?
> > * Has anyone done any "lint" report work for VC++ with bitten?
>
> Not sure.
Started playing around to see how I could write an XSLT that could
transform the msbuild-output.xml into a Bitten lint report. Looks
promising.
> > On Feb 4, 2:01 pm, potter <mpot...@catcomtec.com> wrote:
> >> According to [1] to use x:transform on Windows one needs MSXML version
> >> 3 or later installed.We I run a build recipe that contains the
> >> x:transform I get "No usable XSLT implementation found" error.
> >> According to my control panel I have "MSXML 6.0 Parser (KB933579)"
> >> installed. What am I missing or is that the correct package?
>
> >> [1]http://bitten.edgewall.org/wiki/Documentation/commands.html#x-transform
>
> Do you have XSLT and its Python bindings installed?
The documentation did not mention either of these so this points to
the need for some documentation work. Alas I am currently at a lost
at specifically these are referring to (i.e. what packages from where).
Leandro Conde
On Feb 11, 1:44 pm, Simon Cross <hodges...@gmail.com> wrote:
> Do you have XSLT and its Python bindings installed?
<snip>
The documentation did not mention either of these so this points to
the need for some documentation work. Alas I am currently at a lost
at specifically these are referring to (i.e. what packages from where).
<snip>
I think (only guessing here) that Simon ment these: http://xmlsoft.org/XSLT/python.html, look in that page for a Windows port with the pre-compiled binaries for the python bindings...
Hope this helps, and looking forward to what's being done here :)
Cheers,
leandro
potter
> On Mon, Feb 15, 2010 at 1:01 PM, potter <mpot...@catcomtec.com> wrote:
> > On Feb 11, 1:44 pm, Simon Cross <hodges...@gmail.com> wrote:
> > > Do you have XSLT and its Python bindings installed?
> >
> > the need for some documentation work. Alas I am currently at a lost
> > at specifically these are referring to (i.e. what packages from where).
>
> with the pre-compiled binaries for the python bindings...
Thanks.
Poking around there it appears that for Windows http://users.skynet.be/sbi/libxml-python/
is the one-stop-shopping places to get the needed XSLT and Python
binding. Haven't tried it out, but working on it.
potter
> On Feb 16, 10:01 am, Leandro Conde <lean...@gmail.com> wrote:
> > I think (only guessing here) that Simon ment these:http://xmlsoft.org/XSLT/python.html, look in that page for a Windows port
> > with the pre-compiled binaries for the python bindings...
>
> is the one-stop-shopping places to get the needed XSLT and Python
> binding. Haven't tried it out, but working on it.
Yes that worked; x:transform now works on my slaves. However, ...
potter
> > On Thu, Feb 11, 2010 at 7:35 PM, potter <mpot...@catcomtec.com> wrote:
> > > Questions:
> > > * What does all of this buy me?
> > > * From what I can tell it just makes things better formatted. Is
> > > this true?
>
> Since posting these questions, I took a closer look at the
> msbuild2ccnet.xml file. All it appears to do is format the log
> messages into an HTML table with sections for each project. Not a
> great improvement, but does point towards things that could be done
> (e.g. color highlighting).
>
> However I am surprised that one can issue HTML to standard out and
> have it interpreted as such in the bitten log pane. I could be wrong
> though since I have not yet seen this work because of x:transform not
> working for me yet (below).
Nope, I got the raw HTML displayed in the Log pane after the messages
generated directly from the build step.. Is that the way it is, or do
I need to clean up my Build step such that the only output is the
HTML; thus the opening line would be:
{{{
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://
www.w3.org/TR/REC-html40/loose.dtd">
}}}
potter
* Still getting raw HTML from my build step, but still working on it.
* Have written a first stab at a MSBuild2Lint.xsl. Appears to work.
However, I discovered that Rodemeyer.MsBuildToCCnet.dll custom logger
surpresses ALL warning messages if there are ANY errors.
* Thinking about trying the Kobush.MSBuild.dll [1] custom logger
instead. It looks like this logger does a better job of logging
everything and in a better format for later XSLT transformation.
However this means all of the MSBuild XSLTs would have to be rewitten.
[1] http://geekswithblogs.net/kobush/archive/2005/05/06/39153.aspx
Questions:
* Is it possible to use HTML to format a build step log, or am I just
wasting my time on that?
* Looking at the resulting Lint tab I get on the Bitten-build, this
is a nice summory of the Lint warnings. However looking at the data
that is in the lint.xml report file and what gets store in the
database there is more. Is there a way to have a Lint-Detail report
added to the bitten-build page?
* Looking at the types of Bitten XML report types and the generic way
the data is stored in the database:
* Could other XML report types be sent and then stored or do only
the three defined work?
* Could other attributes be added to the records of the existing
report types and then be stored?
potter
> * Looking at the resulting Lint tab I get on the Bitten-build, this
> is a nice summory of the Lint warnings. However looking at the data
> that is in the lint.xml report file and what gets store in the
> database there is more. Is there a way to have a Lint-Detail report
> added to the bitten-build page?
Looks like there is a ticket on this idea already, [http://
bitten.edgewall.org/ticket/507].
potter
Trying to take a quick stab a writing a Lint-Detail report I
discovered that none of my lint messages are in the database, that is
for example the "Missing docstring" string in the "Lint Reports"
example on the "Report Formats" page [1].
Is this a bug in Bitten or an error in the "Report Formats" page?
[1] http://bitten.edgewall.org/wiki/ReportFormats#LintReports
potter
> the data is stored in the database:
> * Could other XML report types be sent and then stored or do only
> the three defined work?
> * Could other attributes be added to the records of the existing
> report types and then be stored?
From reading the "Mapping XML Reports to Database Tables" section on
the "Data Storage" page [1], it appears that the bitten XML reports
are pretty free style, what is given will get stored; however if one
gives a new report type or attribute it will get stored, just nothing
will use it.
[1] http://bitten.edgewall.org/wiki/DataStorage#MappingXMLReportstoDatabaseTables
On Feb 19, 3:28 pm, potter <potter2...@gmail.com> wrote:
> Trying to take a quick stab a writing a Lint-Detail report I
> discovered that none of my lint messages are in the database, that is
> for example the "Missing docstring" string in the "Lint Reports"
> example on the "Report Formats" page [1].
>
> Is this a bug in Bitten or an error in the "Report Formats" page?
>
> [1]http://bitten.edgewall.org/wiki/ReportFormats#LintReports
Also from reading the same, this appears to be a error in the "Report
Formats" page. I would fix the page if I knew how it should be
encoded. From the "Test Reports" example it should probably be
encoded as "stdout", but that does not seam correct. Other thoughts:
"message", "msg", "description", "desc".
osimons
> On Feb 17, 10:53 pm, potter <potter2...@gmail.com> wrote:
>
> > * Looking at the types of Bitten XML report types and the generic way
> > the data is stored in the database:
> > * Could other XML report types be sent and then stored or do only
> > the three defined work?
> > * Could other attributes be added to the records of the existing
> > report types and then be stored?
>
> From reading the "Mapping XML Reports to Database Tables" section on
> the "Data Storage" page [1], it appears that the bitten XML reports
> are pretty free style, what is given will get stored; however if one
> gives a new report type or attribute it will get stored, just nothing
> will use it.
>
That is the general idea, yes. Bitten will store whatever it gets
handed. It is up to the various bitten.report.* components (plugins)
to add meaning to certain types of reports, and currently coverage,
tests and lint have server-side reporters for summary + graphs.
> > Trying to take a quick stab a writing a Lint-Detail report I
> > discovered that none of my lint messages are in the database, that is
> > for example the "Missing docstring" string in the "Lint Reports"
> > example on the "Report Formats" page [1].
>
> > Is this a bug in Bitten or an error in the "Report Formats" page?
>
> > [1]http://bitten.edgewall.org/wiki/ReportFormats#LintReports
>
> Also from reading the same, this appears to be a error in the "Report
> Formats" page. I would fix the page if I knew how it should be
> encoded. From the "Test Reports" example it should probably be
> encoded as "stdout", but that does not seam correct. Other thoughts:
> "message", "msg", "description", "desc".
Something is indeed not quite right here. I put the lint section onto
that wiki page quite recently as it was missing, and looked primarily
at the pylint code with bitten.tests.pythontools.PyLintTestCase and
the output it generated in the test case. Basically the test case will
generate this xml:
'<problem category="convention" line="42" file="module/
file1.py">Missing docstring</problem><problem category="convention"
line="42" file="module/file2.py">Missing docstring</problem>'
Which is the basis of what I documented.
Looking at some of my actual dev/test builds that runs pylint, like
you I see that the the actual text never makes it to the database. A
line in the XML gets stored like this:
sqlite> select * from bitten_report_item where report=70 and item=1;;
70|1|category|convention
70|1|file|bitten/report/testing.py
70|1|line|1
70|1|type|problem
Which as you also found drops the text from the xml element. Currently
reports and graphs only use counts for the various types of errors,
but I actually expected the details to be in the database too. I don't
know if this is a problem in how the reports are passed to the server,
how they are pared or stored, or if it is just a missing feature for
some know or unknown reason.
I had on my todo to make a Lint annotator for Browse Source, where one
could view a file and get lines marked and tooltip describing the
problem. That obviously depends on the details being available. So, to
store it we should use something like "message" atribute to store it.
That sounds most descriptive to me. Making it optional would be fine
seeing none of the built-in tools depends on the attribute?
:::simon
https://www.coderesort.com
http://www.ohloh.net/accounts/osimons
potter
1. Correct the documentation and any code that generates lint
reports.
2. Change the XML to database loading to code.
Approach 1 is dependent on how much code is there both inside Bitten
and outside that assumes the currently documented format. Seeing that
this document is only 6 weeks old, probably not much. However, I see
that bitten's lint support was based off of the EatLint plug-in, so
maybe there is more. Question: As I understand it this was
originally for support of pylint. How does this documented report
format relate to the output of pylint? I am not familiar with pylint.
Approach 2: The loader already has the concepts of:
* Values comes from attribute values or sub-elements values of each
report element.
* Each value can only be given once, if given more than once the
later value overrides.
* The name of the report element becomes the "type" value.
The XML to db loader could be changed to add:
* The value of the element becomes the "message" value.
This would work perfectly for the current lint report definition and
any code that exists that generates such reports. However it does not
look quite right when compared to the "test" report. In the "test"
report the message appears in the "stdout" value. For simplicity
"stdout" is probably the name that should be used. It is not the best
of names, but it works.
potter
> The question may still be which approach?
> 1. Correct the documentation and any code that generates lint
> reports.
> 2. Change the XML to database loading to code.
>
> Approach 1 is dependent on how much code is there both inside Bitten
> and outside that assumes the currently documented format. Seeing that
> this document is only 6 weeks old, probably not much. However, I see
> that bitten's lint support was based off of the EatLint plug-in, so
> maybe there is more. Question: As I understand it this was
> originally for support of pylint. How does this documented report
> format relate to the output of pylint? I am not familiar with pylint.
Oops, correction. I now see that there is a python:pylint recipe
command which extract the output of pylint. So a new question is:
Does this command correctly extract a message and put it into the
database? If so under what name?
jmb
> Oops, correction. I now see that there is a python:pylint recipe
> command which extract the output of pylint. So a new question is:
> Does this command correctly extract a message and put it into the
> database? If so under what name?
There is also a patch that adds a c:cppcheck recipe command which
extracts the output of cppcheck. See http://bitten.edgewall.org/ticket/508
This patch was based off the python:pylint recipe. Both of them create
a "problem" xml element to send to the server. The problem type and
severity, and the file and line information are stored in properties
for this element. The message text is sent as the element body. I
don't know how this gets mapped into the database on the server side.
potter
> * Still getting raw HTML from my build step, but still working on it.
> * Is it possible to use HTML to format a build step log, or am I just
> wasting my time on that?
Have given up on this. I assume it cannot be done unless someone can
give me a pointer.
I see that in the log window standard error is high-lighted in red and
standard out is normal. I assume that this is the only format control
there is in the log window.
> Status report:
> * Have written a first stab at a MSBuild2Lint.xsl. Appears to work.
> However, I discovered that Rodemeyer.MsBuildToCCnet.dll custom logger
> surpresses ALL warning messages if there are ANY errors.
> * Thinking about trying the Kobush.MSBuild.dll [1] custom logger
> instead. It looks like this logger does a better job of logging
> everything and in a better format for later XSLT transformation.
> However this means all of the MSBuild XSLTs would have to be rewitten.
> [1]http://geekswithblogs.net/kobush/archive/2005/05/06/39153.aspx
Using Kobush XML Logger, I generated the following XSL that transforms
the MSBuild log into a Lint report.
> Questions:
> * Looking at the resulting Lint tab I get on the Bitten-build, this
> is a nice summory of the Lint warnings. However looking at the data
> that is in the lint.xml report file and what gets store in the
> database there is more. Is there a way to have a Lint-Detail report
> added to the bitten-build page?
On Feb 18, 2:21 pm, potter <potter2...@gmail.com> wrote:
> Looks like there is a ticket on this idea already, [http://bitten.edgewall.org/ticket/507].
Started to try to implement a summarizer for this, but bumped into
[http://bitten.edgewall.org/ticket/543], still working.
----------
{{{
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/
Transform">
<xsl:output method="html" version="1.0" encoding="UTF-8" indent="no"/
>
<xsl:template match="/">
<xsl:apply-templates select="//msbuild" />
</xsl:template>
<xsl:template match="msbuild">
<xsl:element name="report">
<xsl:attribute name="category">lint</xsl:attribute>
<xsl:apply-templates select="//warning|//error" />
</xsl:element>
</xsl:template>
<xsl:template match="warning|error">
<xsl:element name="problem">
<!-- Want to set the category based off the the @code value,
but for now all are errors or warnings based of of the node
name. -->
<xsl:attribute name="category"><xsl:value-of select="name(.)"/></
xsl:attribute>
<xsl:attribute name="line"><xsl:value-of select="@line"/></
xsl:attribute>
<xsl:attribute name="tag"><xsl:value-of select="../@name"/></
xsl:attribute>
<xsl:attribute name="file"><xsl:value-of select="@file"/></
xsl:attribute>
<!--
If the message is to be in an attributes instead of the node
value...
<xsl:attribute name="message">
-->
<xsl:if test="@code!=''"><xsl:value-of select="@code" />: </
xsl:if>
<xsl:value-of select="." />
<!--
If the message is to be in an attributes instead of the node
value...
</xsl:attribute>
-->
</xsl:element>
</xsl:template>
</xsl:stylesheet>
}}}
potter
it that the "lint" report messages should be in "msg". An appropriate
change is needed.
Simon Cross
> From my code reading of the patch by hodgestar to ticket #507, I take
> it that the "lint" report messages should be in "msg". An appropriate
> change is needed.
"Should" is perhaps too strong a word. My patch currently puts the
pylint text into msg and reads it out of msg. This avoids having to
modify the receiver in the bitten master (i.e. my patch takes the view
that the pylint uploader was wrong and that the report receiver on the
master functions as intended :).
The receiver in the Bitten master is incredibly generic:
- All report children attributes are saved as report items with name
equal to the attribute name and value equal to the attribute value.
- All report children sub-elements are saved as report items with
name equal to the tag name and value equal to the text of the tag.
The text of the report children themselves is discarded (which is the
pitfall the pylint uploader fell into).
Schiavo
Simon