customize xml generated by ci_reports

[Marlon]

Hi, anybody here knows how to customize xml report generated by
ci_reports addon. I would like to add more attributes. Example, I want
to add <testsuite result> PASS/FAIL </testsuite result>.

In addition to this I'm planning to create some kind of a dashboard in
html form where i can select/check the testcase I want to run.

thanks!

[Tony]

Hi Marlon,

Have modified the ci_reporter to include passed/failed/error in
ci_reporter.
Also have modified the code to create one xml file instead of multiple
files when you try to run testcases present in different classes.

Modify the TestSuite and TestCase present in Ci_reporter --
(ci_reporter\lib\ci\reporter\test_suite.rb)
# Basic structure representing the running of a test suite. Used to
time tests and store results.
class TestSuite < Struct.new
(:name, :tests, :time, :failures, :errors, :assertions, :passed)
attr_accessor :testcases
attr_accessor :stdout, :stderr
def initialize(name)
super(name.to_s) # RSpec passes a "description" object instead
of a string
@testcases = []
end

# Starts timing the test suite.
def start
@start = Time.now
unless ENV['CI_CAPTURE'] == "off"
@capture_out = OutputCapture.new($stdout) {|io| $stdout =
io }
@capture_err = OutputCapture.new($stderr) {|io| $stderr =
io }
end
end

# Finishes timing the test suite.
def finish
self.tests = testcases.size
self.time = Time.now - @start
#self.failures = testcases.inject(0) {|sum,tc| sum +=
tc.failures.select{|f| f.failure? }.size }
self.failures = 0
testcases.each { |tc|
self.failures += 1 if tc.failure?
}
self.errors = testcases.inject(0) {|sum,tc| sum +=
tc.failures.select{|f| f.error? }.size }
####
self.passed =0
testcases.each { |tc|
self.passed += 1 if !(tc.failure? || tc.error?)
}
#puts "PASSED WAS - #{self.passed}"
####
self.stdout = @capture_out.finish if @capture_out
self.stderr = @capture_err.finish if @capture_err
end

# Creates the xml builder instance used to create the report xml
document.
def create_builder
require 'rubygems'
gem 'builder'
require 'builder'
# :escape_attrs is obsolete in a newer version, but should do
no harm
Builder::XmlMarkup.new(:indent => 2, :escape_attrs => true)
end

# Creates an xml string containing the test suite results.
def to_xml
builder = create_builder
# more recent version of Builder doesn't need the escaping
def builder.trunc!(txt)
txt.sub(/\n.*/m, '...')
end
#builder.instruct!
attrs = {}
each_pair {|k,v| attrs[k] = builder.trunc!(v.to_s) unless
v.nil? || v.to_s.empty? }
builder.testsuite(attrs) do
@testcases.each do |tc|
tc.to_xml(builder)
end
builder.tag! "system-out" do
builder.cdata! self.stdout
end
builder.tag! "system-err" do
builder.cdata! self.stderr
end
end
end

end

class TestCase < Struct.new(:name, :time, :assertions)
attr_accessor :failures

def initialize(*args)
super
@failures = []
end

# Starts timing the test.
def start
@start = Time.now
end

# Finishes timing the test.
def finish
#self.time = Time.now - @start
self.time = (Time.now - @start).to_i # roundoff the time to an
integer
end

# Returns non-nil if the test failed.
def failure?
!failures.empty? && failures.detect {|f| f.failure? }
end

# Returns non-nil if the test had an error.
def error?
!failures.empty? && failures.detect {|f| f.error? }
end

# Writes xml representing the test result to the provided
builder.
def to_xml(builder)
attrs = {}
each_pair {|k,v| attrs[k] = builder.trunc!(v.to_s) unless
v.nil? || v.to_s.empty?}
builder.testcase(attrs) do
failures.each do |failure|

# - This is where the type of failure is checked .. error
or failure
if failure.kind_of?(YNOT::YNOTE::TestUnitFailure)
builder.failure(:type => builder.trunc!
(failure.name), :message => builder.trunc!(failure.message)) do
builder.text!(failure.message + " (#{failure.name})
\n")
builder.text!(failure.location)
end
else

builder.error(:type => builder.trunc!
(failure.name), :message => builder.trunc!(failure.message)) do
builder.text!(failure.message + " (#{failure.name})
\n")
builder.text!(failure.location)
end
end

end
end
end

end

Hope this helps ...

Thanks,
Paul

[Dylan]

Hi there! This is something I've been looking for for a while, but
unfortunately i can't seem to get your code working. I had to replace
YNOT::YNOTE::TestUnitFailure with Test::Unit::Failure because it was
not recognizing your code, and it's added the "passed' attribute
correctly, but it is always set to 1, even if the suite fails. Also, I
still get multiple xml files created. Could you give me an example of
how you call it that only results in one file? Thanks!

-Dylan

[Tony]

Hi Dylan,

I had modified the code .. and am not sure where to copy and paste
it ...
Iam including the whole source code for the modified ci reporter -

To use this remove the require for ci_reporter and require the
modified file.
The usage remains the same ....
set the env variable ENV['CI_REPORTS'] = the folder where the reports
need to be stored.
The file also changes the report file slightly to include multiple
test classes that were run.

require 'delegate'
require 'stringio'
require 'fileutils'
require 'test/unit'
require 'test/unit/ui/console/testrunner'
require 'ClassAttr'

module YNOT
module YNOTE

class OutputCapture < DelegateClass(IO)
# Start capturing IO, using the given block to assign self to
the proper IO global.
def initialize(io, &assign)
super
@delegate_io = io
@captured_io = StringIO.new
@assign_block = assign
@assign_block.call self
end

# Finalize the capture and reset to the original IO object.
def finish
@assign_block.call @delegate_io
@captured_io.string
end

# setup tee methods
%w(<< print printf putc puts write).each do |m|
module_eval(<<-EOS, __FILE__, __LINE__)
def #{m}(*args, &block)
@delegate_io.send(:#{m}, *args, &block)
@captured_io.send(:#{m}, *args, &block)
end
EOS
end
end

class ReportManager
@@current_suites = Array.new()
class_attr_accessor(:current_suites)
def initialize(prefix)
@basedir = ENV['CI_REPORTS'] || File.expand_path("#{Dir.getwd}/
#{prefix.downcase}/reports")
@basename = "#{@basedir}/#{prefix.upcase}"
FileUtils.mkdir_p(@basedir)
end

def write_report(suite)
File.open("#{@basename}-#{suite.name.gsub(/[^a-zA-Z0-9]+/,
'-')}.xml", "w") do |f|
f << "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
f << suite.to_xml
end
end

# will do runtime reporting
def write_newreport_allsuites()
File.open("#{@basename}-#{"MYREPORTFILE".gsub(/[^a-zA-Z0-9]+/,
'-')}.xml", "w") do |f|
f << "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<tests>\n"
ReportManager.current_suites.each { |st|
f << st.to_xml
}
f << "</tests>"
end
end

end

class Failure
def self.new(fault)
fault.kind_of?(Test::Unit::Failure) ? TestUnitFailure.new
(fault) : TestUnitError.new(fault)
end
end

# Wrapper around a <code>Test::Unit</code> error to be used by the
test suite to interpret results.
class TestUnitError
def initialize(fault)
@fault = fault
end
def failure?() false end
def error?() true end
def name() @fault.exception.class.name end
def message() @fault.exception.message end
def location() @fault.exception.backtrace.join("\n") end
end

# Wrapper around a <code>Test::Unit</code> failure to be used by
the test suite to interpret results.
class TestUnitFailure
def initialize(fault)
@fault = fault
end
def failure?() true end
def error?() false end
def name() Test::Unit::AssertionFailedError.name end
def message() @fault.message end
def location() @fault.location.join("\n") end
end

# Replacement Mediator that adds listeners to capture the results
of the <code>Test::Unit</code> runs.
class TestUnit < Test::Unit::UI::TestRunnerMediator
def initialize(suite, report_mgr = nil)
super(suite)
@report_manager = report_mgr || ReportManager.new("test")
add_listener(Test::Unit::UI::TestRunnerMediator::STARTED,
&method(:started))
add_listener(Test::Unit::TestCase::STARTED, &method
(:test_started))
add_listener(Test::Unit::TestCase::FINISHED, &method
(:test_finished))
add_listener(Test::Unit::TestResult::FAULT, &method(:fault))
add_listener(Test::Unit::UI::TestRunnerMediator::FINISHED,
&method(:finished))
end

def started(result)
@suite_result = result
@last_assertion_count = 0
@current_suite = nil
@unknown_count = 0
@result_assertion_count = 0
end

def test_started(name)
test_name, suite_name = extract_names(name)
unless @current_suite && @current_suite.name == suite_name
finish_suite
start_suite(suite_name)
end
start_test(test_name)
end

def test_finished(name)
finish_test
end

def fault(fault)
tc = @current_suite.testcases.last
tc.failures << Failure.new(fault)
end

def finished(elapsed_time)
finish_suite
end

private
def extract_names(name)
match = name.match(/(.*)\(([^)]*)\)/)
if match
[match[1], match[2]]
else
@unknown_count += 1
[name, "unknown-#{@unknown_count}"]
end
end

def start_suite(suite_name)
@current_suite = TestSuite.new(suite_name)
@current_suite.start
ReportManager.current_suites << @current_suite
=begin
@current_suite = TestSuite.new(suite_name)
@current_suite.start
=end
end

def finish_suite
if @current_suite
@current_suite.finish
@current_suite.assertions = @suite_result.assertion_count -
@last_assertion_count
@last_assertion_count = @suite_result.assertion_count
##### remove this after you make the changes
######@report_manager.write_report(@current_suite) #old
report -- creates a xmlfile for each class
################################
#@report_manager.write_newreport(@current_suites)
@report_manager.write_newreport_allsuites()
end
end

def start_test(test_name)
tc = TestCase.new(test_name)
tc.start
@current_suite.testcases << tc

#=begin --- This is written for returning results in runtime.. after
each testcase is run... when running only name will be present
@current_suite.assertions = @suite_result.assertion_count -
@last_assertion_count
@last_assertion_count = @suite_result.assertion_count
#######@report_manager.write_report(@current_suite) #old
report -- creates a xmlfile for each class
@report_manager.write_newreport_allsuites()
#@report_manager.write_newreport_testcaselevel
(@current_suites)
#@report_manager.write_newreport(@current_suites)
#=end
end

def finish_test
tc = @current_suite.testcases.last
tc.finish
tc.assertions = @suite_result.assertion_count -
@result_assertion_count
@result_assertion_count = @suite_result.assertion_count
@report_manager.write_newreport_allsuites()
end
end

end
end


module Test #:nodoc:all
module Unit
module UI
module Console
class TestRunner
def create_mediator(suite)
# swap in our custom mediator
return YNOT::YNOTE::TestUnit.new(suite)

end
end
end
end
end
end

If this doesnt work ... i could mail you the whole file.

Thanks,
Tony

[Charley Baker]

Hi Tony,

   This would be a good addition to the advanced examples and frameworks on the wiki if you'd like to add it there: http://wiki.openqa.org/display/WTR/Examples


Charley Baker
blog: http://blog.charleybaker.org/
Lead Developer, Watir, http://wtr.rubyforge.org
QA Architect, Gap Inc Direct

[Dylan]

Thanks Tony! I have tried using your file and I'm receiving this
error: custom_require.rb:31:in 'gem_original_require': no such file to
load -- ClassAttr (LoadError)

The error occurs on the line with "require 'ClassAttr'". Am I missing
some file? Thanks again, I appreciate the help!

-Dylan

> ...
>
> read more »

[Tony]

Hi Dylan,

Ooops sorry missed that...
Basically this class just gives some short cut methods to access the
class variables from ReportManager.

Code in file ClassAttr.rb
class Class

def class_attr_reader(*symbols)
symbols.each do |symbol|
self.class.send(:define_method, symbol) do
class_variable_get("@@#{symbol}")
end
end
end
def class_attr_writer(*symbols)
symbols.each do |symbol|
self.class.send(:define_method, "#{symbol}=") do |value|
class_variable_set("@@#{symbol}", value)
end
end
end

def class_attr_accessor(*symbols)
class_attr_reader(*symbols)
class_attr_writer(*symbols)
end

end

Let me know if you run into any other issues - :)

Thanks,
Tony

[Dylan]

Thanks! I have it all up and running and its perfect! I had to rework
my xml stylesheet but it was totally worth it. :)

-Dylan

[Dylan]

I edited it a little to fit my needs, thought I'd post it:

http://pastie.org/554062

I added a "passed" value to the <tests> tag so I can easily check if
the entire suite passed/failed, along with a "time" value so I know
when the test was run.

Thanks again, Tony!

-Dylan

[Tony]

Hi Dylan,
nice idea .. adding the passed attribute to the <test> tag :)

I made a few changes to the code, removed the global variable that you
were using to check if passed is true or not. I try to avoid global
variables.
Also made the change because the passed attribute would have to be
updated after each testcase run. Right now it was updated after the
whole testsuite is executed.
http://pastie.org/554831

The code was written to use the xml for runtime reporting (link this
xml to an ajax webpage to display reports). The xml would be updated
the moment a test starts or test finishes.
If you dont need this, remove @report_manager.write_newreport_allsuites
() from TestUnit.start_test and TestUnit.finish_test. Now the xml
would be generated only after each testuite(class) has finished
executing all the tests as in the original Ci_reporter.

Thanks,
Tony

[Marlon Mojares]

Hi, I'm still encountering "no such file to load -- ClassAttr
(LoadError)" where should I put the file ClassAttr.rb? I put it inside
ci_reporter\lib\ci\reporter but the script was unable to find the
file.

This is my test script:

require 'watir'
require 'ci/reporter/rake/test_unit_loader.rb'

class TC_TEST_suite < Test::Unit::TestCase
def test_a_search
test_site = "http://www.google.com"
browser = Watir::Browser.new
browser.goto test_site
browser.text_field(:name, "q").set "pickaxe"
browser.button(:name, "btnG").click
end
end

is this correct?

[Dylan]

I just removed the require 'ClassAttr.rb' line and copy-pasted the
ClassAttr code into the top of Tony's modified reporter file.

-Dylan

[Marlon Mojares]

after doing that i now i have this error:

c:/ruby/lib/ruby/gems/1.8/gems/ci_reporter-1.6.0/lib/ci/reporter/test_unit.rb:97:in
`start_suite': uninitialized constant
CI::Reporter::TestUnit::TestSuite (NameError)

thanks

[Tony]

Hi Marlon,

Now you dont need to use "require 'ci/reporter/rake/
test_unit_loader.rb' "
Instead only require the modified file.

Thanks,
Tony

[pallavi shashidhar]

Hi there,
Am using Ci_reporter (not the one you have mentioned. The original one.)
I have an ERP suite which has watir scripts of various projects under it:
This is the structure:
ERP
    -  stores
    -  works
    -  payroll
    -  financials

When i run them as a suite all the testcases are run and there is a single XML report in Test folder which gives the list of all the testcases of all projects. Is there a way to separate out the reporting for individual projects? like :

Stores : Testsuite(32)
   <with results>
Works : Testsuite (3)
   <with results>

and so on?

Regards,
Pallavi

[Dylan]

If I am understanding your question correctly, one method would be to
use Tony's modified reporter file and put each project in its own test
suite. This would produce 1 xml file with each project in its own
<testcase></testcase> tags. Doing this with the original CI_Reporter
will create a separate xml file for each suite

-Dylan

On Jul 23, 2:48 am, pallavi shashidhar <pals.sha...@gmail.com> wrote:
> Hi there,
> Am using Ci_reporter (not the one you have mentioned. The original one.)
> I have an ERP suite which has watir scripts of various projects under it:
> This is the structure:
> ERP
>     -  stores
>     -  works
>     -  payroll
>     -  financials
>
> When i run them as a suite all the testcases are run and there is a single
> XML report in Test folder which gives the list of all the testcases of all
> projects. Is there a way to separate out the reporting for individual
> projects? like :
>
> Stores : Testsuite(32)
>    <with results>
> Works : Testsuite (3)
>    <with results>
>
> and so on?
>
> Regards,
> Pallavi
>

[Marlon Mojares]

Yeah its working now this is a good start.

[pallavi shashidhar]

Hi there,
I have replaced the test_suite.rb with the new one and have done a - require 'ci/reporter/test_suite'
here's the code

require 'watir'
require 'ci/reporter/test_suite.rb'

class TC_TEST_suite < Test::Unit::TestCase
      def test_a_search
            test_site = "http://www.google.com"
            browser = Watir::Browser.new
            browser.goto test_site
            browser.text_field(:name, "q").set "pickaxe"
            browser.button(:name, "btnG").click
      end
end

When i run the testcase, I am getting this error:

c:/ruby/lib/ruby/gems/1.8/gems/ci_reporter-1.5.2/lib/ci/reporter/test_suite.rb:211:in `write_newreport_allsuites': private method `gsub' called for nil:NilClass (NoMethodError)
    from c:/ruby/lib/ruby/gems/1.8/gems/ci_reporter-1.5.2/lib/ci/reporter/test_suite.rb:337:in `start_test'
    from c:/ruby/lib/ruby/gems/1.8/gems/ci_reporter-1.5.2/lib/ci/reporter/test_suite.rb:278:in `test_started'
    from c:/ruby/lib/ruby/gems/1.8/gems/ci_reporter-1.5.2/lib/ci/reporter/test_suite.rb:54:in `to_proc'
    from c:/ruby/lib/ruby/1.8/test/unit/util/observable.rb:78:in `call'
    from c:/ruby/lib/ruby/1.8/test/unit/util/observable.rb:78:in `notify_listeners'
    from c:/ruby/lib/ruby/1.8/test/unit/util/observable.rb:78:in `each'
    from c:/ruby/lib/ruby/1.8/test/unit/util/observable.rb:78:in `notify_listeners'
    from c:/ruby/lib/ruby/1.8/test/unit/ui/testrunnermediator.rb:47:in `run_suite'
     ... 11 levels...
    from c:/ruby/lib/ruby/1.8/test/unit/autorunner.rb:216:in `run'
    from c:/ruby/lib/ruby/1.8/test/unit/autorunner.rb:12:in `run'

[Marlon Mojares]

Did you edit the test_suite.rb as instructed by Tony?

[Tony]

Hi Pallavi,

When using your modified file, you dont need to "require ci_reporter"
nor need to have it installed.
Just require the modified file.

require 'watir'
#require 'ci/reporter/test_suite.rb'
require '<YOUR MODIFIED FILE>'

class TC_TEST_suite < Test::Unit::TestCase
def test_a_search
test_site = "http://www.google.com"
browser = Watir::Browser.new
browser.goto test_site
browser.text_field(:name, "q").set "pickaxe"
browser.button(:name, "btnG").click
end
end

Thanks,
Tony

[pallavi shashidhar]

Hi Tony,

I think i used the previous code in http://pastie.org/554062 instead of http://pastie.org/554831
It is now working fine for me.
Thanx for all your replies.


Regards,
Pallavi

[Test Test]

Hi Tony,

My test code is:

require 'watir'
require 'test'

class TC_TEST_suite < Test::Unit::TestCase
def test_a_search
test_site = "http://www.google.com"
browser = Watir::Browser.new
browser.goto test_site
browser.text_field(:name, "q").set "pickaxe"
browser.button(:name, "btnG").click
end
end

And "test.rb" is in the local directory which contains the code given
in "http://pastie.org/554831".

But when i run it still gives me: no such file to load -- ClassAttr
(LoadError).

Please correct me where i am wrong.

Thanks
D G

On Jul 24, 4:04 pm, pallavi shashidhar <pals.sha...@gmail.com> wrote:
> Hi Tony,
>

> I think i used the previous code inhttp://pastie.org/554062instead ofhttp://pastie.org/554831

> It is now working fine for me.
> Thanx for all your replies.
>
> Regards,
> Pallavi
>

[Test Test]

Tony,
I have added Class.Attr contents and now its working fine but my
requirement is to get report(s) in html format not xml.

Please help.

Thanks
D G

On Jul 24, 4:04 pm, pallavi shashidhar <pals.sha...@gmail.com> wrote:

> Hi Tony,
>
> I think i used the previous code inhttp://pastie.org/554062instead ofhttp://pastie.org/554831

> It is now working fine for me.
> Thanx for all your replies.
>
> Regards,
> Pallavi
>

[Dylan]

To view the results html-like you have to create an xsl stylesheet to
format the xml data. Then add <?xml-stylesheet type="text/xsl"
href="stylesheet.xsl"?> to the top of your results file(s). Then you
can just open the xml file and it will open fully formatted.

Here's the xsl stylesheet I use: http://pastie.org/593268
It probably wont work for you because I've made some changes to how
the reporter outputs the xml file, but it should give you the basic
idea.

-Dylan

On Aug 23, 11:52 pm, Test Test <checktestingthi...@gmail.com> wrote:
> Tony,
> I have added Class.Attr contents and now its working fine but my
> requirement is to get report(s) in html format not xml.
>
> Please help.
>
> Thanks
> D G
>
> On Jul 24, 4:04 pm, pallavi shashidhar <pals.sha...@gmail.com> wrote:
>
> > Hi Tony,
>

> > I think i used the previous code inhttp://pastie.org/554062insteadofhttp://pastie.org/554831

[Test Test]

Thnx Dylan

[Marlon]

Hi, how can I rename the filename of the xml created by ci_reporter? I
need a repository and store the test results generated. ex. TEST-
<Class Name><Date/Time>.xml. Which part of the ci_reporter does the
filenaming/file creation?

Please help

Thanks!

[Dylan]

Using the base ci_reporter install, the report_manager.rb file has an
variable called @basename, which gets the suite name added onto it in
the File.open call in write_report. To add date/time to this just add
Time.now.strftime("put your time formatting here") to the end of the
string in the File.open call.

It should look something like this: http://pastie.org/601390

-Dylan

[Marlon MxM]

Thanks for your help Dylan! It works great.