Tracking
Ludovi...@gmail.com
I'm the QA lead in a startup and used to be the only person doing QA,
up until a month ago. QA used to be some hobby that I would do for open
source projects - so my experience is quite limited. We've written test
cases - and we go thru them manually - plans are to automate in the
future - so we the humans can do others things. The product we work on
is a client/server application, the client being available on Windows
and soon on MacOS, Linux. Our build environment is managed by buildbot.
And we would like to add unit testing to our build process - the thing
is that I want to be able to track what test have been run - with what
build numbers. What unit tests have failed etc ... So I'm looking into
a solution that would enable me to :
* Document test cases.
* Write test plans.
* Track test plans - manual testing - .
* get input data from external source to track what those external
sources are testing.
I've been googling, reading FAQs. I thought about adding a few plug-ins
to the tool we use actually which is Atlassian's confluence. I would
like to keep using confluence to write test cases and document them.
But I don't see how I'm going to keep that tool to track our testing.
Any suggestions ?
Ludovic
--
http://perso.hirlimann.net/~ludo/blog/
Matthias Wolpers
Ludovi...@gmail.com wrote:
> Hello,
>
> I'm the QA lead in a startup and used to be the only person doing QA,
> up until a month ago. QA used to be some hobby that I would do for open
> source projects - so my experience is quite limited. We've written test
> cases - and we go thru them manually - plans are to automate in the
> future - so we the humans can do others things. The product we work on
> is a client/server application, the client being available on Windows
> and soon on MacOS, Linux. Our build environment is managed by buildbot.
> And we would like to add unit testing to our build process - the thing
> is that I want to be able to track what test have been run - with what
> build numbers. What unit tests have failed etc ... So I'm looking into
> a solution that would enable me to :
>
> * Document test cases.
> * Write test plans.
> * Track test plans - manual testing - .
> * get input data from external source to track what those external
> sources are testing.
what you are looking for appears to be a TMT test management tool (that
is test content management _and_ test execution management, AFAICT).
in germany, there is a fairly recent survey on those in print at the
moment, but that may be little use to you unless you read german and are
willing to cough up those 300-400¤ they want. (check
http://www.heise.de/kiosk/special/ixstudie/06/01/ if interested ). they
have checked these:
Borland Silk Central Testmanager 4.3
Compuware QA Director/Center Portal 5.2
Consortium Oblect Web Salome-TMF 2.2
eXept Software expecco 1.2
IBM Rational ClearQuestTestmanager 7.0
imbus TestBench 1.4.4
Mercury Interactive QualityCenter 9.0
Microsoft Visual Studio Team Edition 1.0
Mozilla Foundation Testopia 1.0
SQS Test/Professional 8.0
TEQneers TEQdit 1.3.2
Testing Technologies TTWorkbench 1.0.13
Traq Software Ltd QATraq 6.8
Not all of those are commercial! i have only skimmed that survey and
found that the open source solutions receive mixed feedback, typically
along the lines of "hopeful candidate but still needs a fair lot of
work".
is your shop willing to spend money? then there is hope, maybe: we
looked at a few of those commercial products and found something that we
would like to try. ( but then we are a heavyweight CMMI conscious kind
of shop ). anyhow, integration of automated tests appears to be a
feature commonly boasted, so not all may be lost for you - if you can
find the $$$
hth, mats
Ludovi...@gmail.com
Matthias Wolpers wrote:
> In article <1167826827.4...@v33g2000cwv.googlegroups.com>,
> Ludovi...@gmail.com wrote:
> > build numbers. What unit tests have failed etc ... So I'm looking into
> > a solution that would enable me to :
> >
> > * Document test cases.
> > * Write test plans.
> > * Track test plans - manual testing - .
> > * get input data from external source to track what those external
> > sources are testing.
> what you are looking for appears to be a TMT test management tool (that
> is test content management _and_ test execution management, AFAICT).
Part of is is to do some reporting.
> in germany, there is a fairly recent survey on those in print at the
> moment, but that may be little use to you unless you read german and are
> willing to cough up those 300-400¤ they want. (check
> http://www.heise.de/kiosk/special/ixstudie/06/01/ if interested ). they
> have checked these:
I could can read it or get someone to read it for me :-) Buying that
paper might not be an issue.
>
> Borland Silk Central Testmanager 4.3
> Compuware QA Director/Center Portal 5.2
> Consortium Oblect Web Salome-TMF 2.2
> eXept Software expecco 1.2
> IBM Rational ClearQuestTestmanager 7.0
> imbus TestBench 1.4.4
> Mercury Interactive QualityCenter 9.0
> Microsoft Visual Studio Team Edition 1.0
> Mozilla Foundation Testopia 1.0
> SQS Test/Professional 8.0
> TEQneers TEQdit 1.3.2
> Testing Technologies TTWorkbench 1.0.13
> Traq Software Ltd QATraq 6.8
>
Thanks for the list.
> is your shop willing to spend money? then there is hope, maybe: we
> looked at a few of those commercial products and found something that we
> would like to try. ( but then we are a heavyweight CMMI conscious kind
Which ones ?
Michael Bolton
If you're considering doing unit testing the way I understand the term
(that is: a test, typically written and run by the developer, that
focuses on the smallest chunk of code that we're interested in), the
unit tests would be run by the developer before checking the code in.
You may be referring to what I would call integration, in which units
or modules are put together and then tested at some level lower than
the fully functional system.
> a solution that would enable me to :
>
> * Document test cases.
An ASCII text file could enable you to do this.
> * Write test plans.
An ASCII text file could enable you to do this.
> * Track test plans - manual testing - .
An ASCII text file or Excel spreadsheet would enable you to do this.
> * get input data from external source to track what those external sources are testing.
An email account and a human using ASCII text file or Excel spreadsheet
could enable you to do this. Some agilists have even had success using
paper index cards for everything that you've described.
Suppose you were to use an Excel workbook to track this information.
Would that be sufficient? If not, what tasks do you have in mind that
the text file or the spreadsheet couldn't handle? That is, what
problem are you trying to solve?
---Michael B.
Matthias Wolpers
Ludovi...@gmail.com wrote:
> Matthias Wolpers wrote:
> > In article <1167826827.4...@v33g2000cwv.googlegroups.com>,
> > Ludovi...@gmail.com wrote:
>
> > > build numbers. What unit tests have failed etc ... So I'm looking into
> > > a solution that would enable me to :
> > >
> > > * Document test cases.
> > > * Write test plans.
> > > * Track test plans - manual testing - .
> > > * get input data from external source to track what those external
> > > sources are testing.
> > what you are looking for appears to be a TMT test management tool (that
> > is test content management _and_ test execution management, AFAICT).
>
> Part of is is to do some reporting.
yes of course: if nobody wants to see the results you might as well stop
doing it. (im my head reporting is the final step of execcution. sorry
for the overassumption)
<snip>
> >
>
> Thanks for the list.
>
> > is your shop willing to spend money? then there is hope, maybe: we
> > looked at a few of those commercial products and found something that we
> > would like to try. ( but then we are a heavyweight CMMI conscious kind
>
> Which ones ?
>
> Ludo
for what it is worth: we arrived at the conclusion that the imbus
product suited our needs best. i wont tell you what our criteria were
since that'd come too close to disloyalty to my current employer ( for
my comfort anyway ). but we did play this by the book, and it was worth
it.
i should advise against using this info as a shortcut: what i liked or
thought i'd need is in all probability wasteful overkill in your org (
and vice versa, of course) - especially as michael b pointed you to a
less costly approach (which we use at the moment, more or less, by the
way)
also, i should point out that those commercial tools may not be what you
are looking for at all: none that we saw came without serious
maintenance overhead: your people will be hacking the tool instead of
executing those tests. that may or may not be desirable in your
environment, so once again, ... oh i think you get my drift:-)
mats