QA Testing software
Julian, John N
if I brought a parachute) I'm VERY new to the field of QA Software
Testing, so forgive me if I sound like a blithering idiot.
My managers have requested some research into automated QA software that
will test our applications for "pre-release integrity". The software
package to be tested is used to control access to areas of a Dept of
Energy facility... let's just say it is a MAJOR CRITICAL system... so
the quality of the testing is VERY important. However, funding is
highly limited at the present time. If anyone has any suggestions as to
what software might best serve our needs, your input would be greatly
appreciated.
The software package to be tested is written with Oracle tools (Reports
3.0, Forms 5.0, Developer/2000, Designer/2000) and PL/SQL. The server
and database are running on an HP-UNIX box and the Client is running on
Win95/98 platforms. Let me know if you need more information than this.
Thanks, in advance, for any assistance you can provide,
John N. Julian, Sr.
ACES Technical Lead
LMSI/SD&I - DOE-RL
John_N...@rl.gov
maura_sh...@my-dejanews.com
"Julian, John N" <John_N...@apimc01.rl.gov> wrote:
If anyone has any suggestions as to
> what software might best serve our needs, your input would be greatly
> appreciated.
The FAQ for this group has a very good list of what's out there as far
as products. Unfortunately I haven't decided on any of them yet
myself, so I can't recommend one for you. But at least you can get an
idea of what's available.
Best of luck,
Maura
--== Sent via Deja.com http://www.deja.com/ ==--
---Share what you know. Learn what you don't.---
tester...@my-dejanews.com
"Julian, John N" <John_N...@apimc01.rl.gov> wrote:
remember
> if I brought a parachute) I'm VERY new to the field of QA Software
> Testing, so forgive me if I sound like a blithering idiot.
>
> My managers have requested some research into automated QA software
that
> will test our applications for "pre-release integrity". The software
> package to be tested is used to control access to areas of a Dept of
> Energy facility... let's just say it is a MAJOR CRITICAL system... so
> the quality of the testing is VERY important. However, funding is
to
> what software might best serve our needs, your input would be greatly
> appreciated.
>
(Reports
> 3.0, Forms 5.0, Developer/2000, Designer/2000) and PL/SQL. The server
> and database are running on an HP-UNIX box and the Client is running
on
> Win95/98 platforms. Let me know if you need more information than
this.
>
> Thanks, in advance, for any assistance you can provide,
>
> John N. Julian, Sr.
> ACES Technical Lead
> LMSI/SD&I - DOE-RL
> John_N...@rl.gov
>
>
There are a number of questions which come to my mind before you even
talk about automation...
1. When you say "...used to control access to areas of a Dept of Energy
facility..." - do you mean PHYSICAL access as in a person trying to go
through a controlled doorway or COMPUTER access as in a person trying to
gain access to controlled files/databases?
2. Are the client and server platforms (and all interconnecting
communications lines like a LAN, etc.) all safely within the controlled
perimeter of the facility or are parts accessible from outside? As an
example: the client computer is in a little room outside a fenced in
area so it is easily physically accessible and so are the comm lines
(and the server is indirectly physically accessible if the comm lines
are accessible)to an unauthorized person.
3. Are the client/server/comm lines dedicated to this access function or
are they used for some other purposes? If they are not dedicated and
isolated from unauthorized users - this increases the possibility of
deliberate or accidental corruption of your authorization database (I
assume that's what the database is for: it contains a list of authorized
users, their access "codes" and, possibly, access "priveleges").
What I'm getting at is that you should devise scenarios for attacking
the system while viewing it from the standpoint of unauthorized users
trying to deliberate access the facility, accidently accessing the
facility, deliberately corrupting the access system/database, accidently
corrupting the access system/database, etc.
From there - you can get an idea of what type of test tools you may need
although it seems to me that you could easily make some test scripts in
Visual Basic for the Windows platform (i.e. client) and shell scripts
(or Perl or Python or some other simple scripting language) for the UNIX
box (i.e. server). Really no need for fancy commercial test tools like
X-Runner, etc.
Remember that automated test tools have to be
programmed/setup/maintained just like the software they're being used to
test so they can't replace a human being coming up with unusual means of
accessing the system...
--
Guy Jackson
Design Validation Testing Engineer
MTI Technology Corporation
John Hardman
If the system you talk about is safety critical (i.e. nuclear) or just
critical (e.g. the electricity supply to the city fails) a number of things
you mentioned worry me.
>> My managers have requested some research into automated QA software that
will test our applications for "pre-release integrity".
Automation is not the answer to everything. Automating testing successfully
involves a LOT of effort and does not come cheap. Automated tools do not do
the work for you - you still have to program them (via a scripting language
etc). All that they do is execute the tests that you program. Forget about
record and replay - that is only useful for really small non-critical
things.
>> the quality of the testing is VERY important. However, funding is
highly limited at the present time.
A no-win situation. Good testing is not cheap, but it is good value. Even
better value is performing a QA function throughout the project life-cycle,
not just tacking some testing time on the end of the project schedule. You
want good testing - pay good money for good people. If potential
employees/contractors quote a rate that is less than you pay for your
developers throw their resume in the trash and look at the next one. Don't
skimp on tools, but prioritise your budget sensibly. If you have limited
funding, version control, configuration management and defect tracking tools
are better places to spend your money than automated testing in the first
instance. Again, automation does not come cheaply.
>> the Client is running on Win95/98 platforms
Is this client supposed to be up and running 24 hours a day, 7 days a week
etc? If so, you are on the wrong platform (I await flaming now...). If you
want any degree of reliability on a Windows platform move to Windows NT 4
instead of using 95/98 (further flaming will now occur from the UNIX camp...
yes I do realise that UNIX is more stable than NT, but if Windows has to be
used NT is the one to have).
Regards,
John Hardman.
vant...@my-dejanews.com
>> "John Hardman" <yi...@dial.pipex.com> wrote a whole lot of stuff
>> that you can read from his email.
I just would like to reiterate what John is saying, so you don't think
it is just his opinion. These two statements are contradictory -
1) "let's just say it is a MAJOR CRITICAL system... so
the quality of the testing is VERY important."
2) "funding is highly limited at the present time."
Forget the appearance of management buy-in because they tell you they
buy in. Or because they are hiring QA people. That is a start, but not
necessarily even a good one. If they don't give you the means necessary
to bring quality to the system, then they don't really even want
quality. It almost sounds like they just want to tag something quick on
the backend after development just to check a few things. That's not
quality assurance. Who knows - maybe they're just fulfilling some safety
requirements and don't really care.
Considering previous posts speculating about the nature of this product
you are testing, it is very important that you are involved in the
analysis and design stages as well. And, as John said, you need to do
more than just a quick automated test at the end to check for missing
window titles and slow response times.
--
Tim Van Tongeren
Al White
My managers have requested ...
a MAJOR CRITICAL system...
... testing is VERY important.
...funding is highly limited ...
If anyone has any suggestions . . . RUN!
Either testing is very important, or funding is highly limited.
Not both, unless someone in management is either
a) clueless,
b) pointy-haired, or
c) a political appointee.
Tell them I said so.
Al
Glenn Arsenault
Test Automation is not a "Silver Bullet". Automation requires work and
funding. DO not expect that automation tools are cheap or that you can
eliminate a QA staff when you have them. First, you still need to have
qualified staff to develop your test plans and test cases. Second, you then
need to evaluate tools and select and purchase a tool that fits your needs
in your environment. Third, you need qualified staff that can devise a test
automation strategy, architect and code an automation solution. Test tools
are expensive and you need highly trained staff for them to be useful and
effective. My last comment would be that, you cannot expect that the first
automation project will be cost effective, payback from the use of
automation will be on the 3rd to nth project.
Julian, John N wrote:
> (I feel like I'm jumping out of a plane at 10,000ft and I can't remember
> if I brought a parachute) I'm VERY new to the field of QA Software
> Testing, so forgive me if I sound like a blithering idiot.
>
> My managers have requested some research into automated QA software that
> will test our applications for "pre-release integrity". The software
> package to be tested is used to control access to areas of a Dept of
> Energy facility... let's just say it is a MAJOR CRITICAL system... so
> the quality of the testing is VERY important. However, funding is
> highly limited at the present time. If anyone has any suggestions as to
> what software might best serve our needs, your input would be greatly
> appreciated.
>
> The software package to be tested is written with Oracle tools (Reports
> 3.0, Forms 5.0, Developer/2000, Designer/2000) and PL/SQL. The server
> and database are running on an HP-UNIX box and the Client is running on
> Win95/98 platforms. Let me know if you need more information than this.
>
> Thanks, in advance, for any assistance you can provide,
>
> John N. Julian, Sr.
> ACES Technical Lead
> LMSI/SD&I - DOE-RL
> John_N...@rl.gov
--
To reply by email, remove the NOSPAM from reply email address.
Glenn J. Arsenault
Quality Assurance Manager
Fidelity Investments
Julian, John N
<SOAP BOX MODE ON>
But some of you replied privately with some not so pleasant responses.
I have half a mind to post them publicly. I might politely suggest,
however, that those people participating in this newsgroup for the sole
purpose of boosting their own egos find another hobby. I don't need to
be told that I have no business looking into QA software unless I'm
trained to use it. As my message clearly indicates, I have no choice.
I don't need to be told that the cheap stuff won't do what I need it
to... when, by your replies, you haven't a clue as to exactly what I
need it to do. I certainly don't need to be made to feel about 3 inches
tall by telling me in a three page email that I'm wasting my time.
Either contribute to the newsgroup with something helpful, or don't
contribute at all!!
<SOAP BOX MODE OFF>
My apologies to those who's valuable time I've wasted with this reply --
but some people need a good slapping down now and then.
John Julian, Sr.
(My employer's name is left off this message for good reason -- The
views expressed here are mine and mine alone).
John Hardman
I don't know what people sent to you privately, but nothing I have seen here
in the newsgroup looks out of the ordinary in terms of replies from
QA/testing people. One thing QA/testers soon learn is that the rest of the
world is out to get them, so we learn to be able to justify anything we say
and back it up with lots of experience/evidence. We also get VERY cynical
about certain things - your (managers) question just happened to hit on some
of those things, particularly not providing enough budget to do the job
properly and assuming that the sales speak about automated tools is true.
When people tell you that the things you mention are contradictory they are
telling you for your own good. It is better that you know now so that you
can lobby for more budget etc than it is to find out in 3 months that the
energy system fails because you didn't have enough resources to QA the
systems thoroughly. Personally, in your situation I would prepare a risk
analysis covering what you have been asked to do and the consequences of the
limitations placed upon you.
We could have answered your original questions with ... buy WinRunner and
XRunner because they use the same (presumably) scripting language and cover
both Windows and UNIX platforms. That may have been the answer you were
after but it wouldn't do you much good if your management won't pay for the
products and won't give you the time/people to use them effectively.
Regards,
John "boosting his own ego but with plenty of other hobbies" Hardman.
Julian, John N
does not apply here.
And management is most likely all of the above (a,b,&c) - as they've put
10 different managers on the project over 4 months.
Just don't tell them I said so! :-)
I'll give you a new acronym I picked up here: YTDAW (Your Tax Dollars
At Work).
Bye for now.
> -----Original Message-----
> From: Al White [SMTP:SpamTroll...@notmail.com]
> Posted At: Wednesday, May 12, 1999 8:35 AM
> Posted To: comp.software.testing
> Conversation: QA Testing software
tester...@my-dejanews.com
"Julian, John N" <John_N...@apimc01.rl.gov> wrote:
I'm not trying to stroke my ego here but that's what motivated me to
(publicly) reply to your inquiry in the way that I did. Like that old
saying (I'm REALLY paraphrasing here): when you're up to your butt
(a somewhat politically correct way of saying that) in alligators -
you don't need someone safe on shore telling you to get out of the
swamp...
To, more or less, reiterate what I said in my first reply:
1. A little clarification of the software product by you would be useful
if you want more details responses to your question (that is - if you're
allowed to give more details. Believe me - I've worked on sensitive
projects before and security/proprietary regulations make it a real pain
to go outside of your company...)
2. As John Hardman said in his last post (and what I said in a
roundabout way): it would behoove you to do a risk analysis. You're in
the software equivalent of a medical triage situation so you need to
find out what are the most critical items of the software. And not only
come up with scenarios wherein unauthorized personnel are trying to get
access to the facility but also scenarios where authorized personnel are
denied access to the facility either through accident (e.g. corruption
of the database due to a defect in the client SQL script) or deliberate
"denial of services" through the actions of a malicious unauthorized (or
even authorized) person.
Following the risk analysis - you need to rank the criticality of the
risk items. I would think the software developers should have already
done this or would be willing to assist with this task since they want
to put out a good product, too.
From there - start getting a feel for the size (such as lines of code)
and the complexity of the critical software items. Then get feel for the
inputs and outputs of this software and look for areas that would be
amenable to automation - basically simple, repetitive actions or
time/resource-intensive actions. (and that may not be a complete list of
how automation can assist you but they're the ones which come to mind).
For example - there is physically only one of you. But say you have a
risk scenario which entails the simultaneous (or near-simultaneous)
access to the facility of multiple people: you can't do test this
manually. But it is possible to develop a "virtual users" on another
platform to perform these simultaneous accesses. I'm not familiar with
the parameters of commercially available load/stress testing tools so I
don't know if any of them are suitable/adaptable to your needs but I
could see that a simple program/script could be created which would
simulate various users (like stepping through a small table of
authorization information needed to access the facility). For this
example - I wouldn't really see a great need for the time/expense of a
commercially-available testing tool...just some small programs/scripts
written in Visual Basic, Perl, Python, Tcl, UNIX shell script language -
even a common programming language like C could be used to set up these
small "test harnesses".
But then you might have a scenario wherein it is possible for a
malicious unauthorized person to install a Trojan horse on your client
platform. That scenario doesn't lend itself to automation and manual
testing is probably called for.
Well - I'll stop there since I feel I'm getting long-winded. I just hope
I was able to convey to you some useful information. I respect the
position you're in and I don't envy you (and I've been in "triage"
situations before - believe me).
--
Guy Jackson
Design Validation Testing Engineer
MTI Technology Corporation
Nanda Kishore Panakanti
http://www.stlabs.com/testnet/docs/snakeoil.htm
Nanda
tester...@my-dejanews.com wrote:
In article <6E5FB8BFB7CDD2118F6...@apexnews.rl.gov>,
"Julian, John N" <John_N...@apimc01.rl.gov> wrote:
> (I feel like I'm jumping out of a plane at 10,000ft and I can't
remember
> if I brought a parachute) I'm VERY new to the field of QA Software
> Testing, so forgive me if I sound like a blithering idiot.
>
> My managers have requested some research into automated QA software
that
> will test our applications for "pre-release integrity". The software
> package to be tested is used to control access to areas of a Dept of
> Energy facility... let's just say it is a MAJOR CRITICAL system... so
> the quality of the testing is VERY important. However, funding is
> highly limited at the present time. If anyone has any suggestions as
to
> what software might best serve our needs, your input would be greatly
> appreciated.
>
> The software package to be tested is written with Oracle tools
(Reports
> 3.0, Forms 5.0, Developer/2000, Designer/2000) and PL/SQL. The server
> and database are running on an HP-UNIX box and the Client is running
on
> Win95/98 platforms. Let me know if you need more information than
this.
>
> Thanks, in advance, for any assistance you can provide,
>
> John N. Julian, Sr.
> ACES Technical Lead
> LMSI/SD&I - DOE-RL
> John_N...@rl.gov
>
>
There are a number of questions which come to my mind before you even
talk about automation...1. When you say "...used to control access to areas of a Dept of Energy
--
tester...@my-dejanews.com
"Julian, John N" <John_N...@apimc01.rl.gov> wrote:
And in your defense, John - I don't believe those people who sent you
negative e-mail fully understand the context of your situation...
FYI to all (primarily to those of you who live in countries other than
the US):
John's employer is the Department of Energy. They run virutally all the
nuclear weapons facilities in the US. It was recently disclosed that a
researcher at the Los Alamos research facility is suspected of
disclosing sensitive "nuclear weapons code information" (whatever that
really means) to the Chinese government. The DOE immediately came under
fire for "laxness" in overall security at these facilities and Energy
Secretary Bill Richardson generated a press release last Thursday which
acknowledged some of laxness but defended the overall security steps
taken by his department.
I'm sure the political sensitivity of security access at DOE facilities
has probably reached a fever pitch because of this. That's also probably
putting John here under a lot of attention (and hence - stress). Telling
him that he's wasting his time in a blunt manner is not a good idea...
(As a sideline note: I used to work as a software quality assurance
engineer on the US Air Force's C-17 Airlifter. There were about 10 in
this department riding herd on hundreds (that is no exaggeration) of
software programs being developed for the aircraft. One day after
working there about 2 years - I looked at the Los Angeles Times
newspaper and saw a nice long article which had the headline "GAO
Says C-17 Software Is Riddled With Bugs". The article was a newsbyte
that the newspaper reporter came up with on a report released by the US
GAO (Government Accounting Office - one of the government "watchdog"
organizations) which was somewhat critical of the software development
process on the C-17 (of interest: this report is cited in the references
of the original SEI CMM paper as an example of how many problems of
software development are attributable to "management" issues). So - not
only was our SQA department having to cope with some animosity from the
software developers, from upper management and the customer (i.e. US Air
Force) but now we were getting unwanted criticism from other government
agencies and the media. It's really a blow to your psychological
well-being when you've been doing the best job you can muster yet know
that you cannot solve every problem that faces you...)
Charles
says...
>Either testing is very important, or funding is highly limited.
>Not both, unless someone in management is either
>a) clueless,
>b) pointy-haired, or
>c) a political appointee.
The original poster did say that he "works" for the government so All Of The
Above usually applies. ;-)
_______
Charles
Vince Coen
07 May 13 17:26, you wrote to All:
> CONSULTANT AVAILABLE QA TESTING
> Thanks & Regards,
> Purushotham Reddy
> Technical Recruiter
Your systems need to be tested as they are dumping multi copies of the same
mail.
Perhaps you should employ a system tester.
Vince
