New to Testing

[Brennan Kinney]

Hi everyone :)

I'm from a developer background but have taken up a training course on Software Testing recently. As the ISTQB course material and at times the tutor do not instill much confidence I've started looking for communities and learning more about the industry. There was one thing in the course that wasn't very clear, I've caught several occasions where the terminologies for Test Case and Test Procedure get tossed around with different meanings. I've brought this up but am not getting a clear answer, I've tried looking on the net and again these two terminologies are inconsistent.

Could anyone here with actual experience in the industry help me out with understanding this? Or point me to a more appropriate place to ask? I'm thinking that a Test Procedure is a sequence/list of Test Cases to perform in order. Some are saying it's the steps to perform a Test Case, and I'm also seeing Test Cases within Test Cases, are these two terms interchangeable?

In our course material we have an example of Test Cases, two rows are Test Cases on testing withdrawal on an ATM, checking that that it only accepts multiples of $20. The first Test Case says to try with $1 and that it should not be allowed and show an error message. The 2nd Test Case does exactly the same but instead of $1, it says to test $5, $10 and $15. I'm not sure why these are separate, perhaps it's a bad example, they could all be one Test Case or they could be 4 separate Test Cases? They are all technically one Test Case being reused with a different set of Test Data as well right?

Any help would be greatly appreciated :) This industry seems quite interesting.

[Rich]

Hi Brennan, thanks for posting your question to the group. Your curiosity and ability to point out a major issue with the quality of some parts of the testing industry is welcomed.

First, you are certainly not alone with the confusion of the istqb syllabus. I would encourage you to look at other sources of testing skills such as the good work done by the folks at the http://www.associationforsoftwaretesting.org/training/courses/foundations/who do a set of skills training that may match logic and reality better.

I have done two AST courses and two istqb, and strongly recommend AST.

Second, the atm tests with regard to test procedure and case. I, too, had the same questions at a time several years back. I tackled this with the goals of being realistic and efficient. Here is how I would commonly tackle the problem.

Of course I am making the following assumptions to operate under:

1. that the problem is important to test in the first place

2. That the input values are based on some logic, requirement or specification

3. That I have no other help and I am executing this myself

4. This is to be tested manually, not automated

5. I am testing the logical business rules for data entry using a simulated software and not a real atm

6. That testing efficiency is important 

7. That documentation is less important; lean is okay

8. That using the atm software is complex and requires step by step instructions that can be reused by another 

9. The preconditions are met

  - The account has a positive balance above $x (this is a data pre-condition too)

  - the account is the correct type, currency and status ie open

To focus your test session, I would state your test idea: denomination accuracy 

And with that I would write a test procedure for the generic use case or steps involved without specific mention of the actual input data. The procedure should be as detailed as required to be repeated by yourself and/or others and would include how to navigate and enter withdrawal amounts, confirm, and "receive amount" and system feedback.

You could short cut this process by referring to training material, or a functional spec with the same workflow outlined. 

I would then list out a bunch of input/response pairs, such as the ones you mention.

Eg. 1/not accepted, 5/not accepted etc

The inputs should be based on some model of some sort. You have done something very important in your summary; that is to question. Why have you been asked to test with 1,5,10,15? I would seek to find out why. 

These critical thinking and analysis skills are practised and discussed at local testing meetup groups and 

http://weekendtesting.com/wtanz

This is how I would approach this particular problem, and may not serve the view of others. I wish you well with your curiosity in the testing space.

Richard Robinson

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Software Testers New Zealand" group.
To unsubscribe from this group and stop receiving emails from it, send an email to software-testers-new...@googlegroups.com.
To post to this group, send email to software-teste...@googlegroups.com.
Visit this group at http://groups.google.com/group/software-testers-new-zealand.
For more options, visit https://groups.google.com/d/optout.

[David Greenlees]

Nice Rich...

Brennan, it appears Rich has done what I was planning to do.  :)

DG

[Oliver Erlewein]

Hi Brennan,

Rich is pretty bang on. Question is though, is this question for real life or for the ISTQB exam? If it is the latter, then learn whatever they say off by heart and regurgitate at exam time. Just like school.

Once you're out in the real world follow what Rich said and do keep looking for communities and make up your own mind on what people say, no matter how senior.

Cheers Oliver

--

[David Greenlees]

Oliver is also bang on...

I know I said I would get to this next week, but I can handle it anymore...  ;)

ISTQB... I have the foundation, and have also done the advanced TM course. I learned a great many things from both... the most prominent learning? They are useless. Well, job ads will ask for them, so perhaps not totally useless, but... some say if the company is asking for the certification then you wouldn't want to work there anyway. That's not true all the time because HR departments are commonly the ones asking for it, not the actual team you'd end up working for. Anyway, that subject is worthy of it's own book, let alone the entire debate surrounding ISTQB... that's worthy of a series of books.

I agree with what your approach appears to be... finding other communities. This forum represents one, and there are many more out there. We have a few kicking around locally in Oz and NZ. Engage in those and learn from them. Discussions with peers will gleam far more useful information than you know what.  ;)

What area are you from Brennan? We could point you in the direction of local groups if we know of them.

DG

[Brennan Kinney]

I'm in Auckland. I came across this group http://www.meetup.com/WeTest-Auckland/ that will have a meeting on the 7th.

The AST foundations course is about 200ish USD? Or are there extra costs besides the course and membership? It doesn't appear I'd be able to enrol until September. I'm not sure if I can stay unemployed for that long. Do any companies in auckland do internships for Software Testing? Thankyou for sharing your approach to the ATM test example. Hopefully I can teach myself well enough to land an entry-level/internship position within the next month or two, if that's possible :)

Oliver, it was not a question, the ATM Test Cases I mentioned were taken from our study material where they are listed as an example. I didn't understand why there was two instances of the test case, one with $1 as the Test Data and the other $5, $10, $15. The tutor hasn't been much help as they said my understanding was right initially but then later told another student who said the exact opposite that they were right.

I'm finding the course difficult to take seriously. I'm not sure if it'll interest anybody but I've provided 3 examples of my experience so far below.

Example 1)
At the end of a chapter we are e-mailed a pdf of ISTQB questions, not all are relevant to the chapter or have been covered yet. I've since discovered these poorly written questions are very popular in indian software testing communities. When I asked a week later if we would be provided corresponding answers to check our results I was told to contact support. We also recently got these questions online via the providers portal. We are given one question at a time with the multi-choice selection. While this does provide a % correct page at the end, often longer questions are clipped preventing them from being read. Odd since it's a provider that prides itself in teaching Software Testing.


Example 2)
Prior to posting here I had sent a rather detailed e-mail to my tutor about the terminology issue I was having. I provided timestamps on the recorded lessons for her to see(apparently she cannot access them) as well as page numbers from the course material. I showed where and why confusion was being caused on the terminologies as well as sharing what I made of it. Her response was the following:

Before I respond to your questions…

 

1.  As mentioned previously I don’t have access to the recordings.

2.  What does the ISTQB syllabus say in its definition of the test case and test procedure?  As mentioned previously this is your source of definitions for the exam.  Please give me your definitions from this material and I will then respond to let you know whether you have your definitions correct.

I know what the definitions are, but it wasn't consistent with the course material examples or in our lessons. I don't think she read my e-mail properly.

Example 3)
In a recent lesson I disputed an answer provided by the tutor to a question we were going over. My answer was more efficient and oriented to how the software would work in a real world scenario, in doing so it would test for a critical bug that her answer would miss if it existed. For whatever reason my answer was wrong because of the values I chose over the benefits I stated. Values aside, it was then deemed irrelevant to the question (Write two Test Cases to cover both branches of an if statement in the example pseudo-code). My earlier issue with clarity on what a Test Case is came up and she asked the class if anyone could help me, one of them spouted the textbook ISTQB definition which I already knew. We moved on and I e-mailed the tutor after to try understand why my answer was not valid, we were after all taught to think outside the square and I thought I did just that. Instead of e-mailing a response back to help me understand why I was wrong she decided it would be better to demonstrate to the class at the start of our next lesson. I don't have access to the recordings for either lesson right now, we normally have to wait several days, this is all from memory, so I might not be 100% right with my details. The lesson began and she drew up my answer and then what was supposedly last lessons answer(though it wasn't). She stated that if she was to hire one of two people and each answer belonged to one of them she would hire the one with her answer. Now it's very foggy for me to recall the words, but she did say something along the lines of that I don't seem to be understanding this material and following along as well as the others, and that perhaps I should be put in touch with someone to talk about it. This example/discussion lasted about 10 mins and she stated something like I was holding the class up. I felt that this was very unprofessional, if the rest of the students already agreed, why was there a need to publicly address how wrong I am I e-mailed her about how I felt and she replied with "Sorry Brennan.  I was trying to show the class that this course and testing in general is about getting the best answer although other answers may fit.  And your example I could use."

[Richard Robinson]

Hey Brennan, thanks for sharing those experiences. I think you can see why there are groups in the software testing community that are fed up with the istqb and some associated courses. I am unsure about testing internships in Auckland. WeTest Auckland should be a great group to network through. I am currently based in Sydney, and run a similar type of monthly meeting.

Looks like the cost of joining AST is $95, and the foundations course is $125. That is a good price compared to many. However, it sounds as thought your priorities are paid work at the fore front. 

As far as the course materials go, they are all freely available. There are video lectures you can watch, and documents you can read too. That may go some way to seeing a different way of learning about testing.

http://www.testingeducation.org/BBST/foundations/

When I did the istqb foundations course I had similar experiences to you. I would regularly argue with the instructor about some of the material, and especially the questions.

I remember debating one such question for an hour, and some of the class began to agreed with me. In the end, the instructor said "it doesn't matter what your reasoning is, just that the answer is best based on the syllabus. Therefore, the answer is B".

To which I replied, "If this question comes up in the exam, I will still put D, because I believe it to be the best answer".

This type of learning is incredibly frustrating.

Good luck with the AST BBST Foundation materials. I hope they nourish your interest and help you to forget your istqb experience.

Cheers

Rich

--

[Brian Osman]

H

--

[Brian Osman]

Hi Brennan,

Thank you for sharing your experiences and questions - you ARE in the right place! The issue with the ISTQB syllabus (and by extension the glossary) is that is assumes that what is stated is the one true way.

For example, test case according to the ISTQB glossary is "A set of input values, execution preconditions, expected results and execution 

post conditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [After IEEE 610] " which might seem reasonable at first glance but is it? The question(s) could be reasonably asked is what is meant by a test case? Is it a set of inputs (such as above)? Is it a model? Is it an instance of a test idea (Bach)? As part of the requirements analysis process (some flavours of agile)? A high level document bordering on scenarios?

As you can see there is no right way to define what a test case is because it is dependent on the terminology USED in your project or organisation. My current project project uses the term test case which loosely follows IEEE610 definition above. HOWEVER, we have also use the label test case to mean 'high level scenario'. An important part of using a label is to communicate and idea so that others will have an understanding of the 'weirdo' that you talking about! :)

With regards to your particular scenario..." The first Test Case says to try with $1 and that it should not be allowed and show an error message. The 2nd Test Case does exactly the same but instead of $1, it says to test $5, $10 and $15. I'm not sure why these are separate, perhaps it's a bad example, they could all be one Test Case or they could be 4 separate Test Cases?"

The answer is yes :) - It could be 1 test case or it could be 4 - its entirely up to you and your context. You will find that out in test scripting land that they may be 4x separate test cases in which case the example above is a poor one (this is a quick answer without a better understanding of the context of the question presented).

As my esteemed colleagues have already suggested, there are community wide initiatives to attend to help in with learning the craft of software testing (WeTest meetup, this forum, mentoring session (just ask on this forum), AST, Lets Test OZ and so forth) that will help you become a better tester.  

ISTQB does have some useful information in the small but in the large if falls down as something that is irrelevant, out of date and shallow in the practical context of software testing. 

On Sat, May 3, 2014 at 12:58 AM, Brennan Kinney <brennan....@gmail.com> wrote:

--

[Oliver Erlewein]

Hi again,

Just for the tests proposed by ISTQB. I'm assuming you're quoting correctly. Given that the ATM can only deliver in multiples of $20, it doesn't make sense to test $1,$5,$10,$15! Or rather they are really low value tests. Here is what I'd suggest.

a) find out what upper limit there is and if that can be set individually to an account

b) find out the actual paper denominations for the country in which this is tested (is it the only country???)

c) find out if there are any technical limits that might apply (e.g. field lengths in DB/coding language/OS)

d) technically how many bills can the ATM actually deliver to the client? There IS a limit!

e) what about cases where ATM has run out of money mid transaction?

f) what are the ATM failure scenarios and reactions? This is in software and physical, where it pertains to the use thereof.

e.g. NZ

I'd suggest the following tests:

1) -1$. 

2) Any of $5 and/or $10 (valid denomination)

3) $20 minimum payout

4) $40 more than one bill

5) $260 (more than 255, which is the integer limit) and maybe also $140 as some signed integers end at 127/128

6) $1040 (1024 is another multiple that can be of interest)

7) A test case just under the limit of the account and one over (failure test).

8) Negative cases: -$1 (This is probably not possible via interface. If not, then test is done or you will need to go in via the back-end to test. This can be viable to check for malicious use.), $19 (rounding ceil?), $21 (rounding floor?), $30 (to test if that gives you one $20 bill or an error), ATM limits,....

And this is just what I come up with in the 3min I have been thinking about it. So there will be tons more that I encourage anyone to add to this. 

So giving you $1,$5,$10,$15 is a bit sparse and if there was no explanation as to what the test idea behind those numbers was ...well... make up your own mind there.

And if someone asked me to note more than just my test ideas I'd probably blow my top. Those tests would be straightforward as to execute. What is of value here (and with every test you ever do!) is the reasoning why you did the test. What was the key idea that led you to that test?

There are several mnemonics and methodology tables that can help in running through these and coming up with ideas.

Maybe have a look here and then expand on that.

http://thetesteye.com/blog/2011/09/the-little-black-book-on-test-design/

Cheers Oliver

[David Greenlees]

Perhaps it was a REALLY stupid way to call out equivalence partitioning? In ISTQB speak the 1, 5, 10, and 15 are all part of the same partition (1-19). From memory I don't believe they (ISTQB) suggest testing more than a couple within the same partition, so not sure why they want 4. So perhaps 1 is also supposed to be a boundary test... which it isn't, but I wouldn't be surprised if they think it is!

I've just noticed the smoke from my ears.... I'm walking away now...

[mart...@gmail.com]

Hi Brennan,

It seems to me that the root cause of the problem you outlined is a poor quality trainer. And if the course material just focuses on learning ISTQB terminology to get through the exam, you haven't been taught much about being an effective tester. My recommendation would be to look for a top-notch trainer (Brian Osman is in that category) and course material that goes beyond ISTQB and actually teaches you how to test.

Regards,

Martyn

[Tessa Benzie]

If you're looking for alternative training options I would suggest something like this: http://courses.assurity.co.nz/courses/55-essential-testing. I haven't done this course but am basing this solely on my knowledge of the people who designed and created it. (Katrina Clokie, Aaron Hodder and Adam Howard please stand up).

Not sure if it would be suitable for you just yet but it's worth keeping in mind.

Cheers

Tessa

--

[mart...@gmail.com]

Well, if we're into handing out specific recommendations, IMHO you ought to look at the proven software testing courses from Software Education too!

Regards,

Martyn Jones

    

On Saturday, 3 May 2014 02:58:47 UTC+12, Brennan Kinney wrote:

[Brian Osman]

Hi all,

Tessa and Martyn are both spot on - Assurity and SoftEd both run excellent courses with excellent trainers. There are the odd independents running around as well that provide *quality* training. 

In NZ (and possibly OZ tho' i'm sure ive missed some people like Anne Marie Charrett), other training organisations in the software testing space are a distant third (including places like Universities) which is a pity and disturbing at the same time.

@Brennan - in would be interesting to hear your thoughts on the comments provided so far :)

Cheers

Brian

--

[Tessa Benzie]

@Martyn - of course! Sorry, didn't mean to be stepping on toes. I've done many training course with SoftEd over the years; all of them beneficial.

[Andrew Robins]

Hi guys,

Late to the party on this one. Brennan, as others have pointed out usage of terms will vary from context to context. But if you want a quick heuristic rule (probably useful but not guaranteed) then you could use:

A test case is an artefact. (An instance of a test)

A test procedure is a process (A process that can be applied when testing)

Like others here, I prefer to think about test ideas. These are sometimes expressed as test cases, and sometimes explored via test procedures - (where I think these are worth capturing).

Re the selection of the values for the ATM test case that the ISTQB syllabus identifies... What you are seeing there is the output of a typically shallow analysis process which I call "plucking ideas out of your arse". An even cursory analysis of the values selected shows that the person who wrote the example test cases did not even try to apply the tools contained in the syllabus (equivalence partitioning for example) in the process of selecting values for their test case.

It is for reasons like this (and many others besides) that people like myself and many others in this group, do not place any value at all on ISTQB training.

Cheers and good luck.

Andrew 

Test Manager, Tait Radio Communications

---

This email, including any attachments, is only for the intended recipient. It is subject to copyright, is confidential and may be the subject of legal or other privilege, none of which is waived or lost by reason of this transmission.

If you are not an intended recipient, you may not use, disseminate, distribute or reproduce such email, any attachments, or any part thereof. If you have received a message in error, please notify the sender immediately and erase all copies of the message and any attachments.

Unfortunately, we cannot warrant that the email has not been altered or corrupted during transmission nor can we guarantee that any email or any attachments are free from computer viruses or other conditions which may damage or interfere with recipient data, hardware or software. The recipient relies upon its own procedures and assumes all risk of use and of opening any attachments.

---

[Oliver Erlewein]

I also suggest the RST course from SoftEd when its on.

[Brennan Kinney]

Apologies, I wasn't being notified of new responses which I thought I had set the topic to do.

Oliver, just saw your name on a recent blog post by James Bach :) The scripting side of testing sounded interesting to me as I'm from a developer background. Oddly my tutor discouraged my answers when I tried to improve one she gave with a developers perspective. The ATM being tested is used throughout the course for exercises and examples. The specific two test case examples mentioned previously were shown in an exercise that had about 5 test cases with the following fields Test Case ID, Requirements ID, Priority, Test Condition, Expected Results, Procedure. We had two groups of test cases to look at and decide if there was inadequacies in the information provided, such as lack of traceability to requirements or expected results not expecting an error message to show when the requirements say they should. We had two requirements, one was that withdrawals are only valid in multiples of $20 and the other a daily withdrawal limit of $500. It would be much more extensive in the real world I understand that, this was an exercise/example to work with to see if we could spot was wrong.

I'd have to go over the recorded lesson to be sure, but I don't recall the tutor explaining the reasoning for the two similar test cases. This was possibly due to me bringing up my confusion with test procedures terminology as she stated the two terminologies definitions, then later redefined them again as the opposite of what she had said previously. As we were being taught this formal way to document/design tests I didn't like having them written/stored in a word/excel document, I was thinking they'd be better off dynamic stored in a database (possibly graph-based Neo4j) where I could treat a test case as an object that could be instanced, such as for testing $1, $5, $10, $15 inputs, either via a script or displaying them to the tester. As we learnt that we might not be able to do all our tests due to time constraints we'd need to prioritize tests by risk and being less extensive in some areas, I felt with the database/app I could easily filter the tests based on values like tags/risk/tier. The tutor made this difficult as she would say that I could not treat a sequence of test cases/instances as a procedure that had conditional branching. We later learnt about use cases and how these are mapped in the syllabus which shows exactly that. The approaches really do seem familiar to a programming language, she likely just understand that.


Here is an example from our course material that we had an argument on where I was told she would hire someone with her answer over mine because hers was the better answer.

Pseudo-code:
check_wd_amt
 tot_daily_wd_amt = daily_wd_am+wd_amt;
 if tot_daily_wd_amt >500
  perform daily_wd_error;
 end if;
check_wd_amt_end;

"Your assignment is to create two test cases, that will test each branch in this section of code." The topic is decision testing and coverage. Her answer involved two tests that changed the daily_wd_amt and the wd_amt to produce a value less than and another greater than 500. I understood this but argued that in a real world usage daily_wd_amt will be increased on a valid withdrawal and that she wasn't testing for that which could be a critical bug. I know it wasn't contributing to the specific exercise requirements but didn't see the harm in testing it. I proposed that we just control the wd_amt in the code suggesting $300(I didn't put much thought into the value which got more attention than the benefit I had mentioned, I later suggested $260). This test could be run twice in a row expecting an error message the 2nd time as daily_wd_amt should have exceeded $500. On the following lesson where the tutor felt it was better to demonstrate to the class that I was wrong instead of explain by responding to my e-mail, her example had changed to be my answer but with the values of $500 and the 2nd as $20, she illustrated mine as $260 followed by $260, she ignored me calling her out on that not being the same answer she provided in the previous lesson. Blame was put on to me for holding up the class for ten minutes and that I don't seem to be doing as well as the others who know what a test case is (glossary citation).

@Andrew, this example/exercise was done prior to us learning about partitions, boundary and other techniques. The course material we went through chapter through chapter had a bit of back and forth, I don't think it's laid out very well.

I'll look into those training alternatives, I unfortunately do not have much in savings as I worked very cheaply for experience(and I was hoping a proper contract/position after). If the community has any influence, perhaps they could discuss with The Ministry of Social Development a better education provider to fund training for future unemployed kiwi's? While the one I had was unprofessional it did introduce me to a career path I wasn't aware existed :) Hope to meet some of you at the WeTest meetup in auckland tomorrow!

I also got an unexpected welcome e-mail from James Bach the other day which was pretty cool!

[olivernz]

Hi,

In context also have a look here! http://t.co/720vYXC7FD

That is an interesting and creative way to look at boundary analysis. (also have a read of developsense.com for much much more)

Cheers Oliver

[Joshua Raine]

While I haven't read it (yet), a relatively new (Oct 2013) resource on boundary/equivalence testing to look at would be "The Domain Testing Workbook" (Kaner, Padmanabhan, Hoffman), see http://kaner.com/?p=363 or google for more.

[Nicola Owen]

I'm going to keep this short and sweet.

From my experience, a lot of Testing Terms are interchangeable. There doesn't appear to be a strict right or wrong in a lot of terms like test cases, test scripts etc etc. It depends on where you work and who you talk to :)

[Brennan Kinney]

Understandable, it just made things difficult when trying to learn whats what and understand the process/workflows. The lack of a proper standard caused confusion especially since the tutor I had would use the terms for previous terms redefining what we had been taught, denying that she had done that in the past and ignoring any evidence of that, and then repeating this behaviour instead of explaining that they are interchangeable.

I'm sure it's much less of an issue once you land that first job in testing and everyones roughly communicating with the same understanding/use of the terminology, followed by adapting to it all over again I guess if you change to another company. Just a learning bump mostly due to the tutor and that it made conversations with the other students in the class difficult since we had sorta adapted the terms to what made sense for us as individuals causing issues when discussing testing.

--
You received this message because you are subscribed to a topic in the Google Groups "Software Testers New Zealand" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/software-testers-new-zealand/6_VC-wVtUlA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to software-testers-new...@googlegroups.com.

[srinivas kadiyala]

Hi Bren,

It is also like, when during interviews - you tend to say: what u understand about test case as a definition but, we might not known -what they are exactly referring to.

Hiring Managers - Should see, how we think with practical test ideas rather than debating on the particular definition.

cheers,

S.K.C

To unsubscribe from this group and all its topics, send an email to software-testers-new-zealand+unsub...@googlegroups.com.

[Andrew Robins]

I have also found it is always useful to try and understand why it is that people use terms differently from the way that I do.

You can sometimes learn some interesting new perspectives.

Cheers

Andrew

--

You received this message because you are subscribed to the Google Groups "Software Testers New Zealand" group.

To unsubscribe from this group and stop receiving emails from it, send an email to software-testers-new...@googlegroups.com.

[Richard Robinson]

I usually poke fun at how many different words there might be for
something. In the land of web services testing, with regard to a
particular web service, we can talk about the following terms to
sometimes mean the same thing. They originate from formal definitions,
tool wording, and 'how we talk around here':
- api
- web service
- service
- operation
- xsd
- wsdl
- schema
- interface
- request

The problem here is that the words used really depends on what the
task is. I have to teach my testers what they mean and how they change
depending on the task... tricky for new testers for sure.