Defect Categorization -- should we be assessing more than just Severity?

[Global QA Team]

Smith, Patrice

Dec 1 2011

Today I reviewed the new proposed version of the System Test Plan Template (v1.20).   In the Defect tracking section, I wondered whether we should add another level of evaluation for each defect.   We currently track Severity levels (listed below from the Test Plan Template).  

Should we also track the likelihood of hitting the defect? 

 

My thinking is that a Sev 3 defect that we expect users to hit 70% of the time needs a different resolution timetable than one that we expect users to hit infrequently.   I don't think we're currently capturing this information.   

 

I would think that we could divide the levels by something like (numbers and text could change but you get the idea):

 

• Minimal - users will encounter the issue with less than 20% frequency or only very specialized users will hit the issue
• Significant - The average user will hit the issue 20 - 50% of the time
• Major - The average user will hit the issue 50-70% of the time
• Crippling - The average user will hit the issue more than 70% of their sessions using the product

I'd be interested in comments, maybe some teams are already capturing this information.    The EB projects that use StarTeam for defect tracking use a Priority field to indicate defects that are higher priority than others but it's a simple Yes/No field which doesn't provide the level of granularity I would want. 

 

I got this idea because I attended a UXD Usability clinic.  One of the items they touched on is a two-prong rating system to determine priority for fixing issues:  the combination of an issue's Severity with its Expected Impact Frequency.  

 

From the System Test Plan: 

 

Defect Severity Level Definitions

Severity 1: Critical- system crash, massive performance degradation, data corruption, data loss, security violation

Severity 2: High- operational error, data integrity, some performance degradation, loss of functionality (no workaround)

Severity 3: Medium- same as Severity 2 except there is a workaround

Severity 4: Low- minor problem, misspelling, UI layout, rare occurrence