The way to improve searching for existing test cases and automated tests

[jbe...@soldevelo.com]

Hi all,

for the last few days, I have been working on OLMIS-5550 in order to find a workflow/pattern which will allow us to easily track which features and cases are already covered by tests in order to avoid duplications, and those that are not covered yet. I also need to document this pattern in an appropriate place. I have come up with the solution below so far, and I would appreciate your feedback very much.

I think that we need to create and then, add suitable labels which will serve as keywords concerning a given test case's content to all test cases. For instance, e.g. OLMIS-4259 could be labelled with the following labels: "Orders", "StockCards", "Fulfillment" and "LocalFulfillment". I browsed through all test cases with the "To Do" and "Roadmap" statuses in our system, and came up with the following labels (some of them already exist on our Jira):

1. Adjustments
2. Administration
3. Alerts
4. Automated
5. Autosaving
6. Calculations
7. Catalog
8. CCE
9. Comments
10. Configuration
11. CT (i.e. Contract Test)
12. DemoData
13. Email
14. EmergencyRequisitions
15. Equipment
16. ExternalFulfillment
17. Facilities
18. FacilityTypes
19. FHIR
20. Filtering
21. FT (i.e. Functional Test)
22. FTP
23. Fulfillment
24. FTAPs
25. GeographicZones
26. ISAs
27. LocalFulfillment
28. Login
29. Notifications
30. Offline
31. Orders
32. Password
33. Permissons
34. PoDs
35. PhysicalInventory
36. ProcessingPeriods
37. ProcessingSchedules
38. Products
39. Profile
40. Programs
41. Reasons
42. Receipts
43. RegularRequisitions
44. Reporting
45. Requisitions
46. RequisitionGroups
47. Roles
48. Searching
49. ServiceAccounts
50. Shipment
51. Sorting
52. StockCards
53. StockManagement
54. StockOnHand
55. SupervisoryNodes
56. SupplyLines
57. Tags
58. Templates
59. UI
60. Users
61. VaccineSBR
62. VVM
63. Workflow

I think that using such labels will make searching for suitable test cases much faster. If you agree with this idea, I propose that we create the following follow-up tickets:

• The ticket to browse through existing contract and functional tests and then, browsing through the manual test cases and labeling them with the "Automated" and "CT"/"FT" label, depending on the type of automated tests - In my view, we need to have an easy way to check what features are covered by automated tests and I think this would be the most effective one to achieve this. Also, during my research, I learned that it is done thus in some projects;
• The ticket to create a new section in the "Test Strategy" document concerning the labels and describing how to use them with the test cases. Alternatively, a separate Confluence page could be created;
• The ticket to browse through all manual test cases in the system and label them with the new labels.

Regards,

Joanna

[wes....@villagereach.org]

Thanks for this, Joanna! 

I like the idea of organizing our test cases but it seems like you are mixing different concepts into the same field and I wonder if that will make this confusing both to categorize and to search. For example, the list contains functional areas (StockManagement, Requisitions), actions (Sorting, Searching), state (Offline), and more broad categories (Configuration, UI). Also, given the length of this list, it seems unlikely that people will remember to only use these items which will likely result in a many other labels being created that only really make sense to the person doing the labeling.

Based on that, I have a couple of suggestions:

1. Can we create a more simplified list with fewer, more broad categories? 
   
2. Can we use the existing Component field in JIRA to track the functional area rather than duplicate this information as a label?
3. Can we create a custom field for this test case categorization rather than reuse the Label field?

Thanks,

-Wes

[Sebastian Brudziński]

In addition to what Wes has stated, I don't think it makes sense to have labels/categories for our services - requisition, stockmanagement, etc. We are already using the component field for this, so it seems like an unwanted duplication.

I like the idea of having a predefined list of categories/groups rather than using labels too. That should make both searching and adding them to new test cases easier (Jira should be able to generate that field as a dropdown then)

I don't exactly get what you are referring to when you say we should label contract/functional tests. The goal is to drop manual test case as soon as we fully cover given case with an automated test. We also won't have a related ticket with each functional/contract test. Can you clarify this a bit?

I'm mostly interested in whether after those changes you will be able to easily find a test case or say with confidence that a case is not covered for any given feature in a short time (like less than 5 minutes). Do you think this categorization is enough to achieve that?

Thanks for writing this down!

Best regards,

Sebastian.

--
You received this message because you are subscribed to the Google Groups "OpenLMIS Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openlmis-dev...@googlegroups.com.
To post to this group, send email to openlm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openlmis-dev/7b723f6e-bd58-4c21-8247-8fd5e8a2bcac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sebastian Brudziński
Technical Leader
sbrud...@soldevelo.com

SolDevelo Sp. z o.o. [LLC] / www.soldevelo.com
Al. Zwycięstwa 96/98, 81-451, Gdynia, Poland
Phone: +48 58 782 45 40 / Fax: +48 58 782 45 41

[jbe...@soldevelo.com]

Hi Wes and Sebastian,

thank you both for your suggestions!

I managed to decrease the number of the proposed key words - I excluded those that overlap with the "Component" and "Label" fields, and those that are too specific. Instead, I left only those that pertain to specific screens or features that will make it easier to distinguish the test cases from each other. I think the categories can't be broader, as those under the "Component" and "Label" fields are broad, and they don't render searching for the test cases fast enough - they are not specific enough and there are too few of them. Though of course I think we should still keep using them, along with the new custom field. I think the latter should be called "Keyword", and it should be a drop-down list from which one could select suitable options, as Sebastian suggested, so that we don't create countless options, as is the case as far as the "Label" field is concerned.

As for the "Automated", "CT" and "FT" keywords, I thought that we should label the tickets with manual test cases with these labels/keywords, so that we know what features that had been previously covered with manual tests are now covered with functional/contract tests - it's easier to track it thus then with the use of the links on Jira between the ticket with a manual test case and e.g. the one concerning the creation of a contract/functional test. This could be helpful in making next steps in test automation, e.g. to determine the priorities - for instance, to tell which functional/contract test should be created first. Also, probably this could help the person who would be creating such an automated test - they could check whether there is e.g. an automated test for a similar feature created already, and look at the steps contained in the manual test case, which also could be some tip on how to write the automated test. But if you think that marking the manual test cases with the labels/keywords concerning automated tests is not necessary, I will exclude those keywords.

All in all, in my view, adding such a custom field to our Jira would quicken looking for suitable manual test cases greatly - I think that after introducing the field and marking the test cases, searching for manual test cases concerning a given feature would take up to 5 minutes, even for a person who is not familiar with our test cases.

The current list of keywords is the following:

1. Adjustments
2. Alerts
3. Automated
4. Calculations
5. Catalog
6. CT
7. Email
8. EmergencyRequisitions
9. Facilities
10. FacilityTypes
11. FT
12. GeographicZones
13. ISAs
14. Login

1. Password
2. Permissons
3. PoDs
4. PhysicalInventory
5. ProcessingPeriods
6. ProcessingSchedules
7. Products
8. Profile
9. Programs
10. Reasons
11. Receipts
12. RegularRequisitions

1. RequisitionGroups
2. Roles
3. ServiceAccounts
4. Shipment
5. StockCards

1. StockOnHand
2. SupervisoryNodes
3. SupplyLines
4. Tags
5. Templates

1. Users
2. VVM