QA of QAtrack

[Robin Cole]

I have been asked to consider what QA the department should be performing on QAtrack itself. For instance, what QA should be performed after upgrading to the latest version of QAtrack?

I would be grateful to hear any examples of QA on QAtrack, or to hear general thoughts on the topic.

Cheers

Robin

[Randle Taylor]

Hi Robin,

This is a good discussion topic, so thanks for bringing it up.  

QATrack+ does have a fairly large automated test suite that continues to grow with every release.  The automated test suite is supplemented with ad-hoc testing by developers and also a few physicists in Ottawa.  Obviously automated tests are not always sufficient to catch all issues though (particularly errors in logic as opposed to implementation).

If possible, before updating a clinical database, I would highly recommend clinics clone their database, create a parallel Python environment and do a dry-run of the upgrade on the cloned database.  This will allow you to discover and remedy any potential issues encountered during the upgrade before the actual clinical database is modified.   I hope it goes without saying that a backup of the database is always created before attempting an upgrade on a clinical database!

Depending on the number of test lists in your database, it's likely impractical to perform every single test list to confirm they are functioning as expected after an upgrade.  Selecting and performing a subset of test lists that cover the different test types in use in your clinic can help build confidence that the new version is functioning as expected. In particular, test lists with composite or upload tests deserve a bit of extra scrutiny since they are *by far* the most likely tests to have issues.  Since I have little/no control over how users write their calculation procedures it is impossible to cover all scenarios  with automated tests.

When I was in Ottawa (and this may still be the case) we would usually set up a completely independent QATrack+ instance with the new version which would generally be tested by a few physicists, technologists and usually a therapist.  With this approach, we never ran into any major issues with upgrades on clinical databases since they were caught and fixed ahead of time.  This level of rigour is probably not practical for many centres, but I hope generally people can be confident that there are no major bugs lurking by virtue of having a large centre like Ottawa leading the way with upgrades to new versions.

Definitely interested in hearing other opinions on this topic! If people have ideas about how we can push for even better quality assurance of the software in the future I'm all ears!

Randy

--
You received this message because you are subscribed to the Google Groups "QATrack+" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qatrack+unsubscribe@googlegroups.com.
To post to this group, send email to qat...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robin Cole]

Hi Randy

thanks for the detailed response. A couple of follow up thoughts.

1. Is it possible to get detailed info on the testing performed by the devlopers/physicists on the latest release? This would save us repeating any of their work.

2. The admin section of QA track doesn't make it particularly easy to get an overview of the relationship between components. Would it be possible to make a visual representation, e.g. something like 

http://softwarerecs.stackexchange.com/questions/11346/tool-to-visualize-sql-database-schema

3. I have been considering options for automated testing of QAtrack tests, in particular how to perform a 'consistency' test, comparing the results of data entered into the old and new versions of QAtrack. One idea I had was to use something like pythons Requests package to post data to and query responses from both databases, and compare their results. This could then be reused for routine QA of QAtrack. I could use https://github.com/bastula/qatrackimport as a starting point. Any thoughts on this approach? 

Cheers

Robin

[Randle Taylor]

1.  I can tell you what I do but can't speak for the physicists in Ottawa.  First, we have a fairly extensive test suite (currently 200+ tests and growing) that is automatically run on a variety of operating systems and database combinations (see https://travis-ci.org/randlet/qatrackplus-ci (Linux) and https://ci.appveyor.com/project/randlet/qatrackplus-ci (Windows)) .  This test suite helps us prevent regressions in the code and touches 90+% percent of the Python code in the file.  It does include a few tests that use the Selenium library to launch a browser window and simulate user clicks for some common tasks...definitely an area I will be expanding on though.

Aside, from that I just do ad hoc testing of setting up test lists, performing QA etc.  A more formal checklist of things to test in this manner would be a great addition to the testing protocol!

2. There are [schema diagrams](https://bytebucket.org/tohccmedphys/qatrackplus/wiki/v/0.2.9/developers/schema_v0.2.7.png?rev=69b9d7494579f9cb3dca8556ae7fadd2d328a778) but they are outdated although they should still give you the general idea.  If you're using SQLServer, it can very easily generate these diagrams for you and as your link points out there's a bunch of other tools that can do the same (with varying levels of convenience!).  Updating developer docs has been on my todo list for quite a while but keeps getting pushed further down the list!

3.  A lot of the tests in the test suite do test the full request-response cycle.  I think, the place where this kind of extra automated testing would be most useful is for doing QA on any test lists that involve composite tests.  I obviously can't check everyones tests that they are using in production so automated consistency testing like this for your actual calculations could be useful: it would allow you to be more confident that say updated 3rd party libraries haven't broken any of your calculated tests.

Hope that helps!

Randy

[Robin Cole]

Thanks Randy!

 

Re point 2, my database point man had this response - This is possible (schema diagrams or our QAtrack), however for some reason QATrack uses table types MyISAM which do not enforce foreign keys constraints. This means that the actual foreign keys are not represented in the diagram. At this stage, the only method I can see to allow a proper export of the schema with foreign keys is to do a clean install of QATrack somewhere and use the postgresql or sqlite version of the database and then reverse engineer these. 

What is your advice?

Cheers

Robin

[Randle Taylor]

For better or worse QATrack+ and Django are mostly database agnostic and we do not specify whether to use the MyISAM or InnoDB storage engine. MyISAM was the default engine prior to MySQL 5.5.5 while InnoDB was the default after that, so your database was probably created with some version of MySQL <= 5.5.4.   

If you want you could convert all MyISAM tables to InnoDB.

If it's just the schema you're after, and you can wait a couple of days, I can fire up a SQLServer VM and generate the schema diagram sometime over the next week.

Randy

--

[Matt B]

Hi,

Just wondered if there was a schema available for v0.2.9.1 ?

Cheers,

Matt

On Monday, 24 October 2016 19:00:39 UTC+1, Randle Taylor wrote:

For better or worse QATrack+ and Django are mostly database agnostic and we do not specify whether to use the MyISAM or InnoDB storage engine. MyISAM was the default engine prior to MySQL 5.5.5 while InnoDB was the default after that, so your database was probably created with some version of MySQL <= 5.5.4.   

If you want you could convert all MyISAM tables to InnoDB.

If it's just the schema you're after, and you can wait a couple of days, I can fire up a SQLServer VM and generate the schema diagram sometime over the next week.

Randy

On 24 October 2016 at 04:22, Robin Cole <robma...@gmail.com> wrote:

Thanks Randy!

 

Re point 2, my database point man had this response - This is possible (schema diagrams or our QAtrack), however for some reason QATrack uses table types MyISAM which do not enforce foreign keys constraints. This means that the actual foreign keys are not represented in the diagram. At this stage, the only method I can see to allow a proper export of the schema with foreign keys is to do a clean install of QATrack somewhere and use the postgresql or sqlite version of the database and then reverse engineer these. 

What is your advice?

Cheers

Robin

--
You received this message because you are subscribed to the Google Groups "QATrack+" group.

To unsubscribe from this group and stop receiving emails from it, send an email to qatrack+u...@googlegroups.com.

[Randle Taylor]

Long overdue! Up now here in zoomable SVG: https://bitbucket.org/tohccmedphys/qatrackplus/wiki/v/0.2.9/developers/schema

Thanks for the prompt to get this up finally.

RT

To unsubscribe from this group and stop receiving emails from it, send an email to qatrack+unsubscribe@googlegroups.com.