A Chat on Software/Data Science Testing

[Chang Lee]

How it got started?

I put out a blog post on pytest and a discussion started in the Slack Channel. Besides the testing frameworks, how and what to test for in developing a product was a question that had been haunting me for a long time. 

When you have a question, the best way to get an answer is...ask people. So I decided to set up a penny chat over lunch time and learn from everyone's experience on testing.

Who's in the chat

• Alex Antonison
  
• Chad You
  
• Chang Hsin Lee
  
• Rob Harrigan
  
• Scott Burns
  
• Stephen Bailey

6 people representing 5 different teams!

What we talked about

1. Testing applications

Rob started by talking about how to test applications—

• Separate business and functional layers. Write more functional tests.
  
• Makes it safe to refactor by knowing what the assumptions are.
  
• Test coverage as measure (Codecov,  coverage.py in runnable with Python)
  
• Setting up mocks. If anyone has good resource please provide
  
Getting Started with Mocking in Python

2. Testing data pipelines

Then we talked about other aspects of data science. In particular, how can we test data?

• Great_expectations
  
• Testing in the context of data is closer to monitoring than software testing
  
• Stephen mentioned that there are 3 kinds of things to test:
  

1. One is for development (mostly unit tests). Very important because you’re likely to build stuff on-top of this.
2. Scheduled Reporting. Still important, reporting function should be written in a way to take any kind of data and report on that. So a test could build simple data and verify the correct report is generated.
3. One-off analyses. Likely tough to test and of little value (however, if you begin to share functions between analyses, consider extracting into a library and testing those functions).

3. Unit tests

We talked about how we write unit tests, then Rob mentioned a Python package that the developer can define and equation and test random test cases.

• Hypothesis

Note: I talked about writing assertions, then Rob pointed out that asserts can get compiled away in Python, see https://dbader.org/blog/python-assert-tutorial so it's safer to use “if not” instead of “assert” in production Python jobs.

4. How to encourage the habit of writing tests? How to justify the business value of writing tests?

I asked another question: how can you justify the business value of writing tests to the managers?

• How it’s rolled out in axialHealthcare (Alex, Chad)
• Get everyone using GitHub first. GitHub GUI is more user friendly than command-line Git for data scientists who have never done any version control (Scott). 
• Implement basic CI/CD and reject pull request if test fails. But start from very basic CI (Scott).
• Write you test example and show to your team
• Style check
• Self checking
• Continuously testing

Two great insights from Scott:

• Time is the destroyer of your applications and models. Over time, things change, and these changes will break, so testing becomes more and more important the longer you plan to use something.
• Tests are a way of formalizing “tribal knowledge” about the ways a problem or project can go wrong.
  

Additional sources:

7 Common Types of Software Testing

This review is the result of our collaboration on Google docs