Recommendation for testing block interactions

[Gabriel]

Hi, I currently have unit tests in place for my generators and some basic coverage for custom blocks. I’d like to extend this by testing block interactions, such as opening a mutator and adding elements, interacting with the toolbox and verifying that values persist after reload.

Do you have recommendations for testing these kinds of interactions? Most plugins I’ve seen rely only on unit tests and mocks, but would it make sense to include browser-based tests (e.g. with Playwright) for more complex scenarios and interactions?

[Christopher Allen]

Hi Gabriel,

 

Hi, I currently have unit tests in place for my generators and some basic coverage for custom blocks. I’d like to extend this by testing block interactions, such as opening a mutator and adding elements, interacting with the toolbox and verifying that values persist after reload.

Do you have recommendations for testing these kinds of interactions? Most plugins I’ve seen rely only on unit tests and mocks, but would it make sense to include browser-based tests (e.g. with Playwright) for more complex scenarios and interactions?

I'm not sufficiently familiar with the alternatives* to make any recommendations here, but I note that Blockly itself is tested using a combination of Mocha (test framework), webdriverio (for browser testing) and chai (assertions library).  We have a few different sets of tests, but the two main ones are:

• The "mocha tests", in tests/mocha/ which are are an extensive set of (mostly) unit tests of various components of Blockly.  In this case Mocha runs in the browser, and the tests can be run manually simply by loading the relevant index.html file.  They are also run as part of npm test by using webdriverio to fire up a browser, wait for the tests to complete, and check to make sure they are reported as passing.  Though they are run in a browser they are not interaction tests, and many of them do not require the DOM and could in principle be headlessly using node.js.
• The "browser tests", in tests/browser/, began life as a manual test plan which has been (partially) automated.  For these tests, The Mocha test suite runs in node.js, and the tests use webdriverio to interact with Blockly running in a browser in the same way that a user might.  These are quite slow (though recent experience with similar tests in the blockly-keyboard-experimentation repo suggests they could likely sped up considerably), so are run periodically from a GitHub workflow rather than as part of npm test.

A few observations—and personal opinions not necessarily shared with my colleagues—about our existing set-up:

• We use Mocha (TDD API rather than BDD) as a test framework, and chai for assertions.  They are fine but IMHO it's preferable to have as minimal and simple test infrastructure as possible: as the Go language team observe, use of tools like these tends to result in is the creation of mini-languages just for testing, when it's probably preferable to write tests in very plain Go/JS/TS.  That also helps avoid adding dependencies, which carry a long-term maintenance burden.
• The mocha tests do not do any interaction testing, and could probably be reengineered to run in node.js (using jsdom to support testing of components that require the presence of the DOM).  Being able to run these tests without a browser would greatly simplify our test infrastructure, and make the tests faster and less flaky.
• My own experience of writing interactive tests using webdriverio has not generally been very pleasant, and given the team's underwhelming enthusiasm for writing more such interaction tests I suspect I am not alone in this.**
• The big advantage of using webdriverio is that the test harness runs entirely outside the browser, so tests can simulate user interaction very accurately.  This is great for black-box testing.
• The big disadvantage  of using webdriverio is that the test harness runs entirely outside the browser, making it difficult for the test harness to introspect (or modify) the internal state of the app being tested.  This makes white-box testing much more difficult, and even for black-box testing the complexity of inspecting the DOM state from afar means that one spends quite a lot of time just figuring out how to measure the result of the test.

If I were undertaking to choose an approach to doing such browser interaction testing from scratch, I would want to consider:

• How much do I care about being able to do black-box testing of my app on many different platforms?  Using webdriver makes that possible—but the disadvantage is that writing tests is inevitably going to be a bit painful for the reasons outlined above.
• For some key tests (particularly any involving browser navigation) it may be worth it, but for many interaction tests a more ergonomic approach might be to do white-box testing set as our existing mocha test suite is, where the tests and the code under test run together in the same runtime (page).  This would lose some fidelity (you can't can't reliably simulate all possible kinds of user interaction from within the browser) but for many tests the advantages of being able to directly introspect the state of the app from the tests would make writing such tests much easier and more pleasant.
• Unfortunately it's not possible to use use webdriverio in the browser to have a page to control itself—unfortunate because that would make white-box interaction testing way easier, with the tests and code being tested running in the same run time (as our mocha unit tests currently do) while still having all the flexibility to simulate user interaction that the webdriver protocol provides.
• On the other hand, there are test frameworks such as Cypress which, though they still have a node.js script controlling the browser, claim to "directly in the browser, you can debug failed tests using the in-browser developer tools".  I've not tried it but that sounds like it would make developing tests much easier.

I'd encourage you to try writing a few sample tests in each of a few different frameworks (selenium-webdriver, webdriverio, Playwright, Cypress…) and see which was most pleasant to use (as well as whether any were missing key features) before making a final choice.  After all, the easier and more pleasant it is to write tests the more likely one's app is to be well tested…

Best wishes,

Christopher

---------

* Notably including Playwright, which if nothing else certainly appears to have a very pretty, very GitHub-inspired UI.

** This reply is longer than necessary in part because the next thing on my to-do list is writing more such tests.