Recommendation for testing block interactions

[Gabriel]

Hi, I currently have unit tests in place for my generators and some basic coverage for custom blocks. I’d like to extend this by testing block interactions, such as opening a mutator and adding elements, interacting with the toolbox and verifying that values persist after reload.

Do you have recommendations for testing these kinds of interactions? Most plugins I’ve seen rely only on unit tests and mocks, but would it make sense to include browser-based tests (e.g. with Playwright) for more complex scenarios and interactions?

[Christopher Allen]

Hi Gabriel,

 

Hi, I currently have unit tests in place for my generators and some basic coverage for custom blocks. I’d like to extend this by testing block interactions, such as opening a mutator and adding elements, interacting with the toolbox and verifying that values persist after reload.

Do you have recommendations for testing these kinds of interactions? Most plugins I’ve seen rely only on unit tests and mocks, but would it make sense to include browser-based tests (e.g. with Playwright) for more complex scenarios and interactions?

I'm not sufficiently familiar with the alternatives* to make any recommendations here, but I note that Blockly itself is tested using a combination of Mocha (test framework), webdriverio (for browser testing) and chai (assertions library).  We have a few different sets of tests, but the two main ones are:

• The "mocha tests", in tests/mocha/ which are are an extensive set of (mostly) unit tests of various components of Blockly.  In this case Mocha runs in the browser, and the tests can be run manually simply by loading the relevant index.html file.  They are also run as part of npm test by using webdriverio to fire up a browser, wait for the tests to complete, and check to make sure they are reported as passing.  Though they are run in a browser they are not interaction tests, and many of them do not require the DOM and could in principle be headlessly using node.js.
• The "browser tests", in tests/browser/, began life as a manual test plan which has been (partially) automated.  For these tests, The Mocha test suite runs in node.js, and the tests use webdriverio to interact with Blockly running in a browser in the same way that a user might.  These are quite slow (though recent experience with similar tests in the blockly-keyboard-experimentation repo suggests they could likely sped up considerably), so are run periodically from a GitHub workflow rather than as part of npm test.

A few observations—and personal opinions not necessarily shared with my colleagues—about our existing set-up:

• We use Mocha (TDD API rather than BDD) as a test framework, and chai for assertions.  They are fine but IMHO it's preferable to have as minimal and simple test infrastructure as possible: as the Go language team observe, use of tools like these tends to result in is the creation of mini-languages just for testing, when it's probably preferable to write tests in very plain Go/JS/TS.  That also helps avoid adding dependencies, which carry a long-term maintenance burden.
• The mocha tests do not do any interaction testing, and could probably be reengineered to run in node.js (using jsdom to support testing of components that require the presence of the DOM).  Being able to run these tests without a browser would greatly simplify our test infrastructure, and make the tests faster and less flaky.
• My own experience of writing interactive tests using webdriverio has not generally been very pleasant, and given the team's underwhelming enthusiasm for writing more such interaction tests I suspect I am not alone in this.**
• The big advantage of using webdriverio is that the test harness runs entirely outside the browser, so tests can simulate user interaction very accurately.  This is great for black-box testing.
• The big disadvantage  of using webdriverio is that the test harness runs entirely outside the browser, making it difficult for the test harness to introspect (or modify) the internal state of the app being tested.  This makes white-box testing much more difficult, and even for black-box testing the complexity of inspecting the DOM state from afar means that one spends quite a lot of time just figuring out how to measure the result of the test.

If I were undertaking to choose an approach to doing such browser interaction testing from scratch, I would want to consider:

• How much do I care about being able to do black-box testing of my app on many different platforms?  Using webdriver makes that possible—but the disadvantage is that writing tests is inevitably going to be a bit painful for the reasons outlined above.
• For some key tests (particularly any involving browser navigation) it may be worth it, but for many interaction tests a more ergonomic approach might be to do white-box testing set as our existing mocha test suite is, where the tests and the code under test run together in the same runtime (page).  This would lose some fidelity (you can't can't reliably simulate all possible kinds of user interaction from within the browser) but for many tests the advantages of being able to directly introspect the state of the app from the tests would make writing such tests much easier and more pleasant.
• Unfortunately it's not possible to use use webdriverio in the browser to have a page to control itself—unfortunate because that would make white-box interaction testing way easier, with the tests and code being tested running in the same run time (as our mocha unit tests currently do) while still having all the flexibility to simulate user interaction that the webdriver protocol provides.
• On the other hand, there are test frameworks such as Cypress which, though they still have a node.js script controlling the browser, claim to "directly in the browser, you can debug failed tests using the in-browser developer tools".  I've not tried it but that sounds like it would make developing tests much easier.

I'd encourage you to try writing a few sample tests in each of a few different frameworks (selenium-webdriver, webdriverio, Playwright, Cypress…) and see which was most pleasant to use (as well as whether any were missing key features) before making a final choice.  After all, the easier and more pleasant it is to write tests the more likely one's app is to be well tested…

Best wishes,

Christopher

---------

* Notably including Playwright, which if nothing else certainly appears to have a very pretty, very GitHub-inspired UI.

** This reply is longer than necessary in part because the next thing on my to-do list is writing more such tests.

[Gabriel]

Hi Christopher,

Thank you for the detailed reply, and I apologize for the late response. I haven’t had a chance to set up the tests yet, but I plan to try out both Cypress and Playwright to see which one works better for my use case.

If you’ve already chosen a framework, let me know. It would probably be best to align with the Blockly team’s choice if possible, so we can reuse common assertions and selectors.

If it is helpful for you, I can also provide some demos with both testing frameworks once the setup is done.

Best regards,
Gabriel

[Christopher Allen]

Hi Gabriel,

 

If you’ve already chosen a framework, let me know. It would probably be best to align with the Blockly team’s choice if possible, so we can reuse common assertions and selectors.

As noted, we've been using Mocha + webdriverio and now have quite a large investment in this—but also as noted my own personal experience has been that writing tests this way has been a lot of toil, with debugging tests (i.e.: trying to get the test to work as intended in the first place) often quite frustrating.  Although I think it's unlikely we will change our approach in the foreseeable future I'm certainly interested in understanding more about the alternatives, particularly from blockly application developers who will be doing related kinds of testing.

 

If it is helpful for you, I can also provide some demos with both testing frameworks once the setup is done.

Yes, that would be most interesting to look at.

Christopher

 

[Ben Henning]

Adding another thought:

I think at this point core Blockly doesn't really provide a testing library for external integrations to leverage for their own tests (as emphasis has been placed on ensuring Blockly itself is well tested, though there are some options like the developer tools which might have incidental benefits for testing). This is definitely something that may have value to add in the future, but what would really help is understanding specific use cases that such work would address. In particular: understanding common setup or assertion flows in applications that integrate with Blockly, and why these would benefit as reusable components.

My hunch is that assertions is probably the area with most possible benefit, but care would need to be taken to ensure this is done in a framework-agnostic way since we don't really want to be prescribing any particular testing framework for applications that use Blockly.

There are no plans currently to commit to this type of work, but if you're able to collect any scenarios where having a solution like this would be beneficial, it would be great to know what those are and have them documented for a time when that work does make sense to do.

Ben

--
You received this message because you are subscribed to the Google Groups "Blockly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blockly+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/blockly/CAN0w5eYXGNXsOCOS_OLGTDgZ2zRNi7PVtHSpnpoWH75RCgfAYA%40mail.gmail.com.