I didn't cite any references in that episode, but the pre-test effect is a well-studied phenomenon with research going back to the late 1960's (e.g. see here
). So I don't think you should view this as an empirical test of the idea in the sense that this would be the first time it would be tested. You're probably best off diving into the literature to get more details about how to do this well, and then seeing how your intervention goes (with proper AB testing). I think we all know how difficult it can be to bring research to practice in edtech applications! In particular, there are several reasons this could fail / have bad side effects in edtech contexts:
- Users could churn because they are frustrated with being asked a question they don't know the answer to. (This is less likely if usage is coerced / required by teachers.)
- The effect could be small because users find the fastest way to click "Next" rather than really trying to think about and answer the question. This could end up feeling like "just another screen to click through".
- If it's put as an MCQ and they happen to guess right, perhaps it would backfire and make them overconfident. This is particularly pernicious if they got something right for the wrong reason.
Looking into the literature a little bit more I can see that a lot has been done, but all I brought to the episode is the headline that "pre-tests increase learning", which is the tip of the iceberg, plus my own musings on the issue. The intention of episode 119 was to suggest a mechanism for the pre-testing effect (direction of focus and the overcoming of some cognitive biases), rather than to suggest that the effect exists (since we know that it does).
I'd be very happy to discuss this further if you are interested. Reach out by email if you want to discuss privately. Good luck in trying this out!
p.s. if you haven't already, I would also recommend looking into the literature on word problems, e.g. episode 28 of the podcast (long ago so I knew a lot less then, but the central idea is still important).