out of order input and feedback

25 views
Skip to first unread message

kmul...@gmail.com

unread,
Jan 7, 2022, 10:19:14 AM1/7/22
to CTAT Users
Hello,
This question relates to feedback when a student's answer is out of order in CTAT. Specifically, when a student enters a correct response but one that is "out of order", they get the out of order message from CTAT and their entry is colored red.  This entry stays red until they press return (or tab) once it becomes the next expected entry. (At least that is my understanding of the process based on documentation and testing, but please correct me if I'm wrong). This seems to have the potential to confuse students who may forget to re-hit the enter button for correct but out of order entries colored red once they become "in order". Any suggestions on alternatives (perhaps there is a way to not accept entries that are out of order, i.e., remove them from the interface)? One we currently implemented it to lock entries until they are “in order” but this seems overkill. 

Many thanks in advance, 

Kasia Muldner
Carleton University

Jonathan Sewall

unread,
Jan 9, 2022, 1:48:01 AM1/9/22
to ctat-...@googlegroups.com, Vincent Aleven, Octav Popescu, Maxwell Benson
I don't know that our interaction for responses that are correct but out of order has been a big problem in the past, but I can see that it could be confusing, at least at first. If you (or others on the cc list) have experience or data showing trouble with this, we'd be interested in it. If we tried to improve it, we might want to come up with a solution that doesn't blank the student's input, since it will be correct at a later step, and it might require unwanted effort to reenter it (e.g., some formula or a sentence in a free text entry). Perhaps this should apply only to free text entry questions. One idea would be to change the font color back to black (or whatever the not-yet-evaluated color is). Have you thought of others, in addition to locking entries? Thanks,
               Jonathan

--
You received this message because you are subscribed to the Google Groups "CTAT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ctat-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ctat-users/39100014-62a6-4f37-961f-e081cd6e3fffn%40googlegroups.com.

Vincent Aleven

unread,
Jan 10, 2022, 11:58:15 AM1/10/22
to Jonathan Sewall, Kasia Muldner, ctat-...@googlegroups.com, Octav Popescu, Maxwell Benson
You may be aware (or you may not be) that an author has full control over the flexibility of the order of the steps (e.g., through ordered/unordered groups, and by setting to top level to ordered or unordered).  In general, we recommend that you make your tutors as flexible as possible with regard to step order, and only impose ordering of steps when there is a strong pedagogical reason to do so (e.g., when a mathematical procedure depends critically on order - e.g., going right to left in multicolumn addition).  Perhaps that might avoid the issue?

Another way to avoid (or at least reduce the likelihood of) out of order steps is by making the interface gradually reveal the steps, by means of tutor-performed actions.  Though usually you'd still want to reveal multiple new steps at a time (e.g., subgoal by subgoal).  This approach might also reduce cognitive load by not showing the complete interface all the time.  

Alternatively you could also gradually unlock input boxes that way, although I am not aware of any tutor that does that.  It would not have the advantage of possibly reducing cognitive load. 

A downside of a flexible step order is that hints (at least when used with certain hint policies, such as "follow errors") cannot depend on prior steps having been completed, which means they could be hard to understand (they may appear a bit "out of context") or need to be written in a way that does not assume that these prior steps have been done.  The latter can also be cumbersome.

Selecting the hint policy that follows the preferred path would reduce that problem but then hints don't always follow errors.  Perhaps that is OK.

In CTAT, we have an unwritten policy against having "unevaluated input" sitting in the interface elements, so that's why we do not color out-of-order steps in black, though we could revisit that idea.

Sorry, did not mean to go on for so long.

Please keep as posted as to which way you decide to go.

Best,

Vincent

Vincent Aleven
Professor of Human-Computer Interaction
Director of Undergraduate Programs
Human-Computer Interaction Institute
Carnegie Mellon University
Co-Editor-in-Chief, International Journal of Artificial Intelligence in Education




Reply all
Reply to author
Forward
0 new messages