Randomization for dynamic cardsort and matching

16 views
Skip to first unread message

Andrew Scholer

unread,
Jun 3, 2025, 11:53:23 AMJun 3
to prete...@googlegroups.com
I am interested in expanding the randomization options for the dynamic display of matching style problems. (Those authored in terms of a set of premises that need to be mapped to a set of responses - CardSort and Matching. See: https://pretextbook.org/examples/sample-book/noparts/html/matching-exercises.html)

I am looking for commentary on the markup to do that.

Background:
In CardSort, the mapping is * -> 0/1 (premise/response). Currently the premises are randomized and the responses are fixed in authored order.

In Matching, the mapping is * -> *. Currently both premises and responses are randomized.

For Matching problems, there are times I would like to lock in the presentation order of one or even both of the sides. For CardSort, it might make sense to leave the premises in some logical order while randomizing the responses.

These are different cases, but I think it is important to consider both in figuring out the syntax that might apply.

Question:
For many other interactives there is @randomize="yes"/"no".

Rob suggested:
@randomize="premise response" / "premise" / "response" / "none"

An alternative would be @randomize-premises and @randomize-respones with "yes"/"no".

The former is terser but feels like it has more potential for author surprise - the legal values of @randomize being different for different interactives.

Do we favor conciseness or consistency in cases like this?


Andrew Scholer (he/him/his)
Computer Science Instructor
Chemeketa Community College

Rob Beezer

unread,
Jun 3, 2025, 2:51:08 PMJun 3
to prete...@googlegroups.com
Thanks, Andrew. Last things first.

> Do we favor conciseness or consistency in cases like this?

I'm ambivalent on this one. A coin toss. ;-)

Note that #matching allows @order on both premises and responses, *to be
interpreted* when forming a static version, otherwise the authored order is
used. Not sure that much flexibility is needed, but it came at little cost.

For #cardsort, the markup is used to convey the solution. So too much use of
authored order for static can "give it away". So we suggest that @order on the
premises is pretty much necessary.

So @order is really meant to support an author prescribing a consistent look in
static formats. But it could be interpreted in dynamic contexts. Ideas for
dynamic contexts:

@premise-order="authored|random|prescribed" (similarly for @response-order)

OR

Randomization/ordering control could be reserved for #matching, and not managed
for #cardsort. Every #cardsort can be re-written as a #matching, though the
interface does not take advantage of the many-to-one structure.

Rob

On 6/3/25 08:52, Andrew Scholer wrote:
> I am interested in expanding the randomization options for the dynamic display
> of matching style problems. (Those authored in terms of a set of premises that
> need to be mapped to a set of responses - CardSort and Matching. See: https://
> pretextbook.org/examples/sample-book/noparts/html/matching-exercises.html
> <https://pretextbook.org/examples/sample-book/noparts/html/matching-
> exercises.html#matching-function-types>)
>
> I am looking for commentary on the markup to do that.
>
> *Background:*
> In CardSort, the mapping is * -> 0/1 (premise/response). Currently the premises
> are randomized and the responses are fixed in authored order.
>
> In Matching, the mapping is * -> *. Currently both premises and responses are
> randomized.
>
> For Matching problems, there are times I would like to lock in the presentation
> order of one or even both of the sides. For CardSort, it might make sense to
> leave the premises in some logical order while randomizing the responses.
>
> These are different cases, but I think it is important to consider both in
> figuring out the syntax that might apply.
>
> *Question:*
> For many other interactives there is @randomize="yes"/"no".
>
> Rob suggested:
> @randomize="premise response" / "premise" / "response" / "none"
>
> An alternative would be @randomize-premises and @randomize-respones with "yes"/"no".
>
> The former is terser but feels like it has more potential for author surprise -
> the legal values of @randomize being different for different interactives.
>
> Do we favor conciseness or consistency in cases like this?
>
>
> Andrew Scholer (he/him/his)
> Computer Science Instructor
> Chemeketa Community College
> 503.589.7649
> computerscience.chemeketa.edu/people/andrew-scholer/ <http://
> computerscience.chemeketa.edu/people/andrew-scholer/>
>
> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com <mailto:pretext-
> dev+uns...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/
> CACm44N-83iqj-m_qUNGM9bgoeSxKN3yx%2Bg%2BKNCqMtbDAwK3iew%40mail.gmail.com
> <https://groups.google.com/d/msgid/pretext-dev/CACm44N-83iqj-
> m_qUNGM9bgoeSxKN3yx%2Bg%2BKNCqMtbDAwK3iew%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

Bradley Miller

unread,
Jun 3, 2025, 6:24:42 PMJun 3
to PreTeXt development
Here is another twist to this conversation. Barb and I were talking about the new interface for creating parsons problems today. Our first iteration does not include a way to explicitly set the order of the blocks. When we discussed why that was important, the reason is that she wants all the groups or students to see the same initial ordering. The actual order is not as important as the fact that everyone sees the same thing.

I think WeBWorK people are nodding at this point. So, we talked about having an optional seed value or some kind of flag so that everyone got the same random ordering. I’m not sure if that is universal, but it might be another option to consider for many of the exercises. We would not even have to make the instructor generate a seed, we can always make it a hash value of the question name or something like that.

Brad
> To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/MTAwMDAwNS5iZWV6ZXI.1748976666%40pnsh.

Andrew Scholer

unread,
Jun 3, 2025, 7:12:24 PMJun 3
to prete...@googlegroups.com
As Rob notes, there is a way to set a fixed "random" order for blocks for Parsons problems: @order on <block> elements. It just doesn't get used in the dynamic presentation. An easy step would be to pass the @order from XML to the RS JS and add an @randomize to blocks. If  @randomize is "no" the order is used.

If (as it sounds) people want static random order without having to manually specify @order for each block, we would maybe need @block-order= "random" (default), "static-random", and "prescribed" (use @order)

Then matching could use a similar vocabulary for @premise-order and @response-order: "random" (default), "static-random", "prescribed", "authored".

Since Brad raised the ante... it feels odd that multiple choice problems require opt-in for randomized order (as opposed to all the other RS widgets). They are easier to edit so the answers are in a "static random" order, but it is still odd that they are an exception to the rule. Should they have @randomize "yes" as the default? 

Andrew Scholer (he/him/his)
Computer Science Instructor
Chemeketa Community College

To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/262C7793-4057-4991-8232-2843287209D4%40runestone.academy.

Bradley Miller

unread,
Jun 3, 2025, 7:40:12 PMJun 3
to PreTeXt development
I think that multiple choice was maybe the second exercise type I ever added after activecode!  They have not changed much since then.  From an implementation standpoint the important thing is that when the instructor goes to the interface to look at the distribution of how students responded to each question that it makes sense.  That would mean we need to identify each answer uniquely in a way that we can use after the fact.  I think this is a 12 year old bug, but if we are going to do it we need to do it right.  It looks to me like what instructors care about is whether the students got it right (which works!) not the after the fact analysis of how students answered the question.

Brad
 


Rob Beezer

unread,
Jun 3, 2025, 7:47:30 PMJun 3
to prete...@googlegroups.com
On 6/3/25 16:11, Andrew Scholer wrote:
> As Rob notes, there is a way to set a fixed "random" order for blocks for
> Parsons problems: @order on <block> elements. It just doesn't get used in the
> dynamic presentation. An easy step would be to pass the @order from XML to the
> RS JS and add an @randomize to blocks. If @randomize is "no" the order is used.

That would give every class, every time, for every student the same mixed-up
order (Let's not call it "random", or any such variant? Us mathematicians would
maybe say "permuted".) Names aside, maybe that is what you want. In any event,
now an author's markup is straying into an instructor's preferences. My class
may not want to do what Barb's class does. So an author can give a permutation,
and an instructor can choose to use it (or use a seed, or give every student a
different version. I'm happy to put/allow @order on various blocks, and report
it to Runestone Services.

> it feels odd that multiple choice problems require opt-in for randomized order

Not to me. Somebody should tell me *why* you want a random version, versus a
prescribed order crafted by the author. Do we not want students sharing
answers, or is there more to it than that?

Rob

Andrew Scholer

unread,
Jun 4, 2025, 10:44:37 AMJun 4
to prete...@googlegroups.com
Multiple choice

That would mean we need to identify each answer uniquely

That is already done. There already is randomization of MC exercises and responses are stored as the index of the selected answer as authored, not as displayed.

Do we not want students sharing answers, or is there more to it than that?

People are very bad at generating random distributions? More seriously, I find it much easier to author correct answers first and then add distractors. It is nice to have the presentation layer randomize that instead of having to manually shuffle the options. But willing to leave that on the cutting room floor.

Parsons
I'm happy to put/allow @order on various blocks...

I think we need a solution for "randomly selected permutation" that doesn't depend on an author-specified markup. As an author, I don't care one iota about what permutation is selected to display a Parsons I have written (as long as it is not always the correct order). Marking up a valid permutation is a lot of monkey work. If no one cares exactly which permutation is the end product, it seems silly to require that labor. @order should be a fallback for the rare case when an author thinks there is some ideal permutation to display.


Author/Instructor Control
Presumably an instructor who wanted everyone to see the same ordering for a Parsons would want everyone to see the same order for CardSort, Matching, MultipleChoice,... Perhaps event the same seed for FITB? That sounds like a course level setting. If it is even needed. Perhaps a single randomly generated permutation should be the default for those elements.

In any case, if it is a course level feature in RS, we don't need markup for it. If someone selects "Team teaching mode", or "Single ordering for questions" or whatever, we can check for @order and use it if is there. If not, do what you suggested and seed with a hash of the RS id for the widget. 



Andrew Scholer (he/him/his)
Computer Science Instructor
Chemeketa Community College

--
You received this message because you are subscribed to the Google Groups "PreTeXt development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/MTAwMDAxNS5iZWV6ZXI.1748994447%40pnsh.

Rob Beezer

unread,
Jun 4, 2025, 12:57:08 PMJun 4
to prete...@googlegroups.com
Exercises of these types have markup designed to convey the solution, whereever
possible. So, for example, Parsons have the non-distractor blocks in the
correct order. Cardsort has the cards belonging to a bucket authored in a
grouping with the bucket. Then we rely on software to randomize.

Specifying an @order is not "monkey work". We care very much about less-capable
output formats, such as PDF, EPUB, braille. It is not PreTeXt if all we seem to
care about is HTML and everything else is an after-thought. So we allow an
author to craft a specific order and it is repeatable.

XSL is not great at randomization, nor are many of our output formats. I don't
think we want to ship out exercises to the Python and back for randomization.
Seeds are notorious for producing different results over time. I don't want the
exercises in my PDF changing from edition to edition unless I have a reason and
elect that change.

We already have markup for @order *and* we have a need for it in many cases -
the schema will reflect that. Since an author *will* be doing this anyway, we
might as well leverage that. So I am in support of any randomization that
happens via Runestone Services, or any election to use an ordering prescribed by
an author - in HTML or anywhere else.

Rob

On 6/4/25 07:43, Andrew Scholer wrote:
> *Multiple choice*
> /That would mean we need to identify each answer uniquely///
>
> That is already done. There already is randomization of MC exercises and
> responses are stored as the index of the selected answer as authored, not as
> displayed.
>
> /Do we not want students sharing answers, or is there more to it than that?/
>
> People are very bad at generating random distributions? More seriously, I find
> it much easier to author correct answers first and then add distractors. It is
> nice to have the presentation layer randomize that instead of having to manually
> shuffle the options. But willing to leave that on the cutting room floor.
>
> *Parsons*
> /I'm happy to put/allow @order on various blocks.../
>
> I think we need a solution for "randomly selected permutation" that doesn't
> depend on an author-specified markup. As an author, I don't care one iota about
> what permutation is selected to display a Parsons I have written (as long as it
> is not always the correct order). Marking up a valid permutation is a lot of
> monkey work. If no one cares exactly which permutation is the end product, it
> seems silly to require that labor. @order should be a fallback for the rare case
> when an author thinks there is some ideal permutation to display.
>
>
> *Author/Instructor Control*
> Presumably an instructor who wanted everyone to see the same ordering for a
> Parsons would want everyone to see the same order for CardSort, Matching,
> MultipleChoice,... Perhaps event the same seed for FITB? That sounds like a
> course level setting. If it is even needed. Perhaps a single randomly generated
> permutation should be the default for those elements.
>
> In any case, if it is a course level feature in RS, we don't need markup for it.
> If someone selects "Team teaching mode", or "Single ordering for questions" or
> whatever, we can check for @order and use it if is there. If not, do what you
> suggested and seed with a hash of the RS id for the widget.
>
> *Matching*
> So I think we might be down to just @premise-randomize="yes"/"no" and @response-
> randomize="yes"/"no" with defaults to "yes". If there are @order on items, a
> static display should assume "prescribed". Otherwise it should assume
> "authored". Runestone can do the same for non-randomized. For randomized it can
> either do fully pseudorandom, seeded random, or allow for instructor choice.
>
>
> Andrew Scholer (he/him/his)
> Computer Science Instructor
> Chemeketa Community College
> 503.589.7649
> computerscience.chemeketa.edu/people/andrew-scholer/ <http://
> email to pretext-dev...@googlegroups.com <mailto:pretext-
> dev%2Bunsu...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/
> MTAwMDAxNS5iZWV6ZXI.1748994447%40pnsh <https://groups.google.com/d/msgid/
> pretext-dev/MTAwMDAxNS5iZWV6ZXI.1748994447%40pnsh>.
>
> --
> You received this message because you are subscribed to the Google Groups
> "PreTeXt development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to pretext-dev...@googlegroups.com <mailto:pretext-
> dev+uns...@googlegroups.com>.
> To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/
> CACm44N9MocVSEwc7hE0b%2Bf8%2BNm%3DyHvc9QuyfeZo8PYSbkNCVDQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/pretext-dev/
> CACm44N9MocVSEwc7hE0b%2Bf8%2BNm%3DyHvc9QuyfeZo8PYSbkNCVDQ%40mail.gmail.com?
> utm_medium=email&utm_source=footer>.

Andrew Scholer

unread,
Jun 4, 2025, 2:55:39 PMJun 4
to prete...@googlegroups.com
I think there is a use case for randomly determined permutation of blocks that may vary from one render to the next: If I was printing out a review worksheet or summative assessment that contained a problem seen before (in practice or formative assessment), or a test with multiple forms, having the block order change with each printing would be a valuable thing.

So I remain unconvinced that every author in every situation will view using @order to specify a canonical display order for the blocks as being a good thing.

But I'll leave Parsons alone for now. Anything short term vis-a-vis customizing randomization sounds like can be done strictly on the Runestone side.



Andrew Scholer (he/him/his)
Computer Science Instructor
Chemeketa Community College

To unsubscribe from this group and stop receiving emails from it, send an email to pretext-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/pretext-dev/MTAwMDAxNC5iZWV6ZXI.1749056225%40pnsh.
Reply all
Reply to author
Forward
0 new messages