What's wrong with my counterbalanced list?

72 views
Skip to first unread message

Jeremy

unread,
Apr 3, 2021, 6:35:03 AMApr 3
to ibexexperiments
Dear all,

I conducted a pilot study with 15 subjects. In my AJT, each item should receive data from 7 or 8 participants. But some of the items only received data from 2 subjects. Is there anything wrong with my counterbalanced list?

Best wishes,
Jeremy


I marked the items that only received data from 2 subjects with ?. The other items are normal (collecting data from 7 or 8 subjects). Sentences were deleted for better reading experience.



var shuffleSequence = seq("setcounter", "intro", sepWith("sep", rshuffle(startsWith("AJT"))), sepWith("sep", startsWith("q")) );

[["AJT-CLNM.A.1", 54 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.B.1",  54], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.A.2",  55], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.B.2", 55 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.A.3", 56 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.B.3", 56 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.A.4",  57], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.B.4", 57 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.A.5",  58], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.B.5", 58 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.A.6", 59 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNM.B.6", 59 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-CLNN.A.1", 60  ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.B.1",  60 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.C.1", 60  ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.A.2",  61], "AcceptabilityJudgment", {s: "  }],
 [["AJT-CLNN.B.2",  61 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLNN.C.2", 61  ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLNN.A.3", 62 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.B.3", 62  ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.C.3", 62 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.A.4",  63 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLNN.B.4",  63 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.C.4", 63  ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.A.5", 64  ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.B.5", 64 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLNN.C.5",  64 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLNN.A.6",  65 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.B.6",  65 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLNN.C.6", 65 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-TOPI.1", 66 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-TOPI.2", 67 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-TOPI.3", 68 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-TOPI.4", 69 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-TOPI.5", 70 ], "AcceptabilityJudgment", {s: "  "}],  
[["AJT-TOPI.6", 71 ], "AcceptabilityJudgment", {s: "  "}],  
? [["AJT-CLBF.A.1",  72 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLBF.B.1", 72 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.C.1",  72 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.D.1",  72 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.A.2",  73 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLBF.B.2",  73 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLBF.C.2",  73 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.D.2",  73 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.A.3",  74 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.B.3",  74 ], "AcceptabilityJudgment", {s: "  "}],
? [["AJT-CLBF.C.3", 74  ], "AcceptabilityJudgment", {s: "  "}],
?[["AJT-CLBF.D.3",  74 ], "AcceptabilityJudgment", {s: "  "}],
?[["AJT-CLBF.A.4", 75  ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.B.4",  75 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.C.4", 75  ], "AcceptabilityJudgment", {s: "  "}],
?[["AJT-CLBF.D.4", 75  ], "AcceptabilityJudgment", {s: "  。"}],
?[["AJT-CLBF.A.5", 76  ], "AcceptabilityJudgment", {s: "  "}],
?[["AJT-CLBF.B.5",  76 ], "AcceptabilityJudgment", {s: "  。"}],
[["AJT-CLBF.C.5",  76 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.D.5",  76 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.A.6",  77 ], "AcceptabilityJudgment", {s: "  "}],
?[["AJT-CLBF.B.6",  77 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.C.6",  77 ], "AcceptabilityJudgment", {s: "  "}],
[["AJT-CLBF.D.6",  77 ], "AcceptabilityJudgment", {s: "  "}],




Alexandre Cremers

unread,
Apr 3, 2021, 8:55:28 AMApr 3
to ibexexp...@googlegroups.com
Your groups are not equal, and all the problematic items are in groups with more than 2 items. This is technically allowed, but it will most likely create problems like what you saw.
A

--
You received this message because you are subscribed to the Google Groups "ibexexperiments" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ibexexperimen...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ibexexperiments/70b8dba6-5c7c-47ed-9b5f-080c586c44e7n%40googlegroups.com.

Jeremy

unread,
Apr 3, 2021, 10:28:06 AMApr 3
to ibexexperiments
Thank you Alex! Then what can I do with the group? My experiment includes groups of different number of conditions. Some of them are 2, whereas others are 3 or 4. Do I need to create separete experiments?

Best wishes,
Jeremy

Alexandre Cremers

unread,
Apr 3, 2021, 10:34:06 AMApr 3
to ibexexp...@googlegroups.com
Surely you can't use a latin square then. You need to think more carefully about your experimental design, and we can't really help you with that.
When you know what you want to show to your participants, then we can help you program it...

Jeremy

unread,
Apr 3, 2021, 5:17:02 PMApr 3
to ibexexperiments
Hi Alex. I find that 'equalGroupSizes' has to be set false. I will try it.

Best wishes,
Fei

Alexandre Cremers

unread,
Apr 3, 2021, 5:25:47 PMApr 3
to ibexexp...@googlegroups.com
It is set to false by default. It won't solve your problem: if your groups have different sizes, you can't possibly get the same number of participants for each item.

You could ensure that each item within a group has an equal number of participants (in which case you would need a multiple of 12). In this case you need to ensure that your participants are properly distributed among your latin square cells (check past discussions on this topic).

A

Jeremy

unread,
Apr 4, 2021, 5:17:51 AMApr 4
to ibexexperiments
Hi Alex,

I was wondering it really has something to do with the latein square design. I have another experiment with 6 lists and 8 lists. The experiment works fine. The data in my experiment is too extreme. Some of the items only get assigned to one participant (out of 15).

Best wishes,
Jeremy

Alexandre Cremers

unread,
Apr 4, 2021, 5:44:42 AMApr 4
to ibexexp...@googlegroups.com
On top of the unbalanced list, there's always a potential issue with too many participants getting assigned to the same list. This happens when all participants join the experiment at the same time because they're only counted when they send their results. There have been a few discussions about this in the past, but in short the easiest way to balance participants among your lists is to use URLs with the "withsquare" option.
Best,
Alexandre

Message has been deleted

Jeremy

unread,
Apr 5, 2021, 1:23:03 PMApr 5
to ibexexperiments
Thanks Alex. The 'server.py?withsquare=n' works the best for me.

Best wishes,
Jeremy

Özge BAKAY

unread,
Jun 2, 2021, 5:42:43 AMJun 2
to ibexexperiments
Hi,

I had a similar problem: I have four lists, but one of them is completed by 17 participants whereas the others are by 11 each. I understand the cause of this problem. Above you said that sending the link to the participants with "withsquare" solves the problem. The link to my experiment is this: https://spellout.net/ibexfarm/manage/Offline2 Can you tell where/how to add this to an experiment link?

Best,
Özge

Alexandre Cremers

unread,
Jun 2, 2021, 5:57:55 AMJun 2
to ibexexp...@googlegroups.com
Hi,

This is the link to the managing page for your experiment. The link to your experiment would be something like "https://ibex.spellout.net/experiments/USERNAME/EXPERIMENTNAME/view". Just add "?withsquare=XXX" at the end of that link.

Best,
Alexandre

Özge BAKAY

unread,
Jun 3, 2021, 7:08:40 AMJun 3
to ibexexp...@googlegroups.com
Hi,

I tried this but it did not work: https://spellout.net/ibexexps/ozgebakay/Offline2/experiment.html/view?withsquare=2 I am using IbexFarm, not PC Ibex. Could this be why?

Özge

Alexandre Cremers

unread,
Jun 3, 2021, 7:29:02 AMJun 3
to ibexexp...@googlegroups.com
That one seems to work, although I let you check if it shows the correct items.
A

Alex Drummond

unread,
Jun 3, 2021, 7:55:35 AMJun 3
to ibexexperiments
With the old Ibex Farm you may need to replace 'experiment.html' with
'server.py' in Alexandre's link for the withsquare parameter to take
effect.

Moving over to the new Ibex Farm is strongly encouraged!

Alex
> To view this discussion on the web visit https://groups.google.com/d/msgid/ibexexperiments/CABk4Z5dpx%3D7s3yJkXAHVN8mC3DLBeSm0M9vs-1kCMYuJXrP0KQ%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages