Problem with per-trial weight adjustment using setweight

[LaurensK90]

Hello,

 

I'm trying to create a task-switching experiment that requires participants to categorize a face by either emotion or gender, as indicated by a pre-stimulus cue. Repeat trials (emotion trial following an emotion trial, or gender trial following a gender trial) need to be more common than switch trials (emotion trial following a gender trial, etc.), with a 1:3 switch:repeat ratio. To achieve this, I wrote this script:

 

Dim e as integer
Dim g as integer

'Weighting trials to make repeats more likely than switches

If c.GetAttrib("TaskNr") = "0" Then

 FaceList.SetWeight 1, "3"

 FaceList.SetWeight 2, "1"

 FaceList.SetWeight 3, "3"

 FaceList.SetWeight 4, "1"

 FaceList.SetWeight 5, "3"

 FaceList.SetWeight 6, "1"

 FaceList.SetWeight 7, "3"

 FaceList.SetWeight 8, "1"

 'Counter to keep track of how many consecutive emotion trials have occured

 e = e+1

 g = 0

ElseIf c.GetAttrib("TaskNr") = "1" Then

 FaceList.SetWeight 1, "1"

 FaceList.SetWeight 2, "3"

 FaceList.SetWeight 3, "1"

 FaceList.SetWeight 4, "3"

 FaceList.SetWeight 5, "1"

 FaceList.SetWeight 6, "3"

 FaceList.SetWeight 7, "1"

 FaceList.SetWeight 8, "3"

 'Counter to keep track of how many consecutive gender trials have occured

 g = g+1

 e = 0

End If

'Set weights for emotion or gender trials to zero to prevent too many repeats

'If e >= 5 Then

' FaceList.SetWeight 1, "0"

' FaceList.SetWeight 2, "1"

' FaceList.SetWeight 3, "0"

' FaceList.SetWeight 4, "1"

' FaceList.SetWeight 5, "0"

' FaceList.SetWeight 6, "1"

' FaceList.SetWeight 7, "0"

' FaceList.SetWeight 8, "1"

' e = 0

'ElseIf g >= 5 Then

' FaceList.SetWeight 1, "1"

' FaceList.SetWeight 2, "0"

' FaceList.SetWeight 3, "1"

' FaceList.SetWeight 4, "0"

' FaceList.SetWeight 5, "1"

' FaceList.SetWeight 6, "0"

' FaceList.SetWeight 7, "1"

' FaceList.SetWeight 8, "0"

' g = 0

'End If

 

The first If statement looks at the current trial type and increases the weights of every same trial type to 3, and every other trial type to 1. The trial types are in alternating rows in FaceList. The script is at the end of the core experimental procedure, meaning that the weights are adjusted after each trial is complete. If I've done this correctly, there should be three times as many repeat trials as switch trials, but this doesn't seem to happen. I ran the experiment with 768 trials, then copied the TaskNr column (filled with 0s and 1s) out of the data file and pasted it in Excel. Then I added every cell to the cell below it to determine whether the trial was a repeat (0 or 2) or a switch (1), and counted those. This always results in roughly equal numbers of 0s, 1s and 2s (a 1:2 switch ratio) while running the task without the script gives you as many 1s as 0s and 2s combined (a 1:1 switch ratio). So the switch ratio works as expected without the script but not with it. I'm wondering if there's something wrong with the way I'm counting these because changing the weights from 3 to 4 or 5 makes the maximum number of consecutive repeats increase. With weights set to 3, 10 gender tasks in a row occur 20 times, and when set to 5, 10 in a row occur 46 times. So it's strange the actual switch ratio isn't affected.

 

The second if statement tries to ensure that there are no more than 5 consecutive trials of the same type, by looking at a counter that increments every time a trial passes of one type, and resets every time a trial passes of the other type. If the counter reaches 5, it sets all the trials of the same type to zero so it can't select them again. I haven't tested this extensively but the first time I ran the experiment with it, the number of consecutive repeats was even higher than normal! Have I made a mistake in this script, or might this be the same problem as the previous one?

 

I've asked official E-Prime support for help as well, a week ago, but haven't heard back from them yet, so any additional assistance would be appreciated.

[David McFarlane]

Just a few comments offhand...

First, the text for your code excerpt appears in little tiny type in
my e-mail reader, I had to copy & paste it into a text editor just to
read it. Please spare me that trouble in the future.

Second, whenever I see someone mucking about that much with List
weights in code, I wonder whether the same end might be better
accomplish by some restructuring with multiple Lists or nested
Lists. We all find it difficult to rethink and restructure a program
once we feel it is close to working, but many times I have found that
that sort of rethinking and restructuring has not only fixed the
immediate problem, but also solved other imminent problems. You may
indeed have a good reason for managing List weights in this case, but
think about it.

Third, I confess I found too much detail in the description for me to
work through it all. So I will just toss out the most common mistake
that people make when modifying Lists in code -- did you remember to
use List.Reset (in your case, FaceList.Reset) after making all these
changes? Your code excerpt does not include that line.

Regards,
-----
David McFarlane
E-Prime training
online: http://psychology.msu.edu/Workshops_Courses/eprime.aspx
Twitter: @EPrimeMaster (https://twitter.com/EPrimeMaster )

/----
Stock reminder: 1) I do not work for PST. 2) PST's trained staff
take any and all questions at https://support.pstnet.com , and they
strive to respond to all requests in 24-48 hours, so make full use of
it. 3) In addition, PST offers several instructional videos on their
YouTube channel (http://www.youtube.com/user/PSTNET ). 4) If you do
get an answer from PST staff, please extend the courtesy of posting
their reply back here for the sake of others.
\----

[LaurensK90]

Thank you for the suggestions. Sorry for the small font, I thought that a long list of code would be neater if it was slightly smaller, but apparently Google's preset font sizes do not show up well in e-mails. 

The short version is: I need a trial of task 1 to be followed by another trial of task 1 three out of four times, and by a trial of task 2 one out of four times, and vice-versa (first If statement), but the same trial can't come up more than five times in a row (second If statement). The current version of the script should do that, but doesn't, and I don't know why.

Creating nested lists for both tasks is definitely a good idea, that would reduce the number of SetWeight commands needed by 75%, but as for restructuring the lists: I have thought about using multiple lists but I don't see a way of dynamically varying trial probabilities while preserving randomness and ensuring that each stimulus is used exactly twice. I will take another look at it, though.

I did not know about List.Reset but if it does what I assume it does, I cannot use it because the weight adjustments are needed for the next trial. My trial procedure looks roughly like:

Facelist

StimulusSlide

ResponseSlide

SetWeightScript

Which task to select in the next trial depends on what task the current trial is. Unless I misunderstand, adding List.Reset would undo all the changes made by the script and choose the next trial with equal probability for both tasks.

Does this procedure design cause problems? Since I do not know how to refer to an attribute of the previous trial, this seemed like the only possible way of doing what I'm trying to do.

[LaurensK90]

Now that I have access to E-Basic help again I see that I was mistaken about the function of List.Reset, and appending this to the end of the script caused the experiment to create the appropriate switch ratio. So thank you very much for that suggestion!

 

However, my attempt at preventing the experiment from running more than five consecutive trials of the same type is still not effective. I suspect there's something wrong with the way I'm trying to increment the counter variable, which leads to the second If statement never getting triggered. Could that be the problem?

[David McFarlane]

Well... Implementing specific constraints on randomization at
runtime gets very tricky, try searching for discussions using terms
such as "random", "pseudorandom", "pseudo-random", "constrain", and
"constraint". Offhand, I feel very leery of any approach using
List.SetWeight, and I especially hope that you do not try that while
the List itself is running!

PST shows one time-honored (though crude) method for implementing
on-the-fly randomization with contstraints in their "No Repeats on
Consecutive Trials" examples on their website, and you might adapt
that approach for your program. That method, however, is a sort of
nondeterministic "bogosort" (look that up on Wikipedia) which suffers
several problems.

The cheap answer, which you will find in other discussions, is just
to randomize everything before runtime outside of E-Prime. Construct
a "random" sequence that has the properties you seek, then implement
that as a Sequential List in E-Prime. If you want different random
orders for different subjects, then construct a few more sequences
outside of E-Prime, implement each of your sequences as a nested List
(running in Sequential order), and then have E-Prime pick one of
those nested Lists on each run (perhaps using a main List set to
Counterbalance order). Or, generate your sequence outside of E-Prime
as a properly formatted .txt or .xml file, and then use List
LoadMethod "File" to read in the sequence at runtime.

-- David McFarlane

[David McFarlane]

Oh, and if you do come up with a good way of doing on-the-fly
randomization with constraints in E-Prime, then please, please write
back and let us know! This problem has bedeviled many of us for a
long, long, time. (Then again, some literature argues that
randomization with constraints is a Bad Thing in psychology
experiments, but that is another topic.)

Regards,
-- David McFarlane

[LaurensK90]

You mentioned not to use SetWeight while the list is running... but that's actually exactly what I've been doing. I received a reply from PST support saying that this was impossible, but after following your suggestion of adding List.Reset, the switch ratio works exactly as intended. They also sent me an example design (attached) using multiple lists with different weights, like you suggested, but since this is not able to make a trial selection based on the previous trial, it results in a switch ratio of 1:3.5 instead of 1:3. I know how to solve this, but again, it involves using SetWeight to set the weight of the inappropriate list to zero while it's running.

 

Does List.Reset have especially disruptive effects on timing, or does running it on every trial have other negative effects I'm not aware of? I can see why this method is a bit unorthodox, but since it works, I feel like I need a very good reason not to use it.

 

The cheap solution is probably a good option, but I'd still like to understand why this is so difficult.

[David McFarlane]

Well, you do well to not take my word for any of this (nor the word
of PST staff or documentation) and to test everything for
yourself. So I am glad that I brought this up, and that you tested
my statement and reported back. Looks like my understanding of
E-Prime is a little off again.

I had pretty much assumed that when you run a List (List.Run), it
looks at all the settings activated with the last List.Reset, and
then runs using those just settings, goodness know what doing a
List.Reset during the run would do (terminate the run? be
ignored?). But I confess I never tested that. Until I do that, your
empirical result trumps my assumption. Perhaps a List.Rest can
construct an Order object for running the List on the fly, in which
case doing a List.Reset during the run can affect the rest of the run
in a nicely controlled way such as you found. Might be worth
prodding the PST folks and see if they could reveal more about how
List.Reset and List.Run really work.

So if your tests show that everything works to your satisfaction,
then you may as well proceed. And this opens up more possibilities
(what happens if you change List.ResetCondition or
TerminateCondition, etc., during the run?)...

Thanks,
-- David McFarlane

[LaurensK90]

In their latest e-mail on this subject, PST support told me that "Generally, a List cannot be reset while it is running, though it may be that your experiment falls into a small set of cases when it will work."

I think I will have to check whether the other properties of the experiment are also working as intended, but as far as I could tell, nothing out of the ordinary was happening. I did also manage to solve the problem of avoiding more than five consecutive repeats: as I suspected, the problem was that the script declares the variables anew at each trial, which resets them to zero and prevents the counter variables from remembering their values. When I declare them as global variables in the userscript, it works as intended. I did have to increase the weights from 3 to 6 to compensate for the decrease in repeats, but now it does approximate a 1:3 switch ratio.

I will upload the task here soon if you or anyone else want to have a look at it. For now I am just glad that it works the way I expected it to work!

[LaurensK90]

Ahhh, now I see what's happening. It turns out List.Reset does do something like what I expected of it initially: when the script is run, instead of selecting the next item in the list, it simply starts from the beginning again. In practice, this is equivalent to setting the selection to "random with replacement", as it does not remember which of the items it has already selected. In a sample of three blocks where I expected all of my 96 stimuli to be used exactly three times, there was one omitted and one repeated stimulus. That wouldn't happen if the list kept running like it was supposed to.

 

Now, I could fix this by mucking up the item weights even more, with a giant If statement that sets the weight of each individual stimulus to zero once it is used so that it cannot be selected on the next trial, but I think I'm pushing my luck as it is, and the random selection is doing a pretty good job of it on its own.

 

I attached the final version of my experiment, but I didn't include the stimulus files because I don't know if I'm permitted to distribute those. If anyone wants to test it I suggest you modify the experiment to work with placeholder images.

 

(Reposted because I forgot to modify something in the experiment)