MST Scoring & Data Quality check

[Tanya Datta]

Dear Dr. Stark,

We are looking to conduct a data quality check of the MST task we use. We are trying to determine if we are obtaining enough quality data to conduct a proper analysis at the end of the study, since we have been getting mixed feedback from participants about their understanding of the task.

We use the continuous version in an MRI scanner with participants aged 65-80 years old. We do 5 rounds of 146 trials each = 730 trials total. Within each round there are 17-23 trials of “old” stimuli, 14-27 trials of “similar” stimuli, and 106-114 trials of “new” stimuli for a total of 96 “old” trials, 96 “similar” trials, and 538 “new” trials for all 5 rounds. For pre, post, and 1 year followup, we use sets C, D, and G for older participants and we have switched to using the updated Sets 1, 2 and 3 for newer participants.

Currently we are checking through all of our participants’ MST data for completion before moving on to analysis of their scores, since occasionally we experience technical difficulties in the scanner that force us to cut the task short. We are looking at the minimum number of completed rounds necessary to accurately compare participant scores to each other, as well as the maximum number of trials a participant can miss per round. Based on your knowledge, is there a threshold number of trials necessary for successful analysis of behavioral data? Additionally, do you know the minimum number of trials required for successful analysis of fMRI data?

We have been told that in general, we need 50 events minimum per condition (i.e. in this case old, similar, new) to properly analyze fMRI data. Do you think this would hold true for the MST task in the scanner? (We know you've been asked this question before, but just in case there is new information!).

Many thanks in advance!

Best,

Tanya Datta & Caroline Bandurska

tda...@mgh.harvard.edu

caroline....@mgh.harvard.edu

[Craig E.L. Stark]

A few things to keep in mind.  First, we've found that the video instructions are very helpful in getting people to know what they're supposed to do, what "similar" means, etc.  We use them whenever possible and have several flavors of them (they're just a powerpoint / keynote that I narrate) for the different variants (continuous, old/similar/new, old/new, etc.).

Second, on the behavioral end, keep in mind that you're trying to estimate the probability of two events -- saying "Similar" to a lure and saying "Similar" to a new item (in the standard LDI metric).  Given that we have a fixed number of trials, we're open to quantization error.  If you have only 10 possible lure trials, for example, your scores will always be in steps of 0.1 (1/10, 2/20, ...).  If the true probability was 0.25 and you ran this with 10 trials a bunch of times, you'd get your estimates to be 0.2 a bunch and 0.3 a bunch (and a few 0.1s and 0.4s).  You don't have the number of trials (your denominator here) to get any better resolution.  But, across subjects, you'd buy this back given the fact that you'd get 0.2 a bunch and 0.3 a bunch (which average out to 0.25).

So, if you want a good metric within a subject, we're going to need more than 10 trials.  If you have 20, you have steps of 0.05.  If you have 50, you have steps of 0.02, etc.  In the classic 64-items-per-list variant your steps are 0.0156 and in the 32-items-per-list your steps are 0.031.  In yours - in which you have 96 per, you're at small steps of 0.01 and since I consider the 32-item-per variant suitable for most all behavioral work, you're in great shape there -- assuming you don't have a lot of "no response" trials.  So, take a look at that to make sure you're not missing a ton of trials this way.

One thing we do use, threshold-wise, though is a "number of times they respond 'Similar'" cut.  You can't break this down by trial type (or you'd be massively biasing your results), but given that the standard metric is based on participants using this key, if they don't ever press the key, you don't have a valid estimate of their memory.  (Here, FWIW, is where the video instructions have come in very handy.)  So, if we don't even have 10 "Similar" responses writ large, we know there's no prayer of having a reasonable estimate of the true probability and that they've probably misunderstood the instructions.

Turning to the fMRI -- I think 50 is a high number, personally.  We can't give a hard and fast number here as it comes down to your actual effect size.  When designing things, I look for at least 20-25 per condition and when analyzing things, I'll not even try if I have <10.  What you really want is to pass the variance in your single-subject estimate up to the group level analyses so that this can be hammered out.  But, overall in the task, I routinely stress to people that a) high-res is noisy, and b) these "pattern separation" contrasts were designed to try to index something linked to PS, not to drive the biggest BOLD effect in the hippocampus.  I've seen far bigger BOLD effects in the hippocampus than we get with these.  That doesn't mean this isn't what the hippocampus is doing heavily - it just means our way of getting at things and linking activity to function is far from perfect.  Show low-contrast flickering checkerboards to v1 and you don't get a huge response.  Remove V1 though and you have a devastating effect on the ability to perceive those low-contrast checkerboards.

Craig

[cban...@bu.edu]

Hi Dr. Stark, 

Thank you so much for your response! This is very helpful.

- Caroline 

[Gunes Sevinc]

Hi Dr. Stark, 

I'm currently creating first level  fMRI contrasts for the same data set and wanted to check with you first.

To summarize, we use the continuous version in an MRI scanner with participants aged 65-80 years old. 

We do 5 rounds of 146 trials each, before and after the intervention also at two follow-up points. 

The goal to assess the impact of our intervention on pattern separation and associated BOLD response as opposed to a control intervention.

I was wondering if you had any recommendations about potential contrast(s) of interest such as [p('similar'|lure) - p('old'|lure)]?

Thank you very much in advance, 

Gunes

[Tanya Datta]

Dear Dr. Stark,

 

We would like your input as we begin analysis.

 

As mentioned earlier in the thread, we started off using Sets C/D (counterbalanced for pre-post intervention) and switched to using Sets 1/2 (also counterbalanced for pre-post intervention). We are trying to determine the best way to compare Sets C/D and Sets 1/2 to each other. I read your Postable Converter for converting Sets C/D to EFGH using Zscores. We are thinking of doing something similar for converting Sets C/D to Sets 1/2.

 

It looks like we would just need the Mean and SDs for the LDI and RI with Sets 1/2 for this to work. Since Sets 1-6 is an average of Sets CDEFGH, we were thinking of averaging the means and SDs of Sets CDEFGH to come up with the mean and SD of Set 1/2. Would this be ok? Would you have any alternate recommendations?  

Many thanks in advance.  

Sincerely,
Tanya
(tda...@mgh.harvard.edu) 

[Shauna Stark]

I've calculated the mean and stdev for percent old responses to the lures (for each lure bin) for Sets 1-6 and C-H, so you can come up with a conversion. You'll note how nicely balanced Sets 1-6 are now! :)

Hope this helps!

Shauna

Sets 1-6		%Old
		1	2	3	4	5
Set 1	Average	0.6473	0.4433	0.3108	0.2106	0.0961
	St Dev	0.1057	0.0429	0.0308	0.0287	0.0446
Set 2	Average	0.6450	0.4425	0.3104	0.2100	0.0951
	St Dev	0.1045	0.0431	0.0308	0.0285	0.0448
Set 3	Average	0.6437	0.4415	0.3096	0.2100	0.0945
	St Dev	0.1009	0.0439	0.0312	0.0290	0.0448
Set 4	Average	0.6410	0.4411	0.3094	0.2093	0.0941
	St Dev	0.0995	0.0441	0.0310	0.0288	0.0449
Set 5	Average	0.6391	0.4401	0.3091	0.2090	0.0936
	St Dev	0.0968	0.0429	0.0311	0.0287	0.0448
Set 6	Average	0.6366	0.4405	0.3083	0.2083	0.0931
	St Dev	0.0960	0.0444	0.0306	0.0285	0.0451
Sets C-H		%Old
		1	2	3	4	5
Set C	Average	0.7401	0.5110	0.3435	0.2296	0.0809
	St Dev	0.0935	0.0577	0.0502	0.0357	0.0545
Set D	Average	0.7227	0.5207	0.3672	0.2450	0.1127
	St Dev	0.1136	0.0447	0.0535	0.0357	0.0502
Set E	Average	0.6025	0.3540	0.2079	0.1195	0.0497
	St Dev	0.1010	0.0540	0.0419	0.0199	0.0268
Set F	Average	0.6134	0.4188	0.3017	0.1687	0.0697
	St Dev	0.0984	0.0458	0.0470	0.0317	0.0307
Set G	Average	0.6167	0.4229	0.2862	0.1792	0.0719
	St Dev	0.0863	0.0396	0.0477	0.0328	0.0363
Set H	Average	0.5944	0.3400	0.2167	0.0908	0.0257
	St Dev	0.1220	0.0529	0.0510	0.0252	0.0288