Concerns on working memory task accuracy terms in HCP-YA behavioral CSV file

[Ruiqi Chen]

Dear HCP Professionals,

I am wondering how the terms `WM_Task_Acc`, `WM_Task_0bk_Acc` and `WM_Task_2bk_Acc` were calculated in the behavioral CSV file of the HCP Young Adult project (obtained by the "Export CSV" button on balsa, but an older version downloaded from old ConnectomDB website has the same issue too).

From the latest data dictionary (`HCP_S1200_DataDictionary_Oct_30_2023.csv`), these three terms were explained as:

`WM_Task_Acc`: Accuracy across all conditions in WM task

`WM_Task_0bk_Acc`: Accuracy across all conditions in 0-back

`WM_Task_2bk_Acc`: Accuracy across all conditions in 2-back

One (at least me) would assume that "accuracy" means the proportion of correct trials out of all trials. However, this is clearly not what these columns are encoding, because there were 80 trials in each condition (160 in total), but the values were not dividable by 1/160 (0.625%) or 1/80 (1.25%).

A quick try in Excel revealed that the term `WM_Task_0bk_Acc` is actually the mean over the following 12 terms:

WM_Task_0bk_Body_Acc: Accuracy across all trials in 0-back body condition
WM_Task_0bk_Body_Acc_Target: Accuracy across target trials in 0-back body condition
WM_Task_0bk_Body_Acc_Nontarget: Accuracy across nontarget trials in 0-back body condition
WM_Task_0bk_Face_Acc
WM_Task_0bk_Face_Acc_Target
WM_Task_0bk_Face_ACC_Nontarget
WM_Task_0bk_Place_Acc
WM_Task_0bk_Place_Acc_Target
WM_Task_0bk_Place_Acc_Nontarget
WM_Task_0bk_Tool_Acc
WM_Task_0bk_Tool_Acc_Target
WM_Task_0bk_Tool_Acc_Nontarget

(and similarly for the 2-back.)

These 12 terms seemed to have reasonable value that correspond to the proportion of correct trials within the specified conditions. However, a mean over these 12 variables makes no sense. The first term `WM_Task_0bk_Body_Acc` already summarizes over all trials in the body condition, which includes all target trials (that were double-counted in `WM_Task_0bk_Body_Acc_Target`) and all nontarget trials (double-counted in `WM_Task_0bk_Body_Acc_Nontarget`). Taking an average over "the mean over full set", "the mean over subset 1" and "the mean over subset 2" simply makes no sense.

I believe what most people look for in `WM_Task_0bk_Acc` is just the proportion of correct trials in 0-back condition, which is not what it is currently encoding. And even if `WM_Task_0bk` was designed as a mean over conditional means, it should still be either the average over the four `WM_Task_0bk_[STIMULI]_Acc` variables, or over the eight `WM_Task_0bk_[STIMULI]_Acc_Target` and `WM_Task_0bk_[STIMULI]_Acc_Nontarget` variables, but never as the average over the 12 variables.

Besides, I still haven't figured out how `WM_Task_Acc` is calculated. It is neither the proportion of correct trials (since its not dividable by 0.625%), nor a mean over any combinations of other variables I have tried.

Given the popularity of the HCP WM dataset, I am deeply concerned that the terms `WM_Task_Acc`, `WM_Task_0bk_Acc` and `WM_Task_2bk_Acc` in the behavioral CSV file have been used in many studies without sanity check, which could lead to totally wrong conclusions. If possible, could you please check how these three terms were calculated and correct any errors if they exist? It would also be very helpful to explain these three terms in more details in the data dictionary file.

Looking forward to your reply.

Best,

Ruiqi

[Tim Coalson]

Thanks for bringing this to our attention.  We are currently looking into it.

Tim

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/a1beaa69-a7f3-4d5f-8efd-53c2bf18145cn%40humanconnectome.org.