Problem with Fix processing suddenly not working

Chris Racey

unread,

Nov 7, 2021, 6:29:45 PM11/7/21

to HCP-Users

Dear HCP experts,

I'm experiencing a strange problem where the IcaFixProcessing is now failing where previously it has been running fine.

I have been processing a large dataset running the HCP pipeline, the last few subjects have reliably failed in the same way at the point of running fix.

It appears to be failing to generate the cleaned filtered_func_data and then subsequent steps fail to find necessary files, but I have no idea why or how this has started happening.

This is the output of .fix.log from a failed subject

< M A T L A B (R) >
Copyright 1984-2019 The MathWorks, Inc.
R2019b Update 5 (9.7.0.1319299) 64-bit (glnxa64)
February 19, 2020

For online documentation, see https://www.mathworks.com/support
For product information, visit www.mathworks.com.

aggressive = 0
domot = 0
hp = -1.000000
DOvol = 1
TR = 0.800000
{ Error using file_array/permute (line 10)
file_array objects can not be permuted.

Error in read_gifti_file>gifti_Data (line 206)
d = permute(reshape(d,fliplr(s.Dim)),length(s.Dim):-1:1);

Error in read_gifti_file>gifti_DataArray (line 126)
s(1).data = gifti_Data(t,c(i),s(1).attributes);

Error in read_gifti_file (line 45)
this.data{end+1} = gifti_DataArray(t,uid(i),filename);

Error in gifti (line 100)
this = read_gifti_file(varargin{1},giftistruct);

Error in ciftiopen (line 31)
cifti = gifti([tmpfile '.gii']);

Error in fix_3_clean (line 62)
BO=ciftiopen('Atlas.dtseries.nii',WBC);
}
>>

This is the .fix.log output from a successful subject.

< M A T L A B (R) >
Copyright 1984-2019 The MathWorks, Inc.
R2019b Update 5 (9.7.0.1319299) 64-bit (glnxa64)
February 19, 2020

For online documentation, see https://www.mathworks.com/support
For product information, visit www.mathworks.com.

aggressive = 0
domot = 0
hp = -1.000000
DOvol = 1
TR = 0.800000
Elapsed time is 5.047619 seconds.

ans =

'unaggressive cleanup'

Elapsed time is 10.815345 seconds.
Elapsed time is 2.094347 seconds.
>>

Here is the command line output when running a (failed) fix on a subject (the first part lists settings and all versions of software/pipelines etc):-

cr291@sn-analysis01:/research/cisc2/projects/ward_connectomesyn/scripts_batches/hcpscripts/subjectscripts/CISC28809$ ./IcaFixProcessingBatch_28809.sh
-- IcaFixProcessingBatch_28809.sh: Specified Command-Line Options: -- Start --
StudyFolder: /research/cisc1/projects/ward_connectomesyn/hcpdata
Subjlist: 28809
EnvironmentScript: /usr/local/hcp/SetUpHCPPipeline.sh
RunLocal: TRUE
-- IcaFixProcessingBatch_28809.sh: Specified Command-Line Options: -- End --
28809
InputFile: /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_PA/rfMRI_REST1_PA@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_AP/rfMRI_REST2_AP@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_PA/rfMRI_REST2_PA
About to run: /usr/local/hcp/ICAFIX/hcp_fix_multi_run /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_PA/rfMRI_REST1_PA@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_AP/rfMRI_REST2_AP@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_PA/rfMRI_REST2_PA 2000 /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST/rfMRI_REST FALSE HCP_hp2000.RData 10 FALSE
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: HCPPIPEDIR: /usr/local/hcp
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: CARET7DIR: /usr/local/workbench/bin_rh_linux64
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: FSLDIR: /usr/local/fsl
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: FSL_FIXDIR: /usr/local/fix
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: Showing HCP Pipelines version
v4.3.0
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: Showing Connectome Workbench (wb_command) version
Connectome Workbench
Type: Command Line Application
Version: 1.5.0
Qt Compiled Version: 5.7.0
Qt Runtime Version: 5.7.0
Commit: 76441605d20536d81034a64c9d34ec393aa528b9
Commit Date: 2021-02-16 13:46:47 -0600
Compiled with OpenMP: YES
Compiler: g++ (/export/gcc-4.8.5/bin)
Compiler Version: 4.8.5
Compiled Debug: NO
Operating System: Linux
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: Showing FSL version
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: INFO: Determined that the FSL version in use is 6.0.4:ddd0a010
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: FSL version: 6.0.4:ddd0a010
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: FIX version: 1.06.15
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: FSL_FIX_MATLAB_MODE: 1
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: fMRI Names: /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_PA/rfMRI_REST1_PA /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_AP/rfMRI_REST2_AP /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_PA/rfMRI_REST2_PA
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: highpass: 2000
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: ConcatName: /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST/rfMRI_REST
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: doMotionRegression: FALSE
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: TrainingData: HCP_hp2000.RData
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: FixThresh: 10
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: DeleteIntermediates: FALSE
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: PWD: /research/cisc2/projects/ward_connectomesyn/scripts_batches/hcpscripts/subjectscripts/CISC28809
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: Looping over files and doing highpass to each of them
Sat Nov 6 21:51:47 GMT 2021:hcp_fix_multi_run: Top of loop through fmris: fmri: /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP
parsed 'TCS - MEAN' as 'TCS - MEAN'
Sat Nov 6 21:53:28 GMT 2021:hcp_fix_multi_run: tr: 0.800000
Sat Nov 6 21:53:28 GMT 2021:hcp_fix_multi_run: processing FMRI file rfMRI_REST1_AP with highpass 2000
Sat Nov 6 21:53:28 GMT 2021:hcp_fix_multi_run: Run interpreted MATLAB/Octave with command...
Sat Nov 6 21:53:28 GMT 2021:hcp_fix_multi_run: /usr/local/fix/call_matlab.sh -r addpath('/usr/local/hcp/global/matlab'); addpath('/usr/local/hcp/ICAFIX/scripts'); functionhighpassandvariancenormalize(0.800000 , '2000', 'rfMRI_REST1_AP', '/usr/local/workbench/bin_rh_linux64/wb_command');

<<<EDITED OUT LOTS OF HIGH-PASS LOOP OUTPUT>>>

Sun Nov 7 02:00:19 GMT 2021:hcp_fix_multi_run: Running FIX
Sun Nov 7 02:00:19 GMT 2021:hcp_fix_multi_run: using training data file: /usr/local/fix/training_files/HCP_hp2000.RData
Sun Nov 7 02:00:19 GMT 2021:hcp_fix_multi_run: fix_cmd: /usr/local/fix/fix rfMRI_REST_hp2000.ica /usr/local/fix/training_files/HCP_hp2000.RData 10
Sun Nov 7 02:00:19 GMT 2021:hcp_fix_multi_run: Check rfMRI_REST_hp2000.ica/fix/logMatlab.txt for log output from feature extraction
Sun Nov 7 02:00:19 GMT 2021:hcp_fix_multi_run: Check rfMRI_REST_hp2000.ica/.fix_2b_predict.log for log output from component classification
Sun Nov 7 02:00:19 GMT 2021:hcp_fix_multi_run: Check rfMRI_REST_hp2000.ica/.fix.log for log output from cleanup stage
FIX Feature extraction for Melodic output directory: rfMRI_REST_hp2000.ica
create edge masks
run FAST
registration of standard space masks
extract features
FIX Classifying components in Melodic directory: rfMRI_REST_hp2000.ica using training file: /usr/local/fix/training_files/HCP_hp2000.RData and threshold 10
FIX Applying cleanup using cleanup file: rfMRI_REST_hp2000.ica/fix4melview_HCP_hp2000_thr10.txt and motion cleanup set to 0
Sun Nov 7 02:42:27 GMT 2021:hcp_fix_multi_run: Done running FIX
Could not find a supported file with prefix "rfMRI_REST_hp2000.ica/filtered_func_data_clean"
Sun Nov 7 02:42:28 GMT 2021:hcp_fix_multi_run: While running '/usr/local/hcp/ICAFIX/hcp_fix_multi_run /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_AP/rfMRI_REST1_AP@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST1_PA/rfMRI_REST1_PA@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_AP/rfMRI_REST2_AP@/research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST2_PA/rfMRI_REST2_PA 2000 /research/cisc1/projects/ward_connectomesyn/hcpdata/28809/MNINonLinear/Results/rfMRI_REST/rfMRI_REST FALSE HCP_hp2000.RData 10 FALSE':
Sun Nov 7 02:42:28 GMT 2021:hcp_fix_multi_run: ERROR: '/usr/local/fsl/bin/immv' command failed with return code: 1

===> ERROR: Command returned with nonzero exit code
---------------------------------------------------
script: hcp_fix_multi_run
stopped at line: 793
call: $FSLDIR/bin/immv ${concatfmrihp}.ica/filtered_func_data_clean ${concatfmrihp}_clean
expanded call: /usr/local/fsl/bin/immv rfMRI_REST_hp2000.ica/filtered_func_data_clean rfMRI_REST_hp2000_clean
exit code: 1
---------------------------------------------------

===> Aborting execution!

I dont believe I have made any changes to settings throughout the HCP or fix folders, and when I inspect the 'date modified' of all files in these folders nothing has been modified recently. All core versions of the software (FSL/HCP/Workbench/MATLAB) haven't been updated recently. The logs show that all previous steps of the pipeline completed as normal without error.

I even tried re-running the fix step on a subject that previously worked and it failed in the same way as above so it seems to be unrelated to the data.

The only thing I have been doing recently is starting experimenting with using ApplyHandReClassification.sh. I have only done this on one subject so far and dont believe I've changed any settings or installation files.

Do you have any idea what could be causing this? Or how I can go about debugging?

Best wishes,

Chris

Glasser, Matt

unread,

Nov 7, 2021, 6:32:06 PM11/7/21

to hcp-...@humanconnectome.org

Is this interactive or compiled matlab?

Matt.

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/cdc97c41-9466-4b95-ac12-1a26079a1a09n%40humanconnectome.org.

The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Chris Racey

unread,

Nov 7, 2021, 7:39:46 PM11/7/21

to HCP-Users, glas...@wustl.edu

Interactive I believe, MATLAB_MODE=1

Glasser, Matt

unread,

Nov 7, 2021, 7:41:57 PM11/7/21

to hcp-...@humanconnectome.org

It seems like something went wrong with the GIFTI or CIFTI reader/writer.

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/26542cac-7ce1-42a5-980e-1639aa2f06f6n%40humanconnectome.org.

Chris Racey

unread,

Nov 8, 2021, 6:55:54 PM11/8/21

to HCP-Users, glas...@wustl.edu

I have cifti read/write functions in /usr/local/hcp/global/matlab and /opt/fmrib/matlab_ciftireadwrite. The fmrib folder is the one specified in the fix settings.sh but appears to be an older version than the one in the hcp folder.

Both contain files that are unmodified for many months and I did some tests reading/writing cifti files in matlab with each and they seem to work ok. It's a bit of a mystery why the ciftireadwrite functions would work fine for many subjects and then suddenly stop.

I have tried updating settings to use the newer version in /usr/local/hcp/global/matlab, but after running fix again on a subject I get the same error as below. It's consistently failing on fix no matter what I do.

I'm not sure what could have changed and I dont even know what to try next? If you have any suggestions I would be very grateful?

Best wishes,

Chris

Coalson, Timothy Scott (S&T-Student)

unread,

Nov 8, 2021, 8:30:48 PM11/8/21

to HCP-Users, glas...@wustl.edu

Fix has its own settings, including for compiled versus interpreted matlab in fix/settings.sh, and defaults to compiled, so changing what path to use for the cifti functions won't have an effect by default. The fix tarball doesn't appear to include the .m files for cifti they compiled against, so it is hard to say what might be the condition that triggers that code.

In the current gifti library on github (and the version in the hcp pipelines), the line that uses file_array is basically commented out (in the "else" of an "if true").

Tim

From: Chris Racey <c.ra...@gmail.com>
Sent: Monday, November 8, 2021 5:55 PM
To: HCP-Users <hcp-...@humanconnectome.org>
Cc: glas...@wustl.edu <glas...@wustl.edu>

Subject: Re: [hcp-users] Problem with Fix processing suddenly not working

WARNING: This message has originated from an External Source. This may be a phishing expedition that can result in unauthorized access to our IT System. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/8705c09f-3718-446e-8853-708dae850b58n%40humanconnectome.org.

Harms, Michael

unread,

Nov 8, 2021, 10:41:29 PM11/8/21

to hcp-...@humanconnectome.org, Glasser, Matt

He’s running multi-run FIX in interpreted matlab mode.

The root error is a bit puzzling:

{ Error using file_array/permute (line 10)
file_array objects can not be permuted.

--

Michael Harms, Ph.D.

-----------------------------------------------------------

Associate Professor of Psychiatry

Washington University School of Medicine

Department of Psychiatry, Box 8134

660 South Euclid Ave. Tel: 314-747-6173

St. Louis, MO 63110 Email: mha...@wustl.edu

From: "Coalson, Timothy Scott (S&T-Student)" <tsc...@mst.edu>
Reply-To: "hcp-...@humanconnectome.org" <hcp-...@humanconnectome.org>
Date: Monday, November 8, 2021 at 7:30 PM
To: HCP-Users <hcp-...@humanconnectome.org>
Cc: "Glasser, Matt" <glas...@wustl.edu>
Subject: Re: [hcp-users] Problem with Fix processing suddenly not working

* External Email - Caution *

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/SN6PR01MB4991A1AB674427FBD025645C81929%40SN6PR01MB4991.prod.exchangelabs.com.

Coalson, Timothy Scott (S&T-Student)

unread,

Nov 8, 2021, 11:07:25 PM11/8/21

to hcp-...@humanconnectome.org, Glasser, Matt

The versions of the gifti library on github implement things in read_gifti_file_standalone (now gifti_read), not in read_gifti_file, and I haven't found a version that uses file_array by default (the version that added file_array put it in that "else" of an "if true" immediately, and left it there).

So, there is some kind of complicated backstory to whatever version of the gifti library is actually getting used. Maybe the right first step is to add "which gifti" to the fix_3_clean.m function.

Tim

From: Harms, Michael <mha...@wustl.edu>
Sent: Monday, November 8, 2021 9:41 PM
To: hcp-...@humanconnectome.org <hcp-...@humanconnectome.org>

To view this discussion on the web visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/66E8EDA5-4052-4BEC-AB7F-BBA8EC5F1EC9%40wustl.edu.

Chris Racey

unread,

Nov 10, 2021, 9:41:29 AM11/10/21

to HCP-Users, tsc...@mst.edu, glas...@wustl.edu

Hi Michael, Tim & Matt,

Many thanks for your all of your help, I got to the bottom of it thanks to your suggestions Tim.

As expected, it was my foolish mistake. The version of gifti tools that was being used was the ones packaged with SPM12. A few weeks ago I made some changes to the default matlab path via startup.m for another project/analysis. I had completely forgotten about it and not realised the impact it would have on the HCP pipeline.

Lessons have been learned! :-)

Many thanks again!

Chris

Reply all

Reply to author

Forward