Fiber tracking mismatch between linux(singularity) and windows versions

54 views
Skip to first unread message

Hamsi Radhakrishnan

unread,
Apr 5, 2022, 4:02:45 PMApr 5
to DSI Studio
Hi Frank!

I am trying to automatically track fiber bundles, individually for the whole brain, from previously created .fib.gz files using this command (and the latest singularity version of DSIstudio (03/21/22) on a linux machine):
for trk in Fasciculus Cingulum Aslant Corticos Thalamic_R Reticular Optic Fornix Corpus; do
fib=dsi_extrapolated.src.gz.gqi.1.25.fib.gz
dsi_studio --action=atk --source=${fib} --track_id=$trk 
done

This works beautifully for most participants and most tracts, but it sometimes randomly fails at certain bundles for random participants- and I get empty text files (and no .tt.gz outputs) with the suffix "no_result" instead. It seems to fail at different bundles for different participants.
I thought this meant that it couldn't detect these bundles for that participant, for whatever reason- but when I tried uploading the same fib file on the DSIstudio GUI on my windows machine, did automated fiber tracking and targeted the same bundle- it worked just fine and was able to give me an appropriate result. Any idea why this might be happening? More than happy to share more information if needed!

Thank you so much!

Regards,
Hamsi
  

Frank Yeh

unread,
Apr 5, 2022, 4:23:46 PMApr 5
to dsi-s...@googlegroups.com
Hi Hamsi,

The CLI interface has one additional check that requires tracts to
end at a location whether anisotropy is lower than the anisotropy
threshold. This removes tracts with "premature termination" but will
increase the chance of no finding. GUI has this "check ending"
turned off by default, thus causing the differences.

I can open up a CLI parameter for you that can turn this "check
ending" off with --action=atk, or if you have any suggestions/ideas, I
will modify DSI Studio accordingly.

Please feel free to let me know.

Best,
Frank
> --
> You received this message because you are subscribed to the Google Groups "DSI Studio" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dsi-studio+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/913e6e75-8bbd-495a-aeb7-0c3c33adc089n%40googlegroups.com.

Hamsi

unread,
Apr 5, 2022, 4:40:24 PMApr 5
to dsi-s...@googlegroups.com, Cieslak, Matthew

Hi Frank,

 

That’s exactly what we need- thank you so much! As an aside, is there a way to turn that on in the GUI, so we can compare the outputs of that parameter turned on and off in an output that worked?

 

Cheers,

Hamsi

 

Sent from Mail for Windows

You received this message because you are subscribed to a topic in the Google Groups "DSI Studio" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/dsi-studio/TUnsMb8TJ1I/unsubscribe.

To unsubscribe from this group and all its topics, send an email to dsi-studio+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/CAG_QrtKJSh6G24h1hi0_Nh4zyWAxuADik851Ef%3DRFOZgkpcPXQ%40mail.gmail.com.

 

Frank Yeh

unread,
Apr 5, 2022, 4:51:30 PMApr 5
to dsi-s...@googlegroups.com, Cieslak, Matthew
[Step T3c Options][Tracking Parameters][Advanced Options][Check Ending]

I am going to push a commit that allows you to specify
--check_ending=0 with --action=atk

Best,
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/E2E1CD23-18B0-475D-AA9D-613248ABB9CE%40hxcore.ol.

Hamsi Radhakrishnan

unread,
Apr 5, 2022, 6:28:02 PMApr 5
to DSI Studio
Thank you so much!!!

Frank Yeh

unread,
Apr 5, 2022, 7:40:37 PMApr 5
to dsi-s...@googlegroups.com
The new version is building (status:
https://github.com/frankyeh/DSI-Studio/actions/runs/2099368178 )
Once complete, please update DSI Studio and you can specify
--check_ending=0 with --action=atk

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/2562c279-ecfb-4856-8539-1c34db53cbf6n%40googlegroups.com.

Hamsi Radhakrishnan

unread,
Apr 6, 2022, 4:11:14 PMApr 6
to DSI Studio
Thank you so much! Looks like it is done building- one last question, how would I pull this specific build release as a singularity? I typically just do:  singularity pull docker://dsistudio/dsistudio:latest, but I am guessing that wouldn't work here. 

Frank Yeh

unread,
Apr 6, 2022, 4:15:28 PMApr 6
to dsi-s...@googlegroups.com
That is the right command. 
Singularity will update the sif file once you call the pull command.

Hamsi

unread,
Apr 12, 2022, 2:45:56 PMApr 12
to dsi-s...@googlegroups.com, Cieslak, Matthew

Hi Frank!

 

Thank you so much for your help with all this- it looks like the fix should work, but this command now creates a lot of core.* files with the new singularity- and ultimately results in the following error: 

terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
/var/tmp/gridengine/8.1.9-1/default/spool/2115fmn004/job_scripts/141928: line 43: 224406 Aborted                 $sing dsi_studio --action=atk --source=${fib} --track_id=$trk --thread_count=$NSLOTS --check_ending=0

 

Turning core file creation using ulimit -c 0 doesn't seem to help this. This error also only occurs when I send batch jobs to an SGE cluster, works just fine locally- and even succeeds on random jobs on the grid. I cannot tell if this is an error in our compute set up or in the singularity, but we didn’t have this issue with the previous singularity image (without the check_ending utility). I was wondering if you had any insight on this? Happy to provide more information if needed!

 

Thank you so much for all your help again!

Frank Yeh

unread,
Apr 12, 2022, 2:58:52 PMApr 12
to dsi-s...@googlegroups.com, Cieslak, Matthew
Likely the job used too much memory or disk space, and thus the job
scheduler aborted it.
There could be a place in the job scheduler to increase the upper
limit of the memory.
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/FCB27B2C-CC0C-4D87-ACB9-DAF622D3BFDB%40hxcore.ol.

Hamsi Radhakrishnan

unread,
Apr 12, 2022, 3:16:01 PMApr 12
to DSI Studio
I did think this might be the case, and tried it with a LOT of memory, and still got the same error. It's particularly strange because it works in some jobs and fails in others- even though the calls are almost identical to each other with just different fib files of the exact same size etc. And also because the exact same set of calls works with the older singularity image (for specific bundles).

Frank Yeh

unread,
Apr 12, 2022, 3:24:14 PMApr 12
to dsi-s...@googlegroups.com
I see. This is indeed weird.
What is the log output before the error?
I can trace into the code and see what part went wrong.
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/a3c78685-eb0a-4649-877b-3617071c46een%40googlegroups.com.

Hamsi Radhakrishnan

unread,
Apr 12, 2022, 4:06:59 PMApr 12
to DSI Studio
Here's the output for an example failed run:
DSI Studio "Chen" Apr  5 2022
source=/cbica/projects/csdsi/crash_fibs/qsirecon/qsirecon/sub-3832y/ses-7/dwi/sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi.fib.gz
action=atk
track_id=Fasciculus
target tracks: Arcuate_Fasciculus_L Arcuate_Fasciculus_R Inferior_Fronto_Occipital_Fasciculus_L Inferior_Fronto_Occipital_Fasciculus_R Inferior_Longitudinal_Fasciculus_L Inferior_Longitudinal_Fasciculus_R Middle_Longitudinal_Fasciculus_L Middle_Longitudinal_Fasciculus_R Superior_Longitudinal_Fasciculus1_L Superior_Longitudinal_Fasciculus1_R Superior_Longitudinal_Fasciculus2_L Superior_Longitudinal_Fasciculus2_R Superior_Longitudinal_Fasciculus3_L Superior_Longitudinal_Fasciculus3_R Uncinate_Fasciculus_L Uncinate_Fasciculus_R Vertical_Occipital_Fasciculus_L Vertical_Occipital_Fasciculus_R
thread_count=5
export_template_trk=0
default_mask=0
overwrite=0
export_trk=1
export_stat=1
tip=32
track_voxel_ratio=2
tolerance=16,18,20
length_ratio=1.25
automatic fiber tracking
sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi
processing sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi
tracking pathways
Arcuate_Fasciculus_L
loading sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi.fib.gz
using index file for accelerated loading:/cbica/projects/csdsi/crash_fibs/qsirecon/qsirecon/sub-3832y/ses-7/dwi/sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi.fib.gz.idx
FIB file loaded
loading ICBM152.tt.gz
warping atlas tracks to subject space
downsampling template by 2x to match subject resolution
running normalization
iso loaded
FOV width:156 to 194

terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
/var/tmp/gridengine/8.1.9-1/default/spool/2119fmn002/job_scripts/98514: line 40: 204466 Aborted                 $sing dsi_studio --action=atk --source=${fib} --track_id=$trk --thread_count=5

It always seems to fail after the "FOV width..." line!

Frank Yeh

unread,
Apr 12, 2022, 4:19:33 PMApr 12
to dsi-s...@googlegroups.com
Thanks a lot. This is very helpful.
The problem happened when DSI Studio started to run a linear
registration before nonlinear registration.
I am checking the code and see if I can spot a bug.
Best,
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/4052a1a6-2847-4469-bde4-e01efd1a921cn%40googlegroups.com.

Frank Yeh

unread,
Apr 12, 2022, 4:41:13 PMApr 12
to dsi-s...@googlegroups.com
Hi Hamsi,

The error, based on the message and the log, is likely due to DSI
Studio creating multiple threads for linear registration that exceeds
the system limit. But I could not reproduce the problem on my end.

Did your job scheduler impose any limit on the number of concurrent threads?

A way to test this hypothesis is to run a simple fiber tracking
with thread_count=32 or 64 (e.g. --action=trk --source=1.fib.gz
--thread_count=64)

Best,
Frank

Frank Yeh

unread,
Apr 12, 2022, 4:43:29 PMApr 12
to dsi-s...@googlegroups.com
BTW, if you remove --thread_count=5 from your command line, then DSI
Studio will show the number of "maximum hardware concurrency". This
way we can know how many threads were actually created when it
generated the error.

Hamsi Radhakrishnan

unread,
Apr 15, 2022, 12:09:27 PMApr 15
to DSI Studio
A way to test this hypothesis is to run a simple fiber tracking with thread_count=32 or 64 (e.g. --action=trk --source=1.fib.gz --thread_count=64)
This works!

"BTW, if you remove --thread_count=5 from your command line, then DSIStudio will show the number of "maximum hardware concurrency".
Results in thread_count=24. It still results in the same error, when I send it to the scheduler. 

Frank Yeh

unread,
Apr 15, 2022, 2:23:12 PMApr 15
to dsi-s...@googlegroups.com
Thanks for testing.
Does re-running the job still result in the same error?

Sorry about the problem, and hope we can figure out the cause.
Frank

Hamsi Radhakrishnan

unread,
Apr 15, 2022, 2:33:40 PMApr 15
to DSI Studio
Not always- this error only occurs when I send large batches of jobs, so I don't know if it's an issue with some interference between jobs. I am also able to run this on a single job locally- so it's confusing!

Thank you so much for all your help with this!

Frank Yeh

unread,
Apr 15, 2022, 3:38:02 PMApr 15
to dsi-s...@googlegroups.com
I see. It seems re-running the failed job is the current solution, but I will still keep this issue in mind and hopefully find a solution in the future.
Frank

Hamsi Radhakrishnan

unread,
May 20, 2022, 1:56:55 PMMay 20
to DSI Studio
Hi Frank,

Thank you so much for your help with this these past few months! We're running a fresh batch of subjects through the same pipeline, and are encountering a similar problem, even with check ending turned off. Only some tracks are being detected, while the same bunch of tracks provide "no_results" for almost all subjects. In this case, even the GUI is unable to detect these tracks with the current settings. These subjects are very similar to the previous subjects in terms of acquisition parameters, and the pipeline used- so I am not sure what could be causing this problem. The bundles that are not being detected seem like very common large bundles too  (Cingulum_Frontal_Parahippocampal, Fornix...). Are there any other parameters that can be changed to help solve this? Any feedback would be much appreciated! Thank you so much for all your help- and more than happy to provide any additional information!

Cheers,
Hamsi

Frank Yeh

unread,
May 20, 2022, 3:12:25 PMMay 20
to dsi-s...@googlegroups.com
Hi Hamsi,

Sorry about the problem. I will assist with my best to identify
the problem and fix it.

Is it possible to send me the SRC.GZ file that DSI Studio produces
no result?

Best,
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/1f89071b-2fee-4647-9778-b6a4709172b1n%40googlegroups.com.

Frank Yeh

unread,
May 23, 2022, 8:01:51 AMMay 23
to dsi-s...@googlegroups.com
Hi Hamsi,

I checked the data you uploaded. There is one very subtle but
critical problem in the b-table: a small rotation wrongly applied to
the b-table.

The SRC file was created from FSL's bvec using an older DSI
Studio, which does not flip bvec in the y-direction. Usually, this is
not an issue, because there is a "check b-table" function that can
detect and correct it by flip in the y-direction. However, here the
SRC files also appear to be rotated to align AC-PC, and the rotation
matrix was likely done in DSI Studio or another tool that also assumes
LPS orientation. Since the bvec is still in FSL's LAS-orientation, the
LPS-based rotation applied to the LAS-based b-table will mess up the
rotation.

The solution is to re-gerenerate new SRC files from the upstream
data source (e.g. FSL's NIFTI/bvec/bval). You may need to use the most
updated DSI Studio to re-generate the SRC file again from
NIFTI/bval/bvec. The older version of DSI Studio did not flip bvec
when using it to create the SRC files, thus causing this problem. The
updated DSI Studio will flip the bvec when reading it to avoid the
problem in the following ac-pc rotation.

All your SRC files may also have this problem and need to be
re-generated.

If there is any question, please feel free to let me know.

Best,
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/1f89071b-2fee-4647-9778-b6a4709172b1n%40googlegroups.com.

Hamsi

unread,
May 23, 2022, 9:44:01 AMMay 23
to dsi-s...@googlegroups.com

Hey Frank,

 

Thank you so much for your quick response! I’ve uploaded an example src file using the drop box upload form. I generate the fib files from these using this command:

dsi_studio --action=rec --source=${src} --method=4 --param0=1.25 --record_odf=0 --other_output=qa,nqa,dti_fa,md,ad,rd,gfa,iso,rdi

 

Thank you so much for your help!

 

Sent from Mail for Windows

 

From: Frank Yeh
Sent: Friday, May 20, 2022 3:12 PM
To: dsi-s...@googlegroups.com
Subject: Re: [dsi-studio] Fiber tracking mismatch between linux(singularity) and windows versions

 

Hi Hamsi,

> 

> --

> You received this message because you are subscribed to the Google Groups "DSI Studio" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to dsi-studio+...@googlegroups.com.

> To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/dsi-studio/1f89071b-2fee-4647-9778-b6a4709172b1n*40googlegroups.com__;JQ!!CzAuKJ42GuquVTTmVmPViYEvSg!MEW4dq83SHG41_ZXK73I9WpcfKUArTz0kGtog_L6lyq9HWm2NRvHfKBXc69kLEDKl6Su29ziOIAj4nSFvg$ .

 

--

You received this message because you are subscribed to a topic in the Google Groups "DSI Studio" group.

To unsubscribe from this topic, visit https://urldefense.com/v3/__https://groups.google.com/d/topic/dsi-studio/TUnsMb8TJ1I/unsubscribe__;!!CzAuKJ42GuquVTTmVmPViYEvSg!MEW4dq83SHG41_ZXK73I9WpcfKUArTz0kGtog_L6lyq9HWm2NRvHfKBXc69kLEDKl6Su29ziOIAcnLOz_A$ .

To unsubscribe from this group and all its topics, send an email to dsi-studio+...@googlegroups.com.

To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/dsi-studio/CAG_QrtLWCiqbqBdsMuJmXTgkrkUm-pROm5fX4O-oHWr8w8_K6w*40mail.gmail.com__;JQ!!CzAuKJ42GuquVTTmVmPViYEvSg!MEW4dq83SHG41_ZXK73I9WpcfKUArTz0kGtog_L6lyq9HWm2NRvHfKBXc69kLEDKl6Su29ziOICg2_Dqrg$ .

 

Matthew Cieslak

unread,
May 23, 2022, 10:37:51 AMMay 23
to DSI Studio
Hi Frank,

These bvecs came from qsiprep so they're in LPS+ orientation. In the official qsiprep workflows we use the special flag you added so the bvecs are interpreted literally (not as if they were in LAS+), but these src files were made directly in dsi studio without the flag.

Frank Yeh

unread,
May 23, 2022, 11:40:56 AMMay 23
to dsi-s...@googlegroups.com
The b-table in the SRC file has a problem, and thus this
reconstruction would not work out.
Unfortunately, you may need to re-generate new SRC files (I can check
it again for you to make sure it works).
All other SRC files, even though they may have small rotation, may
also need to be re-generated.
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/833A1930-C933-40E5-83A1-23961CF14AE7%40hxcore.ol.

Frank Yeh

unread,
May 23, 2022, 11:51:40 AMMay 23
to dsi-s...@googlegroups.com
Hi Matt,

I am sorry about the issue caused by DSI Studio.

The older version of DSI Studio will accept bvec "as is" without
flipping it by default, and it will work well with qsiprep.

However, the updated DSI Studio will assume all bvec are from FSL
and is LAS. It will flip it by default when reading/writing it.

> but these src files were made directly in dsi studio without the flag.

It would be okay if there is no further rotation to align AC-PC.
If there is a rotation, then LPS rotation will be applied to LAS bvec
and causing the problem.

To make ac-pc rotation work, when using --action=src to create the
SRC file, we may need to specify --flip_by=0 to use LPS bvec.

Best,
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/b986abea-0c87-4880-a1b6-5250fc642e1fn%40googlegroups.com.

Matthew Cieslak

unread,
May 23, 2022, 7:41:49 PMMay 23
to DSI Studio
I think in future releases of qsiprep I will write out the DSI Studio btable format in addition the the dipy bval/bvec and the mrtrix ".b" formats. Is the btable in LPS+?

Frank Yeh

unread,
May 23, 2022, 8:22:08 PMMay 23
to dsi-s...@googlegroups.com
Yes, it is LPS+.
DSI Studio only flips in y-direction if it reads the table from a bvec file.
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/844890c7-9951-4b98-a523-039e3753de3bn%40googlegroups.com.

Hamsi Radhakrishnan

unread,
May 26, 2022, 1:02:29 PMMay 26
to DSI Studio
Hi Frank,

Thank you so much for all your help! Your fix worked- we flipped in y, and now most tracks have been detected. It looks like the Cingulum Parahippocampal L/R, Vertical Occipital Fasciculus L/R and the Fornix L are still not being detected for some subjects, even after this fix. (This is still a lot better than the 20 or so tracks that were missing before). I was wondering if this was normal- that some tracks are just not detected for some acquisitions, or if this suggests that something is still wrong with our files? Happy to upload additional files if it helps! Thank you so much again!

Cheers,
Hamsi

Frank Yeh

unread,
May 26, 2022, 1:04:58 PMMay 26
to dsi-s...@googlegroups.com
It happens because the difficulties in mapping different pathways are
different. The most challenging ones are those long and curvy.
You may upload a sample SRC or FIB file, and I can double-check at my
side to confirm the quality.
Best,
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/576d67ee-dc95-49c0-9a5f-229e627bb1f6n%40googlegroups.com.

Frank Yeh

unread,
May 26, 2022, 11:50:40 PMMay 26
to dsi-s...@googlegroups.com
I checked the files. They look good!
Frank

Hamsi Radhakrishnan

unread,
May 26, 2022, 11:53:47 PMMay 26
to DSI Studio
Thank you so much!!!!!

Frank Yeh

unread,
10:15 AM (5 hours ago) 10:15 AM
to dsi-s...@googlegroups.com
Hi Hamsi,

There is a recent update in DSI Studio (today's version) to handle
the "no result" problem in --action=atk.

If you have cases with "no result" text files, you may delete
those text files and run atk again to see if updated DSI Studio
generates tracks for those cases.

Best regards,

Hamsi Radhakrishnan

unread,
10:22 AM (5 hours ago) 10:22 AM
to dsi-s...@googlegroups.com
Hi Frank!

Thank you for the update! We're done with analysis for this project right now, but just out of curiosity- what has changed in the script that might enable the updated version to detect those tracks?

Cheers,
Hamsi

Hamsi Radhakrishnan, Ph.D.
(pronounced hum-see; she/her)
Postdoctoral Fellow, Penn LINC
University of Pennsylvania



Frank Yeh

unread,
10:33 AM (5 hours ago) 10:33 AM
to dsi-s...@googlegroups.com
In automatic fiber tracking, DSI Studio detects if the "yield rate" is
lower than the threshold so that it can give up tracking earlier to
avoid spending time on them.

In the previous version, the yield rate calculation failed to consider
the tracts generated in the back buffer. Thus the estimated yield rate
is lower than the actual value, and DSI Studio may decide to terminate
even if the yield rate seems okay.

The tracking results (if generated) are still good, and the update
will help make up those with "no result" from the previous version.

The bug affects the version between Feb and June this year.

Best regards,
Frank
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/CAEzx_n%2B8Jg7w4q27QJbEC%2BEcLjyr_9PmF9%2BKLCZPa5vfKWrccA%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages