Fiber tracking mismatch between linux(singularity) and windows versions

[Hamsi Radhakrishnan]

Hi Frank!

I am trying to automatically track fiber bundles, individually for the whole brain, from previously created .fib.gz files using this command (and the latest singularity version of DSIstudio (03/21/22) on a linux machine):

for trk in Fasciculus Cingulum Aslant Corticos Thalamic_R Reticular Optic Fornix Corpus; do

fib=dsi_extrapolated.src.gz.gqi.1.25.fib.gz

dsi_studio --action=atk --source=${fib} --track_id=$trk 

done

This works beautifully for most participants and most tracts, but it sometimes randomly fails at certain bundles for random participants- and I get empty text files (and no .tt.gz outputs) with the suffix "no_result" instead. It seems to fail at different bundles for different participants.

I thought this meant that it couldn't detect these bundles for that participant, for whatever reason- but when I tried uploading the same fib file on the DSIstudio GUI on my windows machine, did automated fiber tracking and targeted the same bundle- it worked just fine and was able to give me an appropriate result. Any idea why this might be happening? More than happy to share more information if needed!

Thank you so much!

Regards,

Hamsi

  

[Frank Yeh]

Hi Hamsi,

The CLI interface has one additional check that requires tracts to
end at a location whether anisotropy is lower than the anisotropy
threshold. This removes tracts with "premature termination" but will
increase the chance of no finding. GUI has this "check ending"
turned off by default, thus causing the differences.

I can open up a CLI parameter for you that can turn this "check
ending" off with --action=atk, or if you have any suggestions/ideas, I
will modify DSI Studio accordingly.

Please feel free to let me know.

Best,
Frank

> --
> You received this message because you are subscribed to the Google Groups "DSI Studio" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dsi-studio+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/913e6e75-8bbd-495a-aeb7-0c3c33adc089n%40googlegroups.com.

[Hamsi]

Hi Frank,

 

That’s exactly what we need- thank you so much! As an aside, is there a way to turn that on in the GUI, so we can compare the outputs of that parameter turned on and off in an output that worked?

 

Cheers,

Hamsi

 

Sent from Mail for Windows

You received this message because you are subscribed to a topic in the Google Groups "DSI Studio" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/dsi-studio/TUnsMb8TJ1I/unsubscribe.

To unsubscribe from this group and all its topics, send an email to dsi-studio+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/CAG_QrtKJSh6G24h1hi0_Nh4zyWAxuADik851Ef%3DRFOZgkpcPXQ%40mail.gmail.com.

 

[Frank Yeh]

[Step T3c Options][Tracking Parameters][Advanced Options][Check Ending]

I am going to push a commit that allows you to specify
--check_ending=0 with --action=atk

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/E2E1CD23-18B0-475D-AA9D-613248ABB9CE%40hxcore.ol.

[Hamsi Radhakrishnan]

Thank you so much!!!

[Frank Yeh]

The new version is building (status:
https://github.com/frankyeh/DSI-Studio/actions/runs/2099368178 )
Once complete, please update DSI Studio and you can specify

--check_ending=0 with --action=atk

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/2562c279-ecfb-4856-8539-1c34db53cbf6n%40googlegroups.com.

[Hamsi Radhakrishnan]

Thank you so much! Looks like it is done building- one last question, how would I pull this specific build release as a singularity? I typically just do:  singularity pull docker://dsistudio/dsistudio:latest, but I am guessing that wouldn't work here. 

[Frank Yeh]

That is the right command. 

Singularity will update the sif file once you call the pull command.

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/9c353717-62be-474d-b931-dfab338a1842n%40googlegroups.com.

[Hamsi]

Hi Frank!

 

Thank you so much for your help with all this- it looks like the fix should work, but this command now creates a lot of core.* files with the new singularity- and ultimately results in the following error: 

terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
/var/tmp/gridengine/8.1.9-1/default/spool/2115fmn004/job_scripts/141928: line 43: 224406 Aborted                 $sing dsi_studio --action=atk --source=${fib} --track_id=$trk --thread_count=$NSLOTS --check_ending=0

 

Turning core file creation using ulimit -c 0 doesn't seem to help this. This error also only occurs when I send batch jobs to an SGE cluster, works just fine locally- and even succeeds on random jobs on the grid. I cannot tell if this is an error in our compute set up or in the singularity, but we didn’t have this issue with the previous singularity image (without the check_ending utility). I was wondering if you had any insight on this? Happy to provide more information if needed!

 

Thank you so much for all your help again!

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/CAG_QrtJ6uXUm%3DQDNEL0JTJdUbS8Dy%2BBO%2BZaZuMz88z9wBijkZA%40mail.gmail.com.

 

[Frank Yeh]

Likely the job used too much memory or disk space, and thus the job
scheduler aborted it.
There could be a place in the job scheduler to increase the upper
limit of the memory.
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/FCB27B2C-CC0C-4D87-ACB9-DAF622D3BFDB%40hxcore.ol.

[Hamsi Radhakrishnan]

I did think this might be the case, and tried it with a LOT of memory, and still got the same error. It's particularly strange because it works in some jobs and fails in others- even though the calls are almost identical to each other with just different fib files of the exact same size etc. And also because the exact same set of calls works with the older singularity image (for specific bundles).

[Frank Yeh]

I see. This is indeed weird.
What is the log output before the error?
I can trace into the code and see what part went wrong.
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/a3c78685-eb0a-4649-877b-3617071c46een%40googlegroups.com.

[Hamsi Radhakrishnan]

Here's the output for an example failed run:

DSI Studio "Chen" Apr  5 2022
source=/cbica/projects/csdsi/crash_fibs/qsirecon/qsirecon/sub-3832y/ses-7/dwi/sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi.fib.gz
action=atk
track_id=Fasciculus
target tracks: Arcuate_Fasciculus_L Arcuate_Fasciculus_R Inferior_Fronto_Occipital_Fasciculus_L Inferior_Fronto_Occipital_Fasciculus_R Inferior_Longitudinal_Fasciculus_L Inferior_Longitudinal_Fasciculus_R Middle_Longitudinal_Fasciculus_L Middle_Longitudinal_Fasciculus_R Superior_Longitudinal_Fasciculus1_L Superior_Longitudinal_Fasciculus1_R Superior_Longitudinal_Fasciculus2_L Superior_Longitudinal_Fasciculus2_R Superior_Longitudinal_Fasciculus3_L Superior_Longitudinal_Fasciculus3_R Uncinate_Fasciculus_L Uncinate_Fasciculus_R Vertical_Occipital_Fasciculus_L Vertical_Occipital_Fasciculus_R
thread_count=5
export_template_trk=0
default_mask=0
overwrite=0
export_trk=1
export_stat=1
tip=32
track_voxel_ratio=2
tolerance=16,18,20
length_ratio=1.25
automatic fiber tracking
sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi
processing sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi
tracking pathways
Arcuate_Fasciculus_L
loading sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi.fib.gz
using index file for accelerated loading:/cbica/projects/csdsi/crash_fibs/qsirecon/qsirecon/sub-3832y/ses-7/dwi/sub-3832y_ses-7_space-T1w_desc-preproc_space-T1w_gqi.fib.gz.idx
FIB file loaded
loading ICBM152.tt.gz
warping atlas tracks to subject space
downsampling template by 2x to match subject resolution
running normalization
iso loaded
FOV width:156 to 194

terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable

/var/tmp/gridengine/8.1.9-1/default/spool/2119fmn002/job_scripts/98514: line 40: 204466 Aborted                 $sing dsi_studio --action=atk --source=${fib} --track_id=$trk --thread_count=5

It always seems to fail after the "FOV width..." line!

[Frank Yeh]

Thanks a lot. This is very helpful.
The problem happened when DSI Studio started to run a linear
registration before nonlinear registration.
I am checking the code and see if I can spot a bug.
Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/4052a1a6-2847-4469-bde4-e01efd1a921cn%40googlegroups.com.

[Frank Yeh]

Hi Hamsi,

The error, based on the message and the log, is likely due to DSI
Studio creating multiple threads for linear registration that exceeds
the system limit. But I could not reproduce the problem on my end.

Did your job scheduler impose any limit on the number of concurrent threads?

A way to test this hypothesis is to run a simple fiber tracking
with thread_count=32 or 64 (e.g. --action=trk --source=1.fib.gz
--thread_count=64)

Best,
Frank

[Frank Yeh]

BTW, if you remove --thread_count=5 from your command line, then DSI
Studio will show the number of "maximum hardware concurrency". This
way we can know how many threads were actually created when it
generated the error.

[Hamsi Radhakrishnan]

A way to test this hypothesis is to run a simple fiber tracking with thread_count=32 or 64 (e.g. --action=trk --source=1.fib.gz --thread_count=64)

This works!

"BTW, if you remove --thread_count=5 from your command line, then DSIStudio will show the number of "maximum hardware concurrency".
Results in thread_count=24. It still results in the same error, when I send it to the scheduler. 

[Frank Yeh]

Thanks for testing.

Does re-running the job still result in the same error?

Sorry about the problem, and hope we can figure out the cause.

Frank

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/1284021a-dd02-4a58-987a-60ccf8eba3e9n%40googlegroups.com.

[Hamsi Radhakrishnan]

Not always- this error only occurs when I send large batches of jobs, so I don't know if it's an issue with some interference between jobs. I am also able to run this on a single job locally- so it's confusing!

Thank you so much for all your help with this!

[Frank Yeh]

I see. It seems re-running the failed job is the current solution, but I will still keep this issue in mind and hopefully find a solution in the future.

Frank

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/892783cf-bfc7-4b10-ba89-db6335c4c4b0n%40googlegroups.com.

[Hamsi Radhakrishnan]

Hi Frank,

Thank you so much for your help with this these past few months! We're running a fresh batch of subjects through the same pipeline, and are encountering a similar problem, even with check ending turned off. Only some tracks are being detected, while the same bunch of tracks provide "no_results" for almost all subjects. In this case, even the GUI is unable to detect these tracks with the current settings. These subjects are very similar to the previous subjects in terms of acquisition parameters, and the pipeline used- so I am not sure what could be causing this problem. The bundles that are not being detected seem like very common large bundles too  (Cingulum_Frontal_Parahippocampal, Fornix...). Are there any other parameters that can be changed to help solve this? Any feedback would be much appreciated! Thank you so much for all your help- and more than happy to provide any additional information!

Cheers,

Hamsi

[Frank Yeh]

Hi Hamsi,

Sorry about the problem. I will assist with my best to identify
the problem and fix it.

Is it possible to send me the SRC.GZ file that DSI Studio produces
no result?

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/1f89071b-2fee-4647-9778-b6a4709172b1n%40googlegroups.com.

[Frank Yeh]

Hi Hamsi,

I checked the data you uploaded. There is one very subtle but
critical problem in the b-table: a small rotation wrongly applied to
the b-table.

The SRC file was created from FSL's bvec using an older DSI
Studio, which does not flip bvec in the y-direction. Usually, this is
not an issue, because there is a "check b-table" function that can
detect and correct it by flip in the y-direction. However, here the
SRC files also appear to be rotated to align AC-PC, and the rotation
matrix was likely done in DSI Studio or another tool that also assumes
LPS orientation. Since the bvec is still in FSL's LAS-orientation, the
LPS-based rotation applied to the LAS-based b-table will mess up the
rotation.

The solution is to re-gerenerate new SRC files from the upstream
data source (e.g. FSL's NIFTI/bvec/bval). You may need to use the most
updated DSI Studio to re-generate the SRC file again from
NIFTI/bval/bvec. The older version of DSI Studio did not flip bvec
when using it to create the SRC files, thus causing this problem. The
updated DSI Studio will flip the bvec when reading it to avoid the
problem in the following ac-pc rotation.

All your SRC files may also have this problem and need to be
re-generated.

If there is any question, please feel free to let me know.

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/1f89071b-2fee-4647-9778-b6a4709172b1n%40googlegroups.com.

[Hamsi]

Hey Frank,

 

Thank you so much for your quick response! I’ve uploaded an example src file using the drop box upload form. I generate the fib files from these using this command:

dsi_studio --action=rec --source=${src} --method=4 --param0=1.25 --record_odf=0 --other_output=qa,nqa,dti_fa,md,ad,rd,gfa,iso,rdi

 

Thank you so much for your help!

 

Sent from Mail for Windows

 

From: Frank Yeh
Sent: Friday, May 20, 2022 3:12 PM
To: dsi-s...@googlegroups.com
Subject: Re: [dsi-studio] Fiber tracking mismatch between linux(singularity) and windows versions

 

Hi Hamsi,

>> > To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/dsi-studio/913e6e75-8bbd-495a-aeb7-0c3c33adc089n*40googlegroups.com__;JQ!!CzAuKJ42GuquVTTmVmPViYEvSg!MEW4dq83SHG41_ZXK73I9WpcfKUArTz0kGtog_L6lyq9HWm2NRvHfKBXc69kLEDKl6Su29ziOIBkpYNMsA$ .

> 

> --

> You received this message because you are subscribed to the Google Groups "DSI Studio" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to dsi-studio+...@googlegroups.com.

> To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/dsi-studio/1f89071b-2fee-4647-9778-b6a4709172b1n*40googlegroups.com__;JQ!!CzAuKJ42GuquVTTmVmPViYEvSg!MEW4dq83SHG41_ZXK73I9WpcfKUArTz0kGtog_L6lyq9HWm2NRvHfKBXc69kLEDKl6Su29ziOIAj4nSFvg$ .

 

--

You received this message because you are subscribed to a topic in the Google Groups "DSI Studio" group.

To unsubscribe from this topic, visit https://urldefense.com/v3/__https://groups.google.com/d/topic/dsi-studio/TUnsMb8TJ1I/unsubscribe__;!!CzAuKJ42GuquVTTmVmPViYEvSg!MEW4dq83SHG41_ZXK73I9WpcfKUArTz0kGtog_L6lyq9HWm2NRvHfKBXc69kLEDKl6Su29ziOIAcnLOz_A$ .

To unsubscribe from this group and all its topics, send an email to dsi-studio+...@googlegroups.com.

To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/dsi-studio/CAG_QrtLWCiqbqBdsMuJmXTgkrkUm-pROm5fX4O-oHWr8w8_K6w*40mail.gmail.com__;JQ!!CzAuKJ42GuquVTTmVmPViYEvSg!MEW4dq83SHG41_ZXK73I9WpcfKUArTz0kGtog_L6lyq9HWm2NRvHfKBXc69kLEDKl6Su29ziOICg2_Dqrg$ .

 

[Matthew Cieslak]

Hi Frank,

These bvecs came from qsiprep so they're in LPS+ orientation. In the official qsiprep workflows we use the special flag you added so the bvecs are interpreted literally (not as if they were in LAS+), but these src files were made directly in dsi studio without the flag.

[Frank Yeh]

The b-table in the SRC file has a problem, and thus this
reconstruction would not work out.
Unfortunately, you may need to re-generate new SRC files (I can check
it again for you to make sure it works).
All other SRC files, even though they may have small rotation, may
also need to be re-generated.
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/833A1930-C933-40E5-83A1-23961CF14AE7%40hxcore.ol.

[Frank Yeh]

Hi Matt,

I am sorry about the issue caused by DSI Studio.

The older version of DSI Studio will accept bvec "as is" without
flipping it by default, and it will work well with qsiprep.

However, the updated DSI Studio will assume all bvec are from FSL
and is LAS. It will flip it by default when reading/writing it.

> but these src files were made directly in dsi studio without the flag.

It would be okay if there is no further rotation to align AC-PC.
If there is a rotation, then LPS rotation will be applied to LAS bvec
and causing the problem.

To make ac-pc rotation work, when using --action=src to create the
SRC file, we may need to specify --flip_by=0 to use LPS bvec.

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/b986abea-0c87-4880-a1b6-5250fc642e1fn%40googlegroups.com.

[Matthew Cieslak]

I think in future releases of qsiprep I will write out the DSI Studio btable format in addition the the dipy bval/bvec and the mrtrix ".b" formats. Is the btable in LPS+?

[Frank Yeh]

Yes, it is LPS+.
DSI Studio only flips in y-direction if it reads the table from a bvec file.
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/844890c7-9951-4b98-a523-039e3753de3bn%40googlegroups.com.

[Hamsi Radhakrishnan]

Hi Frank,

Thank you so much for all your help! Your fix worked- we flipped in y, and now most tracks have been detected. It looks like the Cingulum Parahippocampal L/R, Vertical Occipital Fasciculus L/R and the Fornix L are still not being detected for some subjects, even after this fix. (This is still a lot better than the 20 or so tracks that were missing before). I was wondering if this was normal- that some tracks are just not detected for some acquisitions, or if this suggests that something is still wrong with our files? Happy to upload additional files if it helps! Thank you so much again!

Cheers,

Hamsi

[Frank Yeh]

It happens because the difficulties in mapping different pathways are
different. The most challenging ones are those long and curvy.
You may upload a sample SRC or FIB file, and I can double-check at my
side to confirm the quality.
Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/576d67ee-dc95-49c0-9a5f-229e627bb1f6n%40googlegroups.com.

[Frank Yeh]

I checked the files. They look good!
Frank

[Hamsi Radhakrishnan]

Thank you so much!!!!!

[Frank Yeh]

Hi Hamsi,

There is a recent update in DSI Studio (today's version) to handle
the "no result" problem in --action=atk.

If you have cases with "no result" text files, you may delete
those text files and run atk again to see if updated DSI Studio
generates tracks for those cases.

Best regards,

[Hamsi Radhakrishnan]

Hi Frank!

Thank you for the update! We're done with analysis for this project right now, but just out of curiosity- what has changed in the script that might enable the updated version to detect those tracks?

Cheers,

Hamsi

Hamsi Radhakrishnan, Ph.D.

(pronounced hum-see; she/her)

Postdoctoral Fellow, Penn LINC

University of Pennsylvania

Twitter | Website

> To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/dsi-studio/913e6e75-8bbd-495a-aeb7-0c3c33adc089n*40googlegroups.com__;JQ!!CzAuKJ42GuquVTTmVmPViYEvSg!IZE6540KPztzxW-_JvxZwJigmmEdb_uLgrjSnTdNRBTzk_PtHyFNQ1rSlG71BcrzCI0v1nui9o84NtrSUQ$ .

--
You received this message because you are subscribed to a topic in the Google Groups "DSI Studio" group.
To unsubscribe from this topic, visit https://urldefense.com/v3/__https://groups.google.com/d/topic/dsi-studio/TUnsMb8TJ1I/unsubscribe__;!!CzAuKJ42GuquVTTmVmPViYEvSg!IZE6540KPztzxW-_JvxZwJigmmEdb_uLgrjSnTdNRBTzk_PtHyFNQ1rSlG71BcrzCI0v1nui9o8jFyt4oA$ .
To unsubscribe from this group and all its topics, send an email to dsi-studio+...@googlegroups.com.
To view this discussion on the web visit https://urldefense.com/v3/__https://groups.google.com/d/msgid/dsi-studio/CAG_QrtKgZOn-5okykKDDX7PzkNaPc_hRbPRbGkgBZDE9d89Wqw*40mail.gmail.com__;JQ!!CzAuKJ42GuquVTTmVmPViYEvSg!IZE6540KPztzxW-_JvxZwJigmmEdb_uLgrjSnTdNRBTzk_PtHyFNQ1rSlG71BcrzCI0v1nui9o_WuebcUg$ .

[Frank Yeh]

In automatic fiber tracking, DSI Studio detects if the "yield rate" is
lower than the threshold so that it can give up tracking earlier to
avoid spending time on them.

In the previous version, the yield rate calculation failed to consider
the tracts generated in the back buffer. Thus the estimated yield rate
is lower than the actual value, and DSI Studio may decide to terminate
even if the yield rate seems okay.

The tracking results (if generated) are still good, and the update
will help make up those with "no result" from the previous version.

The bug affects the version between Feb and June this year.

Best regards,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/CAEzx_n%2B8Jg7w4q27QJbEC%2BEcLjyr_9PmF9%2BKLCZPa5vfKWrccA%40mail.gmail.com.

[Matthew Cieslak]

Hi Frank,

I was re-running this and think I found the issue! The registration and warping seem to be ignoring the thread limit and using all available threads on the compute host. The fiber tracking parts all are limited by --thread_count though. This was causing the resource error on our cluster. I was able to reproduce this behavior using a singularity image build from docker://dsistudio/dsistudio:chen-2022-11-21 with digest e191034158f3.

All the best,

[Fang-Cheng Yeh]

Hi Mat,

Thank you for reporting this issue. I am going to revise the code
and fix it.

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/40ad6b59-7831-4cf9-88a2-c86059b45679n%40googlegroups.com.

[Fang-Cheng Yeh]

Hi Mat,

I found that the command line code has been updated months ago to
fix this problem. What is the version date of DSI Studio having this
issue?
Thanks again for your help!

Best,
Frank

[Matthew Cieslak]

Hi Frank,

It is from dockerhub, where it says the image was uploaded 17 days ago. It has the same shasum as the latest tag, but I pulled from the chen-2022-11-21 tag.

[Fang-Cheng Yeh]

Could you post the output from DSI Studio when you run the --action=atk command?
This would greatly help me identify the issue.

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/05d9d271-ccae-4a51-89c7-506847a0e0f6n%40googlegroups.com.

[Matthew Cieslak]

of course! here is standard out:

+ ESC[1;34mDSI Studio version: Chen"陳" command lineESC[0m
| DSI Studio version: Chen"陳"
| action=atk
| source=/scratch/abcdqsiprep/SGE_2891324/data-2891324/sub-856706691_ses-PNC1_space-T1w_desc-preproc_gqi.fib.gz
| loop=/scratch/abcdqsiprep/SGE_2891324/data-2891324/sub-856706691_ses-PNC1_space-T1w_desc-preproc_gqi.fib.gz
| + ESC[1;34mrun atkESC[0m
| | track_id=Arcuate_Fasciculus_L
| | target tracks: Arcuate_Fasciculus_L
| | tolerance=22,26,30
| | track_voxel_ratio=2
| | yield_rate=1e-05
| | tip_iteration=48
| | export_stat=1
| | export_trk=1
| | overwrite=0
| | export_template_trk=0
| | check_ending=0
| | thread_count=5
| | trk_format=tt.gz
| | output=/scratch/abcdqsiprep/SGE_2891324/data-2891324
| | + ESC[1;34mautomatic fiber trackingESC[0m
| | | + ESC[1;34msub-856706691_ses-PNC1_space-T1w_desc-preproc_gqiESC[0m
| | | | processing sub-856706691_ses-PNC1_space-T1w_desc-preproc_gqi
| | | | + ESC[1;34mtracking pathwaysESC[0m
| | | | | + ESC[1;34mArcuate_Fasciculus_LESC[0m
| | | | | | + ESC[1;34mopen FIB fileESC[0m
| | | | | | | prepare index file for future accelerated loading
| | | | | | | + ESC[1;34mopening sub-856706691_ses-PNC1_space-T1w_desc-preproc_gqi.fib.gzESC[0m
| | | | | | | |_2.356 s
| | | | | | | saving index file for accelerated loading: sub-856706691_ses-PNC1_space-T1w_desc-preproc_gqi.fib.gz.idx

| | | | | | | + ESC[1;34mloading fiber and image dataESC[0m

| | | | | | | | + ESC[1;34mloading image volumesESC[0m
| | | | | | | | |_56 ms
| | | | | | | | + ESC[1;34minitiating dataESC[0m
| | | | | | | | |_4 ms
| | | | | | | |_61 ms
| | | | | | | FIB file loaded
| | | | | | |_2.423 s
| | | | | | + ESC[1;34mloading ICBM152_adult.tt.gzESC[0m
| | | | | | | + ESC[1;34mopening ICBM152_adult.tt.gzESC[0m
| | | | | | | |_57 ms
| | | | | | |_111 ms
| | | | | | + ESC[1;34mopening ICBM152_adult.QA.nii.gzESC[0m
| | | | | | |_0 ms
| | | | | | + ESC[1;34mreading image dataESC[0m
| | | | | | | + ESC[1;34mopening ICBM152_adult.ISO.nii.gzESC[0m
| | | | | | | |_0 ms
| | | | | | | + ESC[1;34mreading image dataESC[0m
| | | | | | | |_69 ms
| | | | | | |_194 ms
| | | | | | + ESC[1;34mrunning normalizationESC[0m
| | | | | | | [thread]iso loaded

And here is standard error:

+ singularity exec --cleanenv --containall -B /scratch/abcdqsiprep/SGE_2891324/data-2891324 /scratch/abcdqsiprep/SGE_2891324/dsistudio-chen-2022-11-21.sif dsi_studio --action=atk --source=/scratch/abcdqsiprep/SGE_2891324/data-2891324/sub-856706691_ses-PNC1_space-T1w_desc-preproc_gqi.fib.gz --check_ending=0 --track_id=Arcuate_Fasciculus_L --thread_count=5
terminate called recursively
terminate called recursively
/var/tmp/gridengine/8.1.9-1/default/spool/211affn019/job_scripts/2891324: line 52: 162826 Aborted                 $sing dsi_studio --action=atk --source=${LOCAL_FIB} --check_ending=0 --track_id=$bundle --thread_count=${DSI_STUDIO_THREADS}

[Fang-Cheng Yeh]

This is very helpful!! I have a much better clue on where the bug is and will fix it soon.

Best,

Frank

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/548cf966-6855-4fca-8097-19478b35ed13n%40googlegroups.com.

[Fang-Cheng Yeh]

Have you tried running docker with CPU limitations?
After reviewing the DSI Studio code, I found that it is very hard to
make sure the thread count is limited.
This would require changes on multiple analysis pipelines.

Best,
Frank

> To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/05d9d271-ccae-4a51-89c7-506847a0e0f6n%40googlegroups.com.

[Matthew Cieslak]

In this case we're using singularity on an HPC. We could try this out, but it would be extremely helpful to be able to limit the cpus from the commandline. Our HPC kills processes that attempt to use too many cpus, and it seems like DSI Studio detects the number of cpus on the compute host rather than the number allotted to it by the scheduler.

Would it be possible to run the registration part of ATK on its own with limited cpus? Then place the map and idx files in the same directory as the fib to run the tractography portion with limited threads?

Thanks again for looking into this!

[Fang-Cheng Yeh]

I did initiate a code revision but it turned out to be very challenging because, in many DSI Studio pipelines, a multi-thread process can initiate another multi-thread, and thus the total amount of running threshold is quite dynamic and much harder to be controlled. The total number of locations that would require modification is also way more than I expected. I am sorry that this feature may not happen in the near future.

Best,

Frank

To view this discussion on the web visit https://groups.google.com/d/msgid/dsi-studio/f17cfa26-b3aa-490e-9b1b-3a43d7655ffbn%40googlegroups.com.

[Matthew Cieslak]

Ah, this is too bad. I will look into seeing if I can limit the cpus seen by singularity, but it looks like any cluster running an os later than centos8 might not be able to do this.

Would there still be too many changes if --thread-count is only added to --action=reg?

[Fang-Cheng Yeh]

The reg part is where the multi-thread is most complicated, and I was not able to complete the change.

One difficulty is that even if the thread count is limited (e.g. say 4), one of the threads can also initiate another set of 4 threads. Lower the thread count may solve the problem, but the efficiency will be greatly reduced.

Another issue is that there are many locations I need to pass the thread count variable. It spans many layers of function calls, and the changes will be quite substantial.

Singularity has a handy solution: https://docs.sylabs.io/guides/main/user-guide/cgroups.html