Twice the cores?

[Michael]

I noticed that my pipelines tend to use twice the number NCORES of cores i set with the MultiProc plugin. 

I mainly used a series of registration with FSL tools and noticed that the number of tasks being processed (as indicated by the number of *_unfinished.json files) corresponds to NCORES although actually 2*NCORES cores are being used. Would someone have an explanation? When i run a task on the command line (with exactly the same command as generated by NiPype) it only uses 1 core.

[Satrajit Ghosh]

hi michael,

since this tends to depend on the underlying interface, could you please provide some details on which processes? afni, for example, uses openmp and by default will use all cores available. we should add a flag in afni to control the number of threads, much like we do for ants.

try setting OMP_NUM_THREADS to 1 in the environment and see if it still does the 2 core thing.

cheers,

satra

On Mon, Oct 21, 2013 at 10:35 PM, Michael <mdayan....@gmail.com> wrote:

I noticed that my pipelines tend to use twice the number NCORES of cores i set with the MultiProc plugin. 

I mainly used a series of registration with FSL tools and noticed that the number of tasks being processed (as indicated by the number of *_unfinished.json files) corresponds to NCORES although actually 2*NCORES cores are being used. Would someone have an explanation? When i run a task on the command line (with exactly the same command as generated by NiPype) it only uses 1 core.

--
 
---
You received this message because you are subscribed to the Google Groups "NiPy Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nipy-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Michael]

Hi Satra, right now this is with FSL flirt. I'll test the OMP_NUM_THREADS variable and report here, thanks for this tip.

[Satrajit Ghosh]

flirt doesn't use more than one core, so i don't know what's happening there.

cheers,

satra

[Michael]

htop indicates two different PIDs (each using 100% core) for the exact same flirt command. Not sure what's going either (when running the command on a terminal only one core is used)

--
 
---
You received this message because you are subscribed to a topic in the Google Groups "NiPy Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nipy-user/CPjld7rIsMk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nipy-user+...@googlegroups.com.

[Chris Filo Gorgolewski]

Is the Flirt in mapnode? Does you CPU support hyperthreading?

[Michael]

It is doing this even in non-map node but i believe this has to do with the number of threads as you suggested. Not only the CPU supports hyperthreading but the output of cat /proc/cpuinfo give a number of cores which is twice the number of threads! (itself twice the number of physical cores...). So it seems that for some reason each single active thread is duplicated in the output of top/htop. 

[Michael Waskom]

I can replicate this issue, currently with the fsl.FILMGLS. I end up seeing twice as many processes as I asked for, with duplicate jobs. I do not see similar behavior for the IPython or Linear plugins.

I'm currently using this commit of master, it appears.

The problem seems to happen when using both Canopy and Anaconda.

Will look into it more.

[Michael Waskom]

And just for the record, doing a git pull does not fix the issue.