Does TVB support parallel processing in python?

[Wayne Wong]

Dear All, 

Hi! I am wondering if TVB supports multiprocessing in python. I tried to create my script using the `multiprocess` module for my TVB simulation. Even though the codes can run without error, the simulation is still running sequentially instead of in parallel. I am not a software engineer so I cannot tell whether I am writing my parallel processing script in a wrong way or TVB doesn't support the parallel simulation at all. Any idea about it? Thank you so much!

Best Regards, 

Yile

[WOODMAN Michael]

hi

if you can share just the part where you use multiprocessing we could check.  Usually it should be something like

def do_one(): pass

with multiprocessing.Pool() as p:

   results = p.map(do_one, ...)

You can also make use of joblib [1] which makes this easier.  As for checking it's working, use your platforms CPU monitor.  Many cores should be used.

cheers,

Marmaduke

[1] https://joblib.readthedocs.io/en/latest/parallel.html

---

From: tvb-...@googlegroups.com <tvb-...@googlegroups.com> on behalf of Wayne Wong <ylww...@gmail.com>
Sent: Thursday, January 20, 2022 7:36:21 AM
To: TVB Users
Subject: [SUSPECTED SPAM] [TVB] Does TVB support parallel processing in python?

 

--
You received this message because you are subscribed to the Google Groups "TVB Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tvb-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tvb-users/0abe1075-cea2-4a86-804f-0f959dac3478n%40googlegroups.com.

[Wayne Wong]

Thanks for your reply! Yes sure the multiprocess part is here: 

https://github.com/yilewang/TVB_workflow/blob/master/G_generator/lfp_sim.py 

In my simulation function (https://github.com/yilewang/TVB_workflow/blob/master/functions/tvb_sim.py), I have three input arguments, caseid, case's group info, and global coupling. The for loop will iterate caseid and the global coupling will be updated too. I tried to use the `joblib.Parallel` you suggested but it is still running my simulation one by one sequentially, with only one core working, and it is the same as when I used `pool.apply()` before. 

Thanks so much for your time!

Best Regards, 

Yile 

[Wayne Wong]

Dear Marmaduke, 

I successfully ran my simulation in parallel now! The problem is coming from the way I wrote my parallel computing script, not TVB at all. I rewrote my codes in a different way and now it is perfectly working! Sorry for your time and inconvenience!

Best Regards, 

Yile

[WOODMAN Michael]

hi

Thanks for the follow up, good to hear,

cheers

Marmaduke

---

From: tvb-...@googlegroups.com <tvb-...@googlegroups.com> on behalf of Wayne Wong <ylww...@gmail.com>

Sent: Friday, January 21, 2022 6:11:48 AM
To: TVB Users
Subject: Re: [SUSPECTED SPAM] [TVB] Does TVB support parallel processing in python?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/tvb-users/71bd990a-ce4a-4e74-9208-3fda03a992a4n%40googlegroups.com.

[Yat-Lok Wong]

Hi~ I just have a new question about the parallel processing in TVB. The situation is that, now I need to do a 10 min simulation using Stefanescu and Jirsa 3D model in the HPC system (which is using SLURM to submit the job). When I required one node and one thread to run my 10 mins simulation, I found that it cost me more than 48 hours to finish the job for a single simulation. I am wondering if there is a way to run a long simulation using multiple threads or nodes in HPC. Is there any documentation or tutorial about it?

Thank you so much!!

Best Regards, 

Yile

[WOODMAN Michael]

hi,

This isn't currently the case though we are working towards this. Unfortunately this model is also the slowest in TVB. If you are willing to spend some time on it, we could help you (a) speed up the model itself and (b) run with multiple threads.

That said, if you are doing a large parameter sweep, you can run one job per core via Slurm.

cheers,
Marmaduke

[WOODMAN Michael]

hi again Yile,

On 1 Apr 2022, at 08:55, WOODMAN Michael <marmaduk...@univ-amu.fr> wrote:

This isn't currently the case though we are working towards this.  Unfortunately this model is also the slowest in TVB.  If you are willing to spend some time on it, we could help you (a) speed up the model itself and (b) run with multiple threads.  

I filed two issues on GitHub for these topics, 

(a) https://github.com/the-virtual-brain/tvb-root/issues/544

(b) https://github.com/the-virtual-brain/tvb-root/issues/543

if you would be interested, we can help you with those.  

cheers,

Marmaduke

[Yat-Lok Wong]

Hi Marmaduke,

Thank you for your reply and offering help! I can share the link of my simulation script and SLURM script here so that you can take a look about the simulation I am doing. I didn't do any parameter sweep in my script, and the only computational-intensive part of my simulation is that it has to last 10 minutes.

My another question is that, if I am not doing a large parameter sweep, is it still possible to still parallelize my single long simulation (using multiple cores to run one job, then concatenate all the results from multiple cores as the final output) in HPC?

Thank you so much! Looking forward to your reply~

Best Regards, 

Yile

[WOODMAN Michael]

hi,

I looked at the scripts you shared, and then seem fine to me. 

On 11 Apr 2022, at 19:26, Yat-Lok Wong <ylww...@gmail.com> wrote:

My another question is that, if I am not doing a large parameter sweep, is it still possible to still parallelize my single long simulation (using multiple cores to run one job, then concatenate all the results from multiple cores as the final output) in HPC?

No, this will not really work unless you have a lot of nodes in your connectivity.  

One thing you can try is to raise the step size (dt), run a short simulation (1e3 is enough), look at the raw data and check that the time series remain the same.  When you see that they start to change, keep the dt where the time series were still the same.  It may be that you can take larger steps, resulting in a faster simulation.  This is also connectome dependent, something to keep in mind, so check across your datasets as well.

cheers,

Marmaduke

[Yat-Lok Wong]

Dear Marmaduke, 

Thank you for your reply! I see the point here.. My connectome only has 16 nodes so probably it wont help the speed much. But I really appreciate the thing you mention about changing the dt. I will try that later~ 

Thank you again!

Best Regards, 

Yile