problems implementing NetPyNE batch simulation

136 views
Skip to first unread message

Irene Bettini

unread,
Dec 9, 2023, 10:46:00 AM12/9/23
to NetPyNE Q&A forum

Hello everyone,

I am writing to ask you for help in implementing my code. To optimize the parameters of my network built in NetPyNE, I started by modifying the netpyne_batch_evol code, inserting my network and adapting the fitness function. The problem I encountered is that although I have minimized the population size for each generation, the program is unable to move effectively to the second generation. Wanting to investigate whether the block resided in my part of the code or in the batch in general, I tried to run the original code (netpyne_batch_evol, downloaded directly from GitHub) and the type of error that occurs is the same: the program reaches the number maximum number of iterations without being able to perform the optimization.

The following screenshot is the iteration output.

 1.jpg

I hope you can help me solve this problem, any assistance you could give me in this matter would be greatly appreciated. Thank you!

Irene Bettini

Salvador Dura

unread,
Dec 18, 2023, 11:11:51 PM12/18/23
to NetPyNE Q&A forum
Irene, sorry for the delay, we are looking into this and will get back to you asap.

J C

unread,
Jan 11, 2024, 10:58:38 PM1/11/24
to NetPyNE Q&A forum
Hello,

Sorry, this is a tricky problem.
I'm looking currently at the tut_batch_evol here: here which I think the netpyne_batch_evol.ipynb notebook turned into

TL:DR. The short answer is that, if you are running tut_batch_evol from the latest NetPyNE development build there should be a directory called `simple` being created in the same directory the batch script is being run in, and depending on the contents of that folder someone may be able to pinpoint what exactly is causing this issue.

Longer answer:
Unfortunately, it seems to run on my current setup so on this end I can't reproduce this error...
```
(dev) [jchen@node08 tut_batch_evol]$ mpiexec -np 4 nrniv -mpi batchRun.py
...
Waiting for jobs from generation 4/4 ...
  M rate=11.9 fit=12; S rate=10.0 fit=5
  Candidate 0 fitness = 8.5
  M rate=34.6 fit=12; S rate=10.0 fit=1000
  Candidate 1 fitness = 506.2
  M rate=14.8 fit=12; S rate=10.0 fit=1
  Candidate 2 fitness = 6.7
  M rate=10.2 fit=12; S rate=10.0 fit=11
  Candidate 3 fitness = 11.8
  M rate=13.2 fit=12; S rate=10.0 fit=2
  Candidate 4 fitness = 7.4
  M rate=26.5 fit=12; S rate=10.0 fit=319
  Candidate 5 fitness = 165.9
completed: 6
--------------------------------------------------------------------------------
  Completed a generation  
--------------------------------------------------------------------------------
Generation Evaluation      Worst       Best     Median    Average    Std Dev
---------- ---------- ---------- ---------- ---------- ---------- ----------
         4         30 165.929144 6.71056604 7.98638681 34.5260874 58.7907408

Best Solution:
[0.054396752928195916, 0.07467499759418919, 10.257649870402997] : 6.7105660463246295
--------------------------------------------------------------------------------
   Completed evolutionary algorithm parameter optimization  
--------------------------------------------------------------------------------
```

As an additional note, there are various ways this can fail without providing much useful input.
As far as I know, the current Batch toolkit works by:

1. generating a directory/files containing the necessary code to run an individual simulation
2. creating a new thread to run the simulation (which relies on subprocess--and, additionally--in the case of `mpi_bulletin`--NEURON's parallel context)
3. monitoring the filesystem for completion, specifically through the creation of a result file (for every method except `mpi_bulletin`)
4. collecting relevant data

Unfortunately, the printout to terminal is hindered by the fact that these individual simulations are running in their own threads.
some of the information on these simulations (error messages and the status of the runs) are found in the directory created in step 1

in this case, the `batchRun.py` specifies this directory as `simple`
```
   b.batchLabel = 'simple'
   b.saveFoler = './'+b.batchLabel
```
the files generated should look something like this before the simulations are run if step 1 is working.

```
(dev) [jchen@node08 tut_batch_evol]$ tree simple | head -n7
simple
├── batchScript.py
├── gen_0
│   ├── gen_0_cand_0_cfg.json 
│   ├── gen_0_cand_1_cfg.json
│   ├── gen_0_cand_2_cfg.json
│   ├── gen_0_cand_3_cfg.json
│   ├── gen_0_cand_4_cfg.json
...

(dev) [jchen@node08 tut_batch_evol]$ tree simple | tail -n4
├── _seed.seed
├── simple_batch.json
├── simple_batchScript.py
└── simple_netParams.py
```

If not, then something occurred during directory creation

Otherwise, if the thread is successfully created in Step 2, output from those simulations (both errors, and run printouts) will also be written to files within the `simple` directory, the resulting data file should be created

```
(dev) [jchen@node08 tut_batch_evol]$ tree simple | head -n7
simple
├── batchScript.py
├── gen_0
│   ├── gen_0_cand_0_cfg.json 
│   ├── gen_0_cand_0_data.json <- generated when the simulation is finished
│   ├── gen_0_cand_0.err <- generated while the simulation is running
│   ├── gen_0_cand_0.run <- generated while the simulation is running
```

In this example there is a folder called `simple` being created with files for the different simulations including `.err` and `.run` file
if these .err and .run files are generated, please see if they may indicate some issue. If not, then something occurred with thread allocation, in the case of mpi_bulletin, this may be due to NEURON parallel context or subprocess (other batch configurations only use subprocess).

The `_data.json` file indicates the simulation completed, and if this file exists, then somehow the collection of relevant data is not being triggered.

Irene Bettini

unread,
Jan 24, 2024, 8:25:20 AM1/24/24
to NetPyNE Q&A forum

Hello,

Unfortunately, I haven't been able to solve the problem despite your comprehensive help.

Regrettably, I have not yet been able to resolve the issue despite your thorough assistance. I've carefully examined the problem from the beginning, re-downloading the code from the provided link and executing it. The generations run successfully, generating all the specified files, including both the .run and .err files (though the latter only contains that kind of messages).

 1.png

 

 2.png

However, the .data files are not being saved, except in the singular instance where the iteration concludes successfully (occurs only once, consistently in the first generation).

I've also conducted a test to verify that the issue doesn't lie in the mpi_bulletin execution, as indicated on the website https://neuron.yale.edu/neuron/static/docs/neuronpython/ballandstick5.html, and it passed successfully.

May the issue stem from the fact that the batch function is not supported in the Visual Studio simulation (where I have previously programmed successfully)?

Is there any profiling procedure on the code that can be carried on to identify the issue? Can other implemented examples help in identifying it?

 Thanks for any further help you may provide.

Best regards,

Irene

J C

unread,
Jan 24, 2024, 5:37:29 PM1/24/24
to Irene Bettini, NetPyNE Q&A forum
Thanks for the followup Irene,

I'm not sure what is happening, but it seems like some resource is not being deallocated properly once the first run is completed, or somehow more resources were requested than available to the shell. I'm not sure whether this is due to Visual Studio simulation, although I believe if it is being run via the terminal plugin it has all the resources available on the computer (not sure if it is being executed directly by the IDE)

Can you send me the terminal command you used to run the code? In the meantime I will take another look at this particular script.

--
You received this message because you are subscribed to a topic in the Google Groups "NetPyNE Q&A forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/netpyne-forum/DAImjzQzENA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to netpyne-foru...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/netpyne-forum/4b59ad16-b95b-4311-92b2-beadb482de88n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages