could not read simulation outputs

Mostafa Zakaria

unread,

May 5, 2018, 3:43:20 PM5/5/18

to HNNSolver

Hey everyone,

We have a problem running simulation on HNN either directly from my machine or from a virtual box environment, after following the tutorial step by step and launching the default simulation.

It keeps returning this message:

Starting simulation. . .

*** An error occurred in MPI_Init_thread

*** on a NULL communicator

*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,

*** and potentially your MPI job)

*** An error occurred in MPI_Init_thread

*** on a NULL communicator

*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,

*** and potentially your MPI job)

[apples-MacBook-Pro-3.local:16815] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

*** An error occurred in MPI_Init_thread

*** on a NULL communicator

*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,

*** and potentially your MPI job)

[apples-MacBook-Pro-3.local:16816] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

*** An error occurred in MPI_Init_thread

*** on a NULL communicator

*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,

*** and potentially your MPI job)

[apples-MacBook-Pro-3.local:16813] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

[apples-MacBook-Pro-3.local:16814] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

--------------------------------------------------------------------------

mpiexec noticed that the job aborted, but has no info as to the process

that caused that situation.

--------------------------------------------------------------------------

WARN: could not read simulation outputs: dict_values(['/Users/mostafazakariakhalil/hnn/data/default/dpl.txt', '/Users/mostafazakariakhalil/hnn/data/default/rawspec.npz', '/Users/mostafazakariakhalil/hnn/data/default/spk.txt', '/Users/mostafazakariakhalil/hnn/data/default/param.txt'])

Thank you,

Mostafa & Sido

sam n

unread,

May 6, 2018, 11:22:16 PM5/6/18

to Mostafa Zakaria, HNNSolver, McDougal, Robert, Daniels, Dylan

Based on your error, it looks like you have a problem with your MPI installation, and since the simulation uses MPI, there is no output produced (as indicated in the later error).

Do you get the same error(s) on Mac and on your Virtualbox with the HNN image (note that the image uses Ubuntu) ?

For your Mac - can you confirm that you followed all the instructions provided here? https://github.com/jonescompneurolab/hnn/blob/master/installer/mac/mac-install-instructions.txt

(without errors reported...)

I'm cc'ing Robert McDougal & Dylan Daniels, in case they have suggestions (Robert developed the Mac installation instructions).

--
You received this message because you are subscribed to the Google Groups "HNNSolver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hnnsolver+unsubscribe@googlegroups.com.
To post to this group, send email to hnns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hnnsolver/c61ddd06-b532-4f85-bc28-1847e9e59e21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Sid Seed

unread,

May 7, 2018, 5:12:04 PM5/7/18

to HNNSolver

Hello everyone,
Allow me to reply instead of Mostafa as we were trying to run hnn.py together and we're facing the same problem.
I'm using a fresh OS install and followed the install instructions given by the link above, everything went smooth and without errors.

After running a the first simulation I was getting

--------------------------------------------------------------------------

mpiexec noticed that the job aborted, but has no info as to the process

that caused that situation.

--------------------------------------------------------------------------

WARN: could not read simulation outputs: dict_values(['/sidseed/hnn/data/default/dpl.txt', '/Users/sidseed/hnn/data/default/rawspec.npz', '/Users/sidseed/hnn/data/default/spk.txt', '/Users/sidseed/hnn/data/default/param.txt'])

I also noticed a message in the simulation log (the simulation log window was disappearing fast so I could only screen capture it).

What I understood was that the openmpi isn't installed in a default location and an LD_LIBRARY_PATH to be added to .bash_profile.

So, I've tried to set LD_LIBRARY_PATH to libmpi in /Applications/anaconda3/lib/,but it didn't work. Below is the result:

SidSeeds-Mac:hnn sidseed$ python hnn.py hnn.cfg

Starting simulation. . .

[SidSeeds-Mac.local:05364] [[54122,0],0] mca_oob_tcp_recv_handler: invalid message type: 15

--------------------------------------------------------------------------

mpiexec noticed that the job aborted, but has no info as to the process

that caused that situation.

--------------------------------------------------------------------------

loaded new dpl file: /Users/sidseed/hnn/data/default/dpl.txt

At the end I've solved the problem by removing mpi4py and openmpi using:

sudo conda remove mpi4py

sudo conda remove openmpi

and installing them following these Instructions:

Install OpenMPI

1.a Download OpenMPI: https://www.open-mpi.org/software/ompi/v1.10/ (openmpi-1.10.7.tar.gz)

1.b Extract and configure

cd ~/Downloads/openmpi-1.10.7
mkdir build
cd build
# You can use --prefix=/usr/local to put it there, but I prefer to put it in /opt/openmpi

../configure --prefix=/opt/openmpi --with-devel-headers --enable-binaries
make
sudo make install

1.c Add the following to your bash profile

export DYLD_LIBRARY_PATH=${DYLD_LIBRARY_PATH}:/opt/openmpi/lib
export PATH=${PATH}:/opt/openmpi/bin

Install mpi4py

git clone https://github.com/mpi4py/mpi4py.git ./mpi4py.git
cd mpi4py.git
python setup.py build --mpicc=/opt/openmpi/bin/mpicc
python setup.py install

Last but not least, was to correct a small mistype I believe in the install instructions (https://github.com/jonescompneurolab/hnn/blob/master/installer/mac/mac-install-instructions.txt):

export NRN_PYLIB="/Applications/anaconda3/bin/../lib//libpython3.6m.dylib" instead of export NRN_PYLIB="/Applications/anaconda/bin/../lib//libpython3.6m.dylib" Since we're using Python 3.6 version (Anaconda distribution).

Now it works, I thought to make this post maybe someone will face similar problems.

I'm still facing a new problem though, whenever I try to load data using "File -> Load data file" I'm getting this error :

Loaded data in /Users/sidseed/TryHnn/hnn/data/MEG_detection_data/yes_trial_S1_ERP_all_avg.txt
RMSE: 42.2957913668
Avg. RMSE:42.3
simdat ERR: could not plotextdat

It maybe from simdat.py from SIMcanvas function, but i'm not sure!

Thank you,
Sid

To unsubscribe from this group and stop receiving emails from it, send an email to hnnsolver+...@googlegroups.com.

sam n

unread,

May 9, 2018, 8:46:41 AM5/9/18

to Sid Seed, HNNSolver, McDougal, Robert

Sid - thanks for the tips/work for your Mac installation. We recently adjusted the Mac install instructions to use conda and this seems to have led to some MPI issues having to MPI version. We plan to make an installation troubleshooting section on the website later on and we'll add your notes for other people facing the same issue.

As far as the data load error, I'm working on fixing some other bugs and will look at that particular problem as soon as I can.

To unsubscribe from this group and stop receiving emails from it, send an email to hnnsolver+unsubscribe@googlegroups.com.

To post to this group, send email to hnns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hnnsolver/fcd5388f-ad74-4657-a012-8808c2d323b6%40googlegroups.com.

McDougal, Robert

unread,

May 9, 2018, 10:09:29 PM5/9/18

to Sam Neymotin, Sid Seed, HNNSolver

Sid:

Can you confirm: your python is installed under /Applications/anaconda3 and not under /anaconda3?

Sam N

unread,

May 9, 2018, 11:11:51 PM5/9/18

to HNNSolver

Sid - I just fixed the problem with plotting external data mentioned below; please do a git pull from the github repo (in your hnn

folder with python files) and try again, and let me know if you still see a problem.

Sid Seed

unread,

May 10, 2018, 5:07:04 AM5/10/18

to HNNSolver

Yes Robert my python is installed under /Applications/anaconda3 as mentioned in the instructions.

Sid:

Sid Seed

unread,

May 10, 2018, 5:47:27 AM5/10/18

to HNNSolver

Sam, It's working right now :) thank you !! I'll it give a few tests, and you let you know in case of any problem.

Mostafa Zakaria

unread,

May 13, 2018, 7:37:14 AM5/13/18

to HNNSolver

Hey Sam,

It's working with me also, Thank you.

Just i cannot load the data on novel screen as in ERP tutorial step-1 (load/view data),

I think this might not be a problem currently while going through tutorials as i can run the simulation then load the experimental data,

However later on when we start to load novel data then craft a simulation it might become a problem,

Check it out please and let me know if i am missing something,

Sorry for posting in weekend but it is working day in Qatar :)

Regards,

Mostafa

sam n

unread,

May 13, 2018, 10:36:33 PM5/13/18

to Mostafa Zakaria, HNNSolver

On Sun, May 13, 2018 at 7:37 AM, Mostafa Zakaria <moustafa....@gmail.com> wrote:

Hey Sam,

It's working with me also, Thank you.

Just i cannot load the data on novel screen as in ERP tutorial step-1 (load/view data),

I fixed the code a few days ago. Can you try pulling the latest version from github? (from a terminal in the directory with the HNN python files, run the following command: git pull).

I think this might not be a problem currently while going through tutorials as i can run the simulation then load the experimental data,

However later on when we start to load novel data then craft a simulation it might become a problem,

Check it out please and let me know if i am missing something,

Sid mentioned he was able to get the newest version to load data properly. Please let me know if that's not the case.

Sorry for posting in weekend but it is working day in Qatar :)

No problem :) ... & thanks for trying out HNN

Regards,

Sam

To unsubscribe from this group and stop receiving emails from it, send an email to hnnsolver+unsubscribe@googlegroups.com.

To post to this group, send email to hnns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/hnnsolver/29ce65c6-8e96-4e0f-a552-aa119e4854a2%40googlegroups.com.

Reply all

Reply to author

Forward