MPI error win64 - FDS 6.2

[steve...@gmail.com]

Hi everyone, I am experiencing a wierd issue with mpiexec and I was wondering if someone has faced it before.

I am trying to run an FDS file with mpiexec, 12 processes and the command line is returning this error:

mpiexec -hosts 1 localhost fds test.fds

[mpiexec@myPC] ..\hydra\utils\sock\sock.c (157): bind error (Unknown error)

[mpiexec@myPC] ..\hydra\utils\sock\sock.c (1065): unable to listen on port

[mpiexec@myPC] ..\hydra\pm\pmiserv\pmiserv_pmci.c (354): unable to create PMI port

[mpiexec@myPC] ..\hydra\ui\mpich\mpiexec.c (1035): process manager returned error launching processes

It is worth noting that the same file run with mpiexec on the same machine last week.

Unfortuantely it not my machine so I am not 100% sure what happened in between in terms of updates or other things...

I have tried to log off, restart, unistall and reinstall FDS with no luck.
Any suggestion?

Thanks

Ste

[steve...@gmail.com]

Sorry,

I realized there is a typo in the line I am using to launch FDS

The command giving me problems is:

mpiexec -hosts 1 localhost 12 fds test.fds

Thanks

Ste

[陳佑任]

What  is your MPICH 2 version ?

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/52338e08-da6b-46f0-8b63-dd1963fe0b94%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[steve...@gmail.com]

That's the one:

mpiexec --version

Intel(R) MPI Library for Windows* OS, Version 5.0 Update 3 Build 20150128

Copyright (C) 2003-2015, Intel Corporation. All rights reserved.

I have reinstalled the lasted FDS release for win64 so it should be the one released with it 

Thanks

Ste

[Coraline]

Hi all,

Any update on this? I have the exact same problem.

Thanks and regards, 

Coraline

On Tuesday, May 19, 2015 at 10:04:11 AM UTC+1, steve...@gmail.com wrote:

[Coraline]

FYI it works fine with FDS 6.1.2

[steve...@gmail.com]

Thanks Coraline,

it works for me as well

Is it a bug??

Thanks

ste

[Kevin]

All -- FDS 6.2.0 uses a different MPI scheduler than past versions. We use Intel's MPI for Windows, and we now use the scheduler that Intel recommends. I suggest that you uninstall FDS from your machine and do a clean install of FDS 6.2.0 so that you do not have two competing schedulers. Let me know if that helps.

[Coraline Faurie]

Hi Kevin,

Thanks for your message.

I've red about the MPI issue somewhere else on the forum, so I already tried that, it didn't work. I observed the problem above on two different computers, one of them was new and FDS 6.2 was the only version of FDS ever installed!

I'll give it another go tomorrow, we never know.

Thanks,

Coraline

--
You received this message because you are subscribed to a topic in the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fds-smv/4xJqYiFjkSQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fds-smv+u...@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/3e5f6db1-d83e-421d-a773-5136322a6e7c%40googlegroups.com.

[steve...@gmail.com]

Hi Kevin,

same for me!

I have unistalled FDS and reinstalled it and the problem is still there

Thanks

Ste

[Kevin]

I do not understand why mpiexec works on some machines and not others. One guess is that we select a certain range of ports for MPI communication, and it is possible that these ports are already in use.

[Adam Pakonski]

I experience the same issue. Two weeks ago i installed 6.2.0 and it worked fine for a few days and then i received the same error (see attached). I reinstalled 6.2.0 and tried too see if it was a problem on my network and I couldn't get it to work so went back to 6.1.2. A few days later i thought i would try again and i reinstalled 6.2.0 and for some unknown reason it worked. That was last week, but now last night it's displayed the same error, i've reinstalled everything again and it doesn't want to work. I cannot explain why.

[Kevin]

When FDS is installed on a Windows computer, it assigns the port numbers in the range 8670:8690 for the MPI communication. You can see this if you look at the environment variable MPIEXEC_PORT_RANGE. I am concerned that this erratic behavior of the MPI version might be caused by other applications using these same ports. I do not know enough about how the ports are assigned to know for sure.

[Coraline Faurie]

It might be a silly idea, but do you think the problem could be coming from pyrosim? I normally use it to generate fds files which I run from the command prompt. I know pyrosim is not up to date with fds 2.0, and that the installation of pyrosim comes with an internal install of FDS. Maybe there is some kind of competition between the two?

--
You received this message because you are subscribed to a topic in the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fds-smv/4xJqYiFjkSQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/3d382124-d323-42c4-83cc-8e1f7c6447c4%40googlegroups.com.

[Kevin]

There are no silly ideas when it comes to Windows. I suspect there is some conflict with another program using those ports, but I do not know which. Check which versions of FDS you have

where fds

To unsubscribe from this group and all its topics, send an email to fds-smv+unsubscribe@googlegroups.com.

[Glenn Forney]

assuming you are able to reinstall pyrosim after an uninstall, try 

1. uninstalling both pyrosim and fds. type fds at a command line to make sure there are no other copies of fds around.

2. reboot

3. install fds (not pyrosim)

then see if fds works.

To unsubscribe from this group and all its topics, send an email to fds-smv+u...@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/3d382124-d323-42c4-83cc-8e1f7c6447c4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/eabccd63-5b50-4f2c-98cf-6fc70227404b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Glenn Forney

[Coraline]

Pyrosim seems to be the issue for me.

FDS 6.2 works perfectly without (MPI, OpenMP and combination of both). Once installed, I obtain the same error again. 

On Tuesday, May 19, 2015 at 10:04:11 AM UTC+1, steve...@gmail.com wrote:

[Kevin]

Wow, that's interesting. I wonder if PyroSim is using a different scheduler. In any case, can you use the MPI version that comes with PyroSim?

[Glenn Forney]

Are you using any other programs that use mpi such as pyrosim?

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/6407e004-faa5-42d3-a157-643d5f685e83%40googlegroups.com.

[Bryan Klein]

Hello all,

Our main developer responsible for this feature (Charlie) is currently out of the office, but let me see if I can shed some light on the issue from the PyroSim side.

The MPI service that we install is started with the following command ""C:\Program Files\PyroSim 2015\fds\smpd.exe" -port 52700"

It should show up in the list of Local Services as 'PyroSim 2013 MPI x64".

When I run the command 'netstat -ab', I see one TCP listener with the following info that is in the range that Kevin mentioned.

 TCP    0.0.0.0:8678           Klein-TECi:0           LISTENING

For 'smpd.exe' which is the process manager service for MPICH2 applications.

We are working on a new version of PyroSim that supports Hydra and the latest version of FDS, which might resolve this conflict with the SMPD process that we use for MPICH2.

-Bryan Klein

Thunderhead Engineering

[Bryan Klein]

Hello,

Attached is a screenshot of the output of 'netstat -aon | more' and the task manager showing which process belongs to the offending ID number.

-Bryan

[Glenn Forney]

try 

1. opening up a windows shell using the "Run As Administrator" option.  (go to Start Menu>Accessories and right click on "Command Prompt" then select Run As Administrator"

2.  type smpd -stop

3.  now try running fds using mpiexec

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/cc77305e-e127-4542-be9a-340c68c979d2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Glenn Forney

[Christian Kraft]

Hello,

stopping the smpd worked find to solve the problem.

Christian

[Bryan Klein]

The smpd.exe listening on the conflicting port was from an older FDS 6 installation on my computer. The smpd.exe file that PyroSim sets up listens on a port in the 50k range.

Installing PyroSim should not conflict with installations of FDS downloaded from the NIST project.

-Bryan

You received this message because you are subscribed to a topic in the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fds-smv/4xJqYiFjkSQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fds-smv+u...@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/822f5185-f35c-4f11-976f-5ce0c47568ea%40googlegroups.com.

[Coraline]

Hi Bryan, 

Sorry, but in my case Pyrosim is definitely the problem. 

Attached you can see 3 screenshots: first one on the left is with FDS 2.0 alone installed on my computer (mpiexec works fine), then in the middle I've installed Pyrosim 15 (error as described above), and then on the right is a run from pyrosim itself, which works in parallel but uses FDS 1.2. The same happened with 2 different computers, one has ANSYS CFX on it which could be a problem, but the other doesn't have any other CFD software installed. 

Unfortunately, the 'smpd -stop' didn't work for me, "smpd is not recognize as an internal or external comand...". I manually uninstall Pyrosim each time I'm running FDS 2.0 in parallel now.

Regards, Coraline

On Tuesday, May 19, 2015 at 10:04:11 AM UTC+1, steve...@gmail.com wrote:

[Charlie Thornton]

For anyone who needs to uninstall/reinstall PyroSim to make FDS 6.2 work, there is a different workaround that may be a bit quicker. Windows has an editor for services that you can use to disable the offending SMPD service. You can disable the interfering scheduler when you want to run FDS directly, and re-enable it if you decide you want to run the MPI version of FDS via PyroSim.

To disable the SMPD service that PyroSim installs (win7):

1. Run services.msc (I usually type the name in the windows search box, then hit Enter)

2. Scroll down the list of services until you see PyroSim 2013 MPI (or similar).

3. Right-click the service and click Properties

4. Set the Startup type option to: Disabled

5. Click Stop

[Adam Pakonski]

Coraline you are a genius! :) I too have Pyrosim installed and it never came to me to consider the fds install with Pyrosim as i use Pyrosim to generate the fds file which i then run from the command line.

Steps taken:

1) Given the dialogue (greatly appreciated by all) I installed 6.2.0 onto two machines and tried to re-run mpi again with the same earlier error <image 1>

2) The "smpd -stop" command didn't work and I received the same error as Coraline so I opened the task manager and ended the smpd process and the mpi ran! :) <image 2>

3) I restarted the computer and then tried again, same error received again <image 3>

4) Now I tried Charlie's advice which also ended the smpd.exe in the task manager and the mpi ran :) <image 4>

I don't know why but the "netstat -aon" command showed that a process or something (sorry i'm not a computer guy) remained allocated to 0.0.0.0.8679 however I do have other machines in the office currently turned on with 6.1.2 installed so perhaps this is from one of those other machines. I'll try reinstall 6.2.0 on the other five machines and report back later.

Thanks to everyone for the discussion and assistance,

Adam

[Adam Pakonski]

FYI 6.2.0 is now installed on seven windows machines and working well! <see image 5> .. for now at least. 

There is still something on the 0.0.0.0.8679 port which is a concern given Kevin's information that the 8670-8690 range is used by FDS for MPI communication.

[Kevin]

This is all a bit of black magic, and we (the developers) are not experts in this area. Windows was just not designed for this, and it's a minor miracle that we can get MPI running on a Windows domain network. The port numbers are a good example -- it is so random how we choose these ports, and even with tens of thousands of port numbers, inevitably we'll hit a land mine. Hopefully now we'll all move forward with the Hydra communicator rather than smpd. Intel claims that this is a much better way of scheduling, but the details allude me. We have to adapt to the newer technology or we'll eventually fall too far behind.

Thanks for helping us sort out this problem.

[Coraline Faurie]

Yes, Charlie's method to disable the SMPD service did the trick for me. Thanks everyone for your help!

Coraline

--

You received this message because you are subscribed to a topic in the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fds-smv/4xJqYiFjkSQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/9988108a-e6df-416d-aa94-3c2bf1f83648%40googlegroups.com.

[Manuel]

I first had problems running fds Job with mpi and shutting down concurrent MPI Services (SMPD or IBM Platform  MPI) solved these.

But there is another Issue I cannot solve. After launching a fds case with the following command line "mpiexec -env OMP_NUM_THREADS 1 -n 4 fds.exe mpi-test1.fds" the mpiexec process is listening on port 8970 ( looked up with netstat -aon ).

Now while running this job I tried to run the other one "mpiexec -env OMP_NUM_THREADS 1 -n 4 fds.exe mpi-test2.fds". This job won't run. Checking with netstat -aon tells me another process is also listening on port 8970, which is the second call on mpiexec.
Even when job one finished, nothing happens with the second. There is no error message till aborting execution with "ctrl-c". Then the following error message is printed on screen.

Following pictures show the the running job one, the output of netstat -aon and the process monitor. The examplecases mpi-test1.fds and mpi-test2.fds are included in this message.

 

Help on this is welcome. Thanks Manuel

[Kevin]

Someone else also had this problem. You cannot run two MPI jobs under the current Windows configuration. To run multiple jobs, you need a proper job scheduler. On linux machines, most people use TORQUE, along with PBS (Portable Batch System). On windows, there is Windows HPC (High Performance Computing), which is an extension to the operating system that accommodates job scheduling and multiple MPI runs. We chose not to develop a specific FDS release for Windows HPC, deciding to let those users compile and build there own. The Windows release of FDS allows basic MPI functionality, meaning 1 job at a time.