Hardware for FDS calculation of 5 million cells at least
475 views
Skip to first unread message
Roger
Feb 25, 2011, 2:46:13 AM2/25/11
to FDS and Smokeview Discussions
We wanna build a work station for fire dynamic simulations(to
calculate at least 5million cells,large spaces or some special mesh
needed for a thin gas layer on the floor of a space with very fine
cells which each of them is about 1 cubic mm ). Does anybody have any
experiences of hardware requirements,what problems might have?
Hardware we plan to build:
mother board: 4 CPU sockets, 32 RAM slots etc.
CPU: 2 CPU with 8 cores of each (total 16 cores)
RAM: 4*4 GiB
Harddisk: 1 TB
OS: Haven't desided which version to be utilized yet, tend to install
64bit win7 or 64bit Linux, it depends.
P.S.
1. We've never used 64bit OS before, see if you guys have any
constructive suggestions.
2. Is there any unexpected error or crash of 'openmp' might occur
during calculation in 64bit OS?
All regards!
Roger
calculate at least 5million cells,large spaces or some special mesh
needed for a thin gas layer on the floor of a space with very fine
cells which each of them is about 1 cubic mm ). Does anybody have any
experiences of hardware requirements,what problems might have?
Hardware we plan to build:
mother board: 4 CPU sockets, 32 RAM slots etc.
CPU: 2 CPU with 8 cores of each (total 16 cores)
RAM: 4*4 GiB
Harddisk: 1 TB
OS: Haven't desided which version to be utilized yet, tend to install
64bit win7 or 64bit Linux, it depends.
P.S.
1. We've never used 64bit OS before, see if you guys have any
constructive suggestions.
2. Is there any unexpected error or crash of 'openmp' might occur
during calculation in 64bit OS?
All regards!
Roger
Kevin
Feb 25, 2011, 8:08:26 AM2/25/11
to FDS and Smokeview Discussions
If you are buying new hardware and/or software, it would not make
sense to buy a 32 bit OS. Even Windows 7 is 64 bit. Mac OSX and most
linux versions have gone to 64 bit some time ago. If you want to do
serious computing, I suggest a modern linux cluster. We just purchased
a rack with 32 nodes, 8 cores per node, for about US$100 k.
To truly exploit OpenMP, you must have 64 bit OS. Otherwise, your
memory limit (about 2 GB) will make the OpenMP useless.
sense to buy a 32 bit OS. Even Windows 7 is 64 bit. Mac OSX and most
linux versions have gone to 64 bit some time ago. If you want to do
serious computing, I suggest a modern linux cluster. We just purchased
a rack with 32 nodes, 8 cores per node, for about US$100 k.
To truly exploit OpenMP, you must have 64 bit OS. Otherwise, your
memory limit (about 2 GB) will make the OpenMP useless.
rmcdermo
Feb 26, 2011, 2:33:56 PM2/26/11
to fds...@googlegroups.com
Maybe Glenn can send the specs for our new linux cluster. Last week I ran a 9.4 million cell case, linux 64 bit. It took about 24 hours on 128 cores.
Glenn Forney
Feb 26, 2011, 3:17:13 PM2/26/11
to FDS and Smokeview Discussions
If I'm reading your specs right you are getting a computer with 16
cores and 16GB of memory, ie 1GB per core. I would get at least 2GB
per core and suggest that if you want to run really big cases you
should get 4GB per core. (at least keep your options open, ie buy
memory now that you won't have to "throw away" if you decide later to
get more memory). Disks are cheap, I would get at least another
disk.
We have more experience with Linux for running mpi versions of FDS -
so that is what we would suggest.
Definitely get a 64 bit OS. You won't be able to use more than 4GB of
your memory otherwise.
cores and 16GB of memory, ie 1GB per core. I would get at least 2GB
per core and suggest that if you want to run really big cases you
should get 4GB per core. (at least keep your options open, ie buy
memory now that you won't have to "throw away" if you decide later to
get more memory). Disks are cheap, I would get at least another
disk.
We have more experience with Linux for running mpi versions of FDS -
so that is what we would suggest.
Definitely get a 64 bit OS. You won't be able to use more than 4GB of
your memory otherwise.
Roger
Feb 27, 2011, 9:22:56 PM2/27/11
to FDS and Smokeview Discussions
I'm a rookie,and one of my friend who's good at programming helps me
to deal with the mpi in linux,and he told me that he found a better
way to do mp work. I haven't met him yet since last week i'll meet him
late this week and see how far he made for it.
Anyway,now we have 4 pc with intel dual-core e5300 2.6GHZ,2G memory,
320G harddisk of each. We installed multi-boot system,winxp 32bit &
ubuntu 32bit on these pc we already have,going to run some case before
we get the new.
Glenn,in your reply,what do you mean "You won't be able to use more
than 4GB of your memory"? 4GB for a single core?
to deal with the mpi in linux,and he told me that he found a better
way to do mp work. I haven't met him yet since last week i'll meet him
late this week and see how far he made for it.
Anyway,now we have 4 pc with intel dual-core e5300 2.6GHZ,2G memory,
320G harddisk of each. We installed multi-boot system,winxp 32bit &
ubuntu 32bit on these pc we already have,going to run some case before
we get the new.
Glenn,in your reply,what do you mean "You won't be able to use more
than 4GB of your memory"? 4GB for a single core?
Glenn Forney
Feb 27, 2011, 10:06:18 PM2/27/11
to FDS and Smokeview Discussions
A 32 bit OS can address no more than 2^32 bytes which is 4GB.
Uusually some of this memory is reserved for the OS itself and not
available to the user. I believe with 32 bit Windows XP the max
memory available to the user is 3GB. I think the limit is much closer
to 4GB with Linux.
If I understood your comment correctly, you have 2GB of memory on each
of your 4 PCs. So, you could add 2 more GB's to each computer without
changing the OS. If you want to have more than 4GB you'll need to
switch to a 64 bit OS.
Uusually some of this memory is reserved for the OS itself and not
available to the user. I believe with 32 bit Windows XP the max
memory available to the user is 3GB. I think the limit is much closer
to 4GB with Linux.
If I understood your comment correctly, you have 2GB of memory on each
of your 4 PCs. So, you could add 2 more GB's to each computer without
changing the OS. If you want to have more than 4GB you'll need to
switch to a 64 bit OS.
wyldckat
Feb 28, 2011, 6:05:13 AM2/28/11
to FDS and Smokeview Discussions
Greetings to all!
Just to make a little note: it will depend on the OS, because for
example, a 32bit application on Windows XP can only use 2GB max, even
if the OS uses PAE to access more than 2GB of RAM.
Some references:
- http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx
- http://en.wikipedia.org/wiki/Physical_Address_Extension
Some more investigation and info on Linux OS'es should arise.
Best regards,
Bruno
Just to make a little note: it will depend on the OS, because for
example, a 32bit application on Windows XP can only use 2GB max, even
if the OS uses PAE to access more than 2GB of RAM.
Some references:
- http://msdn.microsoft.com/en-us/library/aa366778%28v=vs.85%29.aspx
- http://en.wikipedia.org/wiki/Physical_Address_Extension
Some more investigation and info on Linux OS'es should arise.
Best regards,
Bruno
DavidShep
Feb 28, 2011, 7:34:41 AM2/28/11
to FDS and Smokeview Discussions
A question for Randy, the cluster didn't barf on a 9.4 million cell
case which means that the cluster could handle the model. Did you use
all 128 nodes, and how much run time did you get in 24 hours?
case which means that the cluster could handle the model. Did you use
all 128 nodes, and how much run time did you get in 24 hours?
rmcdermo
Mar 1, 2011, 8:07:22 AM3/1/11
to fds...@googlegroups.com
Dave,
This was the Sandia 1 m methane pool fire case in our validation suite. It is 20 sec of real time for 8 mm resolution. Each mesh was 48x48x32. Much of the cost was output, next was communication.
R
Paul Hart
Mar 1, 2011, 3:08:19 PM3/1/11
to FDS and Smokeview Discussions
Does NIST use any type of cluster management system for the linux
cluster?
cluster?
Kevin
Mar 1, 2011, 3:32:18 PM3/1/11
to FDS and Smokeview Discussions
We just purchased a 32 node, 256 core cluster. I believe we use PBS,
but Glenn knows more. He has simplfied the scripting process.
> > R- Hide quoted text -
>
> - Show quoted text -
but Glenn knows more. He has simplfied the scripting process.
>
> - Show quoted text -
Roger
Mar 1, 2011, 9:16:44 PM3/1/11
to FDS and Smokeview Discussions
hi everyone,
Can you guys gimme a resolution of linux mpi configuration? it would
be great if you write an detailed instruction step by step,appreciate
for that
ps. i'm an linux idiot. :(
Can you guys gimme a resolution of linux mpi configuration? it would
be great if you write an detailed instruction step by step,appreciate
for that
ps. i'm an linux idiot. :(
Kevin
Mar 2, 2011, 7:57:11 AM3/2/11
to FDS and Smokeview Discussions
If you really are a linux idiot, you need to find someone to help you.
If you are just being modest, I suggest you take a look at the notes
that Paul Hart posted a few weeks ago, keeping in mind that his
version of linux and MPI may be different than yours.
If you are just being modest, I suggest you take a look at the notes
that Paul Hart posted a few weeks ago, keeping in mind that his
version of linux and MPI may be different than yours.
Paul Hart
Mar 2, 2011, 1:08:25 PM3/2/11
to FDS and Smokeview Discussions
If you are new to linux, it will be a labor of love, and hate
sometimes. But, if you use the Fedora Core distros which I use, they
have come a long way since FC4. The info Kevin refers to is basically
the instructions I use to reinstall. From experience I have learned
not to use the latest distro being offered. In the distant past, I
experienced too many bugs with them.
> > ps. i'm an linux idiot. :(- Hide quoted text -
sometimes. But, if you use the Fedora Core distros which I use, they
have come a long way since FC4. The info Kevin refers to is basically
the instructions I use to reinstall. From experience I have learned
not to use the latest distro being offered. In the distant past, I
experienced too many bugs with them.
Laurie Odgers
Mar 2, 2011, 7:19:49 PM3/2/11
to fds...@googlegroups.com
It might be a good idea to use CentOS (free version of Redhat Enterprise
Linux) instead - http://www.centos.org . its less bleeding edge than
Fedora Core - FC is used as a test bed for RHEL.
Linux) instead - http://www.centos.org . its less bleeding edge than
Fedora Core - FC is used as a test bed for RHEL.
CentOS are still working on compiling version 6, but it should be out
(hopefully) in the coming weeks.
Roger
Mar 3, 2011, 1:53:09 AM3/3/11
to FDS and Smokeview Discussions
My friend found an instruction on following weblink ,http://
techtinkering.com/articles/?id=32.
Now i'm trying to set mpi by following this overall guide. I notice
that this instruction's based on Debian,but my friend told me the
operations in Ubuntu are quite similar to those in Debian. Has anyone
tested it before and willing to test it? Thanks a lot.
techtinkering.com/articles/?id=32.
Now i'm trying to set mpi by following this overall guide. I notice
that this instruction's based on Debian,but my friend told me the
operations in Ubuntu are quite similar to those in Debian. Has anyone
tested it before and willing to test it? Thanks a lot.
Emanuele Gissi
Mar 3, 2011, 2:40:46 AM3/3/11
to FDS and Smokeview Discussions
Setting up FDS and MPI on Ubuntu Linux is clearly described in the
following coursebook:
"An introduction to fire simulation with FDS and Smokeview".
Disclaimer: I wrote it.
It's open and free (Creative Commons licensed).
You can download it from my home site: http://emanuelegissi.eu
Emanuele Gissi
following coursebook:
"An introduction to fire simulation with FDS and Smokeview".
Disclaimer: I wrote it.
It's open and free (Creative Commons licensed).
You can download it from my home site: http://emanuelegissi.eu
Emanuele Gissi
Reply all
Reply to author
Forward
0 new messages