FDS 6.7.0 Too large .sf files

[bje...@ncsu.edu]

Hello,

I am a graduate student who is making the transition from 6.5.1 to 6.7.0 on a new campus cluster. Running the same input file on the new FDS version yields .sf files over 15 GB each, compared to 0.8 GB each on the old FDS version. nframes is kept at the default 1000 in each run. Any thoughts as to what may be causing a significant larger .sf file to appear on the new FDS version?

Thank you.

[Randy McDermott]

We have not noticed this.  Please attach you input file.

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/de99f2cd-188c-475c-81a1-a03fb644f889%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bje...@ncsu.edu]

Attached is the input file. Thank you.

[Glenn Forney]

DT_SLCF is set to 1.0 second.  Since T_END is 10000 s, your slice files will have 10000 frames not 1000.  Are you sure DT_SLCF is the same for both of your runs?

On Tue, Jan 15, 2019 at 2:11 PM <bje...@ncsu.edu> wrote:

--

You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/de99f2cd-188c-475c-81a1-a03fb644f889%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Glenn Forney

[bje...@ncsu.edu]

Attached is the file from the old campus cluster that uses 6.4.0 (not 6.5.1 like I said originally). T_END and DT_SLCF are the same in each file.

[Randy McDermott]

I don't see any difference in the input files.

Can you show us the ls -l for the .sf files and also attach the .out files for each case?

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/efd19348-25e0-4945-9ed0-f05171be97e7%40googlegroups.com.

[bje...@ncsu.edu]

Because the total Fermi folder came to roughly 400 GB, it had to be deleted instantly. Some of the files remain in my Recycle Bin, which is what I can show. Hopefully that is enough.

[Randy McDermott]

Sorry, but nothing jumps out at me as the obvious cause.  The time steps appear to be the same, both finished successfully.  We will have to try to reproduce this problem on our end.  Not sure how long that will take.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/1df06ea4-153d-48dc-b6e9-d9f0e45a8474%40googlegroups.com.

[bje...@ncsu.edu]

No worries, Randy. Thank you to you and Glenn for helping out. I guess it could be an issue with the new cluster, although I do not know what would cause such an issue. I will talk to IT here and see if they can install both versions on the new cluster so I can test this. I am interested to see what you find out from your test, but no rush on that at all. Take care.

[Randy McDermott]

I reduced resolution so I could test on my macbook.  I ran the two attached simplified cases with FDS 6.4.0 and FDS 6.7.1 (latest source) for OSX and see the same slice file size.  So, looks like this is either a Windows issue or something to do with your system.

molly:bjells rmcdermo$ ls -l *.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v640_01.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v640_02.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v640_03.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v640_04.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v671_01.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v671_02.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v671_03.sf
-rw-r--r--  1 rmcdermo  staff  301282 Jan 16 07:33 test_v671_04.sf

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/b910d08a-784f-4f0c-865b-a760328ac44e%40googlegroups.com.

[bje...@ncsu.edu]

Thank you for the quick run, Randy. I will contacting IT here about a possible system issue. Can you clarify what you mean by a Windows issue?

[Randy McDermott]

Just that I ran on a Mac. The Windows exe could possibly have different behavior.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/b9ebaacf-00fc-4a70-83bb-17688ab68328%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Sent from my iPhone

[Glenn Forney]

see if you can reproduce the problem with a case with fewer grid cells and shorter T_END.  ie do you see the problem with a 50 x 25 x 25 case running to 100 seconds (instead of 200 x 100 x 100 running to 10000 seconds)

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/CAC6wax2wkSBP7HpUb1WCeDu94ntamOyE4fM-LEN9UCrurvtRqg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Glenn Forney

[bje...@ncsu.edu]

Thank you Glenn and Randy. Currently, there are no open resources on the new cluster. My current simulation will be done over the weekend, so Tuesday I will try Glenn's suggestion. It is good to know Randy had no issue running on a mac. Hopefully it is not an issue with the Windows .exe file.

[o...@aquacoustics.biz]

Hi folks, I have just completed Randy's reduced resolution and time run on Windows 7 x64 using FDS 6.7.0.  The model ran to completion in  7.4 seconds.  The maximum .sf file size was 417 KB.  The devc.csv file was 1282 KB.  Scaling up, the .sf file sizes of the original model should be about 2.6 GB.  It doesn't seem to be a Windows problem.  Let me know if you'd like me to run the model on a Linux cluster.  t. 

[Glenn Forney]

So are you saying the SF size looks ok? Fyi A 2d slice file size should be approximately the following.

Ntimesteps*nrows*ncols*4

You can verify ntimesteps by;  loading SF in smokeview. Is this formula consistent with what you ate seeing?

--

You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/f0469c48-88d9-438e-bc92-99b6eff30796%40googlegroups.com.

[Tim O'Brien]

Thank you for your email Glenn.

 

There are precisely 100 time steps for the reduced model, DT_SLICE = 1.00 and T_END = 100.

 

Intuitively the file size of the .sf files looks okay for the reduced resolution and run-time model, but I'm not saying they are correct.  However the scaling for time and resolution that I have applied should be about right.

 

Using your formula we have 100 time steps * 25 rows * 50 columns * 4 = 500,000 bytes = 500 KB.  This is close enough to 417 KB so I figure we are in the ball park.

 

Unfortunately the slice file format description in the SmokeView manual is not definitive for file size as it does not describe the Fortran data types or precision for each field of each record, and converting the .sf file to ASCII doesn't resolve the issue.  Windows reports file sizes in 8 bit Bytes in integer n multiples of  10^(3*n), ie Bytes, KB, MB, American GB ...

 

t.  

--
You received this message because you are subscribed to a topic in the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fds-smv/JanksZ_oXIA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fds-smv+u...@googlegroups.com.

To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/CA%2BGOW%2BKueRjvqG%2B-P8%3DiqY8zXCQQeaLsJYYF%2Bmf7Gx0SRPxKHw%40mail.gmail.com.

[bje...@ncsu.edu]

I have contacted IT here at SDSU about putting FDS 6.4 on the new Linux cluster. As of now, I tested Randy's test files and found that 6.4 on our older Linux cluster gives the same file sizes that Randy had, but 6.7.1 on the new cluster gives drastically higher files sizes. Based on the .out file, there were 318 time steps. 318*20*10*4 = 254,400 bytes, so in the ballpark of 301,282 bytes.

$ ls -l

total 36736

-rw-r--r-- 1 bells 1049089  301282 Jan 23 17:54 test_v640_01.sf

-rw-r--r-- 1 bells 1049089  301282 Jan 23 17:54 test_v640_02.sf

-rw-r--r-- 1 bells 1049089  301282 Jan 23 17:54 test_v640_03.sf

-rw-r--r-- 1 bells 1049089  301282 Jan 23 17:54 test_v640_04.sf

-rw-r--r-- 1 bells 1049089 7553964 Jan 23 17:56 test_v671_01.sf

-rw-r--r-- 1 bells 1049089 9381570 Jan 23 17:56 test_v671_02.sf

-rw-r--r-- 1 bells 1049089 9618168 Jan 23 17:56 test_v671_03.sf

-rw-r--r-- 1 bells 1049089 9841898 Jan 23 17:56 test_v671_04.sf

[Glenn Forney]

I am not able to reproduce the problem.  I downloaded the "official" FDS 6.4.0 and FDS 6.7.1 distributions.  I ran a case using each distribution.  The corresponding slice files created by both versions of fds are identical in size .  

If you upload your case I will look to see if there is somethign special about your case  

[bje...@ncsu.edu]

I used the same input files as Randy poster earlier, but I will post them here too. I am also attaching the .err files for each version. Note the very unusual time stepping recorded in the v671 error file compared to the v640 error file. In either case, I do not actually see an error occurring but the files output nonetheless. This seems to be trending towards an issue with the new linux cluster here on campus, considering the mac, windows, and linux versions have now been tested off-campus and all have no issues.

[Glenn Forney]

I'm still getting identical file sizes. 

what command are you using to run your case?  Looking at your .err file it appears that there are 45 instances of fds running when there should be only one (it is a one mesh case).  This could explain the larger file size .  Each instance of fds would be outputting data to the same .sf file - are your .sf files for 6.7 45 times larger than the same .sf run with 6.4 ?

you should use something like

path_to_fds casename.fds

or 

mpiexec -np 1 path_to_fds casename.fds

[bje...@ncsu.edu]

The largest slice files from 6.7.1 are roughly 32 times larger than the slice files from 6.4.0. I am using a script file structured according to what IT has told us to do. The process of calling nodes and processors is different from the cluster where 6.4.0 is located. Both are attached.

[Glenn Forney]

this line from your .pbs file is the problem.  if your input file has 1 mesh you should only use one process on the mpirun line ie -np 1 .  With -np 45 there will be 45 copies of fds running each outputting data to the .sf files resulting in files 45x larger than they should be.  

mpirun -np 45 /nas/app/FDS/FDS6/bin/fds test_v671.fds

--
You received this message because you are subscribed to the Google Groups "FDS and Smokeview Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fds-smv+u...@googlegroups.com.
To post to this group, send email to fds...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fds-smv/1e5f3c16-cdb0-4174-8a60-724a0bcd0d14%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Glenn Forney