RE: [EXTERNAL] Quick RAVEN question

24 views
Skip to first unread message

Stephen Heagy

unread,
Jan 24, 2023, 4:40:25 PM1/24/23
to Congjian Wang, Jarrett Valeri, inl-rav...@googlegroups.com

Hi Congjian,

Writing to csv files using the OutStream Print option worked great for getting data for individual simulator executions as they completed for normal MultiRun executions.  However, when I tried for a RELAP5-3D ensemble model none of the csv files were written until the end of the execution, at which time they were all written at once.  See sample input excerpts below.  Any advice?

 

Additionally, in older versions of RAVEN there was a csv file that was generated in the lowest level simulation directories (out~<STEP NAME>.csv) but it seems to know longer be generated.  Is there an option to still generate this file?

 

Best regards,

Steve

 

  <Steps verbosity="debug">

    <MultiRun name="Ensemble">

      <Input class="Files" type="">RISA_ss.i</Input>

      <Input class="Files" type="">rellic.bin</Input>

      <Input class="Files" type="">tpfh2o</Input>

      <Input class="Files" type="">tpfhe</Input>

      <Model class="Models" type="EnsembleModel">RELAP_ss_tr</Model>

      <Sampler class="Samplers" type="MonteCarlo">mc1</Sampler>

      <Output class="DataObjects" type="HistorySet">RELAPss_HISTORY</Output>

      <Output class="Databases" type="HDF5">RELAPss</Output>

      <Output class="OutStreams" type="Print">SS_OUT_HIST</Output>

      <Input class="Files" type="">RISA_tr.i</Input>

      <Output class="DataObjects" type="HistorySet">RELAPtr_HISTORY</Output>

      <Output class="Databases" type="HDF5">RELAPtr</Output>

      <Output class="OutStreams" type="Print">TR_OUT_HIST</Output>

    </MultiRun>

</Steps>

 

  <Models verbosity="debug">

    <EnsembleModel name="RELAP_ss_tr" subType="">

      <Model class="Models" type="Code">RELAPss

        <Input class="Files" type="">RISA_ss.i</Input>

        <TargetEvaluation class="DataObjects" type="PointSet">RELAPssTE</TargetEvaluation>

        <Input class="Files" type="">rellic.bin</Input>

        <Input class="Files" type="">tpfh2o</Input>

        <Input class="Files" type="">tpfhe</Input>

        <Output class="DataObjects" type="HistorySet">RELAPss_HISTORY</Output>

        <Output class="Databases" type="HDF5">RELAPss</Output>

        <Output class="OutStreams" type="Print">SS_OUT_HIST</Output>

      </Model>

      <Model class="Models" type="Code">RELAPtr

        <Input class="Files" type="">RISA_tr.i</Input>

        <TargetEvaluation class="DataObjects" type="PointSet">RELAPtrTE</TargetEvaluation>

        <Input class="Files" type="">rellic.bin</Input>

        <Input class="Files" type="">tpfh2o</Input>

        <Input class="Files" type="">tpfhe</Input>

        <Output class="DataObjects" type="HistorySet">RELAPtr_HISTORY</Output>

        <Output class="Databases" type="HDF5">RELAPtr</Output>

        <Output class="OutStreams" type="Print">TR_OUT_HIST</Output>

        <metadataToTransfer alias="sourceID" source="RELAPss">prefix</metadataToTransfer>

      </Model>

    </EnsembleModel>

From: Congjian Wang <Congji...@inl.gov>
Sent: Monday, November 28, 2022 10:21 AM
To: Stephen Heagy <steve...@fpolisolutions.com>
Cc: Jarrett Valeri <jarrett...@fpolisolutions.com>
Subject: Re: [EXTERNAL] Quick RAVEN question

 

Hi Steve,

 

Thanks for the suggestion. I agree with you it will be very useful for RAVEN to leverage the features of HDF5. FYI, the Databases can be also included in the MultiRun Step. Since the HDF5 support SWMR, I think you can directly retrieve the information that you need. In order to do this, you need to include the Databases as Output in the MultiRun Step. For example:

 

 

 

By the way, could you send your future questions to "INL RAVEN Users Group" <inl-rav...@googlegroups.com>? 

 

Best,

 

Congjian 

From: Stephen Heagy <steve...@fpolisolutions.com>
Sent: Monday, November 28, 2022 7:19 AM
To: Congjian Wang <Congji...@inl.gov>
Cc: jarrett...@fpolisolutions.com <jarrett...@fpolisolutions.com>
Subject: RE: [EXTERNAL] Quick RAVEN question

 

Thanks for the response, Congjian.  That seems like a good solution.  Of course we may need to consider cleaning up large amounts of files that are created. 

 

It looks like the HDF5 file format supports Single-Writer Multiple-Reader (SWMR) so you can add data while other processes read from the file (implemented in version 1.10 https://portal.hdfgroup.org/display/HDF5/New+Features+in+HDF5+Release+1.10).  Is this a feature that could be leveraged in RAVEN to essentially use the HDF5 as an outstream?

 

Best regards,

Steve

 

From: Congjian Wang <Congji...@inl.gov>
Sent: Monday, November 21, 2022 5:43 PM
To: Stephen Heagy <steve...@fpolisolutions.com>
Cc: Jarrett Valeri <jarrett...@fpolisolutions.com>
Subject: Re: [EXTERNAL] Quick RAVEN question

 

Hi Steve,

 

I guess you can use OutStream Print option to print the results into csv files for each realization. For example, you can include additional Output in the MultiRun as indicated in the following screenshot to print the collected results into CSV files. These CSV files will keep updated for each completed simulations. Your web application can read these CSV files to get results for each realization. 

 

 

Best,

 

Congjian 

From: Stephen Heagy <steve...@fpolisolutions.com>
Sent: Monday, November 21, 2022 2:49 PM
To: Congjian Wang <Congji...@inl.gov>
Cc: jarrett...@fpolisolutions.com <jarrett...@fpolisolutions.com>
Subject: [EXTERNAL] Quick RAVEN question

 

Hi Congjian,

 

Just have a quick question that came from a customer request and want to run it by you.  I’ll be out for the rest of the week, so there is no big urgency.

 

The customer wanted to be able to use our web application and stream results for individual simulator executions as the complete.  The sample use case runs 500 perturbations that run the chain of 1 RELAP simulation (for the system/plant response) and 9 COBRA calculations (for the core response).  So they want to be able to see output parameters for each of the COBRA runs as they complete.  Since, we get the data from the HDF5 which is performed in an I/O step after all simulator executions in Sequence/Simulation step are completed, this seems to be a challenge.  Do you have any recommendations or is there any RAVEN functionality that supports this that I’m not aware of?

 

A couple workarounds I can think of are:

  1. Instead of calling simulator (RELAP5-3D, COBRA, etc.) directly for each realization in the Models->Code->executable node, a wrapper script could be entered here that includes the execution of the simulator and an external post-processor that reads the results from the output files and brings them back to the web application.
  2. Have the web application run a separate service to check the results of the RAVEN simulation.  This service would be continuously run and would march down into the simulation directory hierarchy to see when simulations have completed and post-process their output files to get results and bring them back to the web application.

 

Thanks for your help!

Steve

 

Congjian Wang

unread,
Jan 24, 2023, 8:55:44 PM1/24/23
to Stephen Heagy, jarrett...@fpolisolutions.com, inl-rav...@googlegroups.com
Hi Stephen,

I think I need more information in order to diagnose the problem. FYI, we have several regression tests to test the features that you mentioned. Regarding your first question, you can take a look at the test "raven/tests/framework/ensembleModelTests/test_ensemble_model_linear_threading.xml", the outputs will be printed once a simulation is finished. Regarding your second question, you can take a look at the test "raven/tests/framework/ensembleModelTests/test_ensemble_model_with_2_codes.xml", a csv file is generated for each simulation at the lowest level simulation directories (out~<code input file name>.csv or user predefined out file name).
If you still have the problem, could you provide the full input file and attach the whole screen output with verbosity="debug" for Simulation?

Best,

Congjian 

Stephen Heagy

unread,
Jan 25, 2023, 12:13:35 PM1/25/23
to Congjian Wang, Jarrett Valeri, inl-rav...@googlegroups.com, Cesare Frepoli

Congjian,

 

Thanks for the prompt response! 

 

I took a look at the csv output printing regression test (test_ensemble_model_linear_threading.xml) and it does what you suggest.  However, that’s not exactly what I’m looking for.  I would like to print out csvs for history sets for every RELAP run (every sampled uncertainty case), for multiple RELAP steps (steady-state and transient), as soon as each RELAP execution completes.  I attached my current RAVEN input and associated files (RELAP inputs and stdout) in Ensemble_Model_CSV_Stream.zip.  If you look at the time stamps provided below, you can see that the RELAP steady-state case (realization 1) finishes at a time of 11:10, while the transient case (realization 1) finishes at 11:11.  Upon completion of the transient cases, the csv files are generated with a time stamp of 11:11.

 

(raven_libraries) steveheagy@maximus:~/RAVEN_stream_test/Ensemble_Model_CSV_Stream> ls -ltr run_dir_stream/Ensemble/*++1/

run_dir_stream/Ensemble/RELAPss++1/:

total 448944

-rwxr-xr-x 1 steveheagy users       204 Jan 25 11:10 rellic.bin

-rwxr-xr-x 1 steveheagy users   1146252 Jan 25 11:10 tpfhe

-rwxr-xr-x 1 steveheagy users   1191700 Jan 25 11:10 tpfh2o

-rwxr-xr-x 1 steveheagy users    356614 Jan 25 11:10 RISA_ss.i

-rw-r--r-- 1 steveheagy users  64851212 Jan 25 11:10 restrt

-rw-r--r-- 1 steveheagy users 335435384 Jan 25 11:10 plotfl

-rw-r--r-- 1 steveheagy users  56715912 Jan 25 11:10 out~RISA_ss.o

-rw-r--r-- 1 steveheagy users      6519 Jan 25 11:10 out~RISA_ss

 

run_dir_stream/Ensemble/RELAPtr++1/:

total 1142356

-rwxr-xr-x 1 steveheagy users        204 Jan 25 11:10 rellic.bin

-rwxr-xr-x 1 steveheagy users    1146252 Jan 25 11:10 tpfhe

-rwxr-xr-x 1 steveheagy users    1191700 Jan 25 11:10 tpfh2o

-rwxr-xr-x 1 steveheagy users       8228 Jan 25 11:10 RISA_tr.i

-rw-r--r-- 1 steveheagy users   34888516 Jan 25 11:11 restrt

-rw-r--r-- 1 steveheagy users 1117335384 Jan 25 11:11 plotfl

-rw-r--r-- 1 steveheagy users   15182833 Jan 25 11:11 out~RISA_tr.o

-rw-r--r-- 1 steveheagy users       6747 Jan 25 11:11 out~RISA_tr

 

(raven_libraries) steveheagy@maximus:~/RAVEN_stream_test/Ensemble_Model_CSV_Stream> ls -ltr run_dir_stream/

total 46468

drwxr-xr-x  2 steveheagy users      42 Jan 25 11:10 DatabaseStorage

drwxr-xr-x 22 steveheagy users    4096 Jan 25 11:11 Ensemble

-rw-r--r--  1 steveheagy users 2162997 Jan 25 11:11 SS_OUT_HIST_0.csv

-rw-r--r--  1 steveheagy users 2589451 Jan 25 11:11 TR_OUT_HIST_0.csv

-rw-r--r--  1 steveheagy users 2146013 Jan 25 11:11 SS_OUT_HIST_1.csv

-rw-r--r--  1 steveheagy users 2589520 Jan 25 11:11 TR_OUT_HIST_1.csv

-rw-r--r--  1 steveheagy users 2159868 Jan 25 11:11 SS_OUT_HIST_2.csv

-rw-r--r--  1 steveheagy users 2588822 Jan 25 11:11 TR_OUT_HIST_2.csv

-rw-r--r--  1 steveheagy users 2173906 Jan 25 11:11 SS_OUT_HIST_3.csv

-rw-r--r--  1 steveheagy users 2589281 Jan 25 11:11 TR_OUT_HIST_3.csv

-rw-r--r--  1 steveheagy users 2152449 Jan 25 11:11 SS_OUT_HIST_4.csv

-rw-r--r--  1 steveheagy users 2588954 Jan 25 11:11 TR_OUT_HIST_4.csv

-rw-r--r--  1 steveheagy users 2160362 Jan 25 11:11 SS_OUT_HIST_5.csv

-rw-r--r--  1 steveheagy users 2587386 Jan 25 11:11 TR_OUT_HIST_5.csv

-rw-r--r--  1 steveheagy users 2161061 Jan 25 11:11 SS_OUT_HIST_6.csv

-rw-r--r--  1 steveheagy users 2589700 Jan 25 11:11 TR_OUT_HIST_6.csv

-rw-r--r--  1 steveheagy users 2163852 Jan 25 11:11 SS_OUT_HIST_7.csv

-rw-r--r--  1 steveheagy users 2589285 Jan 25 11:11 TR_OUT_HIST_7.csv

-rw-r--r--  1 steveheagy users 2160502 Jan 25 11:11 SS_OUT_HIST_8.csv

-rw-r--r--  1 steveheagy users 2589143 Jan 25 11:11 TR_OUT_HIST_8.csv

-rw-r--r--  1 steveheagy users    9765 Jan 25 11:11 SS_OUT_HIST.csv

-rw-r--r--  1 steveheagy users 2160290 Jan 25 11:11 SS_OUT_HIST_9.csv

-rw-r--r--  1 steveheagy users    6833 Jan 25 11:11 SS_OUT_HIST.xml

-rw-r--r--  1 steveheagy users    4298 Jan 25 11:11 TR_OUT_HIST.csv

-rw-r--r--  1 steveheagy users 2588597 Jan 25 11:11 TR_OUT_HIST_9.csv

-rw-r--r--  1 steveheagy users    4373 Jan 25 11:11 TR_OUT_HIST.xml

 

To provide full context, we are looking to improve our web interface and provide a way for a customer to investigate long running simulations (with hundreds of uncertainty cases) and explore the results as individual cases complete.

 

As far as my second question goes, when running a RELAP case using an old version of RAVEN it generates the out~.csv file but with the new version of RAVEN it does not.  Note, that when I ran the same test for a SAM (instead of RELAP) case it did successfully generate the out~*.csv using both versions of RAVEN.  I attached a zip (RELAP_Grid_Search.zip) with the input and outputs for both versions of RAVEN.  Note that I had to clean up some outputs to make the zip small enough to send.

 

(raven_libraries) steveheagy@maximus:~/RAVEN_out_csv_test/RELAP_Grid_Search> ls -ltr run_dir_*/Step1/1/

run_dir_old_raven_version/Step1/1/:

total 24288

-rwxr-xr-x 1 steveheagy users  1191700 Jan 25 11:55 tpfh2o

-rwxr-xr-x 1 steveheagy users      204 Jan 25 11:55 rellic.bin

-rw-r--r-- 1 steveheagy users   175783 Jan 25 11:55 UPTF.i

-rw-r--r-- 1 steveheagy users  8603892 Jan 25 11:56 restrt

-rw-r--r-- 1 steveheagy users 11926084 Jan 25 11:56 plotfl

-rw-r--r-- 1 steveheagy users  2834547 Jan 25 11:56 out~UPTF.o

-rw-r--r-- 1 steveheagy users     6483 Jan 25 11:56 out~UPTF

-rw-r--r-- 1 steveheagy users   116098 Jan 25 11:56 out~UPTF.csv

 

run_dir_new_raven_version/Step1/1/:

total 24172

-rwxr-xr-x 1 steveheagy users  1191700 Jan 25 12:02 tpfh2o

-rwxr-xr-x 1 steveheagy users      204 Jan 25 12:02 rellic.bin

-rw-r--r-- 1 steveheagy users   175803 Jan 25 12:02 UPTF.i

-rw-r--r-- 1 steveheagy users  8603892 Jan 25 12:02 restrt

-rw-r--r-- 1 steveheagy users 11926084 Jan 25 12:02 plotfl

-rw-r--r-- 1 steveheagy users  2834547 Jan 25 12:02 out~UPTF.o

-rw-r--r-- 1 steveheagy users     6559 Jan 25 12:02 out~UPTF

 

Old RAVEN Version:

commit 756c644d6e99115d1f2e2391a597dd579bfe3af7

Author: JiaZhou-PU Jia....@inl.gov

Date:   Fri Nov 15 19:58:14 2019 -0700

 

New RAVEN Version:

commit ea9c8a931028f2fae8716d28b6a25322e0496280

Author: Joshua J. Cogliati joshua-co...@users.noreply.github.com

Date:   Mon Nov 28 09:46:29 2022 -0700

Ensemble_Model_CSV_Stream.zip
RELAP_Grid_Search.zip

Stephen Heagy

unread,
Jan 25, 2023, 12:56:06 PM1/25/23
to Congjian Wang, Jarrett Valeri, inl-rav...@googlegroups.com, Cesare Frepoli

Congjian,

 

Resending with attachment name filename.inl.zip.

 

Thanks for the prompt response! 

 

I took a look at the csv output printing regression test (test_ensemble_model_linear_threading.xml) and it does what you suggest.  However, that’s not exactly what I’m looking for.  I would like to print out csvs for history sets for every RELAP run (every sampled uncertainty case), for multiple RELAP steps (steady-state and transient), as soon as each RELAP execution completes.  I attached my current RAVEN input and associated files (RELAP inputs and stdout) in Ensemble_Model_CSV_Stream sub-directory.  If you look at the time stamps provided below, you can see that the RELAP steady-state case (realization 1) finishes at a time of 11:10, while the transient case (realization 1) finishes at 11:11.  Upon completion of the transient cases, the csv files are generated with a time stamp of 11:11.

As far as my second question goes, when running a RELAP case using an old version of RAVEN it generates the out~.csv file but with the new version of RAVEN it does not.  Note, that when I ran the same test for a SAM (instead of RELAP) case it did successfully generate the out~*.csv using both versions of RAVEN.  I attached the run directories (RELAP_Grid_Search sub-directory) with the input and outputs for both versions of RAVEN.  Note that I had to clean up some outputs to make the zip small enough to send.

filename.inl.zip

Congjian Wang

unread,
Jan 26, 2023, 10:45:55 AM1/26/23
to Stephen Heagy, jarrett...@fpolisolutions.com, inl-rav...@googlegroups.com, frepolc
Hi Stephen,

Regarding your first question, I have checked your screen outputs, it seems to me the outputs are printed into the csv once they are available, see the following picture, the Printing data to CSV happened once the job is finished and collected. FYI, when you include OutStream Print in Multi-Run, for every step (i.e., every executions), all available data in the DataObject will be reprinted in the csv files instead of appending the csv files with new available data. This is the reason that the time for the printed CSV are the same. I think you should be able to access the csv files for finished job through the Multi-Runs.



Regarding your second question, I have checked the RELAP code interface, it seems the option to let the RELAP code interface to print outputs into csv inside the lowest executing folder is disabled. The code interface will return a dictionary contains all the outputs instead of csv file. If you would like to enable this feature, I think you need to make a small modifications in the method "finalizeCodeOutput" of Relap5Interface.py to print the returns from "finalizeCodeOutput" into csv files. Please feel free to create a PR if you would like to contribute your changes into RAVEN main repository. You can also create a feature request issue here (https://github.com/idaholab/raven/issues), and we will try to find some time to make the changes.

Please let me know if you have additional questions.

Best,

Congjian 

Stephen Heagy

unread,
Jan 27, 2023, 11:07:45 AM1/27/23
to Congjian Wang, Jarrett Valeri, inl-rav...@googlegroups.com, Cesare Frepoli

Congjian,

 

Jarrett and I investigated the Ensemble Model a little bit more by increasing the run-times and number of RELAP sample cases (to exceed the number of processors on our computing server), we observed what I believe you are describing below.  Basically, it runs a single chain of Steady-State and Transient runs and then writes the outstream csv files for that chain.  That should satisfy our use case of examining results for individual runs as they complete.  It would be useful to print the SS outstream immediately after the SS RELAP run completes, but it is not required. 

 

Regarding the second question.  I was able to update the initialization method of the Relap5Interface class to set the base class variable (self._writeCSV) to True.  This results in the out~<code input file name>.csv files being generated for every RELAP execution.  I don’t think I’m going to submit a pull request after all because I think I understand the motivation for removing this feature from the Relap interface: Why print all of the csvs when RAVEN has all of the data in a dictionary/data object and the user can just write the data to an outstream via the RAVEN input file? 

 

Thanks again for your help and responsiveness.  Have a great weekend!

Congjian Wang

unread,
Jan 27, 2023, 5:08:31 PM1/27/23
to Stephen Heagy, jarrett...@fpolisolutions.com, inl-rav...@googlegroups.com, frepolc
I'm glad that works. 

Congjian 
From: Stephen Heagy <steve...@fpolisolutions.com>
Sent: Friday, January 27, 2023 9:07 AM

Congjian Wang

unread,
Jan 27, 2023, 6:08:15 PM1/27/23
to Stephen Heagy, jarrett...@fpolisolutions.com, inl-rav...@googlegroups.com, frepolc
Hi Stephen,

Please see the following comments from Andrea. There is a node variable that you can use in your Input Code xml section to control the print. 


From: Stephen Heagy <steve...@fpolisolutions.com>
Sent: Friday, January 27, 2023 9:07 AM
Reply all
Reply to author
Forward
0 new messages