Memory error in compiling a file

[Andrianarivo Tantelitiana RAKOTOARIJAONA]

Hi,
For my work, i try to use scram for doing a calculation. when i use a small tree as xml file,I got a nice result. But when the file is bigger it's cause a trouble like this for example:
First i compile the file : (i don't success to use the configuration mode when i want to compile, so i did it directly wihtout  the configuration file)
scram example9601-basic-event.xml example9601.xlm>result_file.xml.
After the execution , i have as a result:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Abandon.
This kind of trouble dont't let me to work with scram when i want to use a big xml file for doing calculation.
So my question is: what can I do to resolve or to avoid this kind of problem? And how can we get a more lighter file containing the result?

Thank you for your answers.

Andri

[D L]

Although I am a newcomer myself and I do not have an answer I am interested in learning how to debug/troubleshoot.

Intuitively, what do you see with --verbosity option?

When I run command. ..

scram --help

I see these options to try ..

--validate                Validate input files without analysis
--verbosity int           Set log verbosity

[Olzhas Rakhimov]

Hello,

On Tuesday, July 4, 2017 at 4:48:35 AM UTC-7, Andrianarivo Tantelitiana RAKOTOARIJAONA wrote:

Hi,
For my work, i try to use scram for doing a calculation. when i use a small tree as xml file,I got a nice result. But when the file is bigger it's cause a trouble like this for example:
First i compile the file : (i don't success to use the configuration mode when i want to compile, so i did it directly wihtout  the configuration file)
scram example9601-basic-event.xml example9601.xlm>result_file.xml.

This is indeed a demanding non-coherent model with over 10 million resultant products.
Currently, only BDD algorithms generate all the products;
for other algorithms, you should start slow with approximations.
That is, limit the product order to 3, then 4, and so on,
till the performance-vs-accuracy is acceptable to you.
Keep this in mind when dealing with large models.

After the execution , i have as a result:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

In order to debug this, we need to know the setup you are using (i.e., version, custom compilation).
This error suggests that you are not using one of the custom allocators (TCMalloc or JMalloc)
usually provided with the distribution packages with SCRAM.

Abandon.
This kind of trouble dont't let me to work with scram when i want to use a big xml file for doing calculation.
So my question is: what can I do to resolve or to avoid this kind of problem? And how can we get a more lighter file containing the result? 

Start slow; then crank up the accuracy.
The current XML format is indeed inefficient.
If you compress it with gzip,
the file becomes 40-50 times smaller.
For your model, it would be reduction from 2+GB to 40-50 MB.
However, this takes time (around 3x performance overhead).
You can directly pipe the output of scram into gzip to get the compressed file.

Another approach is to work with the API directly,
which provides access to ZBDD;
This approach pretty much eliminates reporting-translation overhead,
but you need to get the actual code and work in C++.
Hopefully, there will by Python front-end in future.

[D L]

scram example9601-basic-event.xml example9601.xlm>result_file.xml.
This is indeed a demanding non-coherent model with over 10 million resultant products.

I am following this discussion from the sidelines.  But I am curious.   in trying to learn I have missed a few chapters in this thread.

Where is the link to this "non coherent model" to be viewed by us novices?

[Olzhas Rakhimov]

​In 'input' directory of the scram installation (share/scram/input),

or https://github.com/open-psa/mef-examples​.

--
You received this message because you are subscribed to the Google Groups "SCRAM Support and User Discussions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scram-users+unsubscribe@googlegroups.com.
To post to this group, send email to scram...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scram-users/fca9a918-ebd9-4ab6-b597-93ea53dd0220%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrianarivo Tantelitiana RAKOTOARIJAONA]

Thank you for your reply.
 I am still a  beginner in this field so I don't really get what to do when you said to use the API.
And for the other way, i can't even get the xml file containing the result because of the fact that
the compilation don't success or my PC crashes  by generating a very heavy file.
By the way, i did the test with the example file you gave in GITHUB. I used the file named
CEA9601.xml .

[Olzhas Rakhimov]

​​

On Wed, Jul 5, 2017 at 1:15 PM, D L <dl6...@gmail.com> wrote:

To gain experience I tried to open above file in scram-gui and received this error message.

  Initialization Error

scram error: in file '/home/dl/scram-develop/inputCEA9601/CEA9601.xml', line 23:


Undefined basic-event e1 with base path cea9601

​You have to open the corresponding basic-events file as well.

The GUI allows opening multiple files at once.​

[Andrianarivo Tantelitiana RAKOTOARIJAONA]

Hi,

About the scram-gui +file name , I could not launch this command, i got each time a message who said that this command in not found . Moreover, i tried to document myself (http://scram.readthedocs.io/en/latest/doc/gui.html)more about how to launch or install scram-gui but finaly I don't success  to launch it. So I don't really know how to do .

 

 

[D L]

I am another novice so perhaps we can compare notes.
I am documenting my workflow as I learn more.

First, it would help if you can define your development OS. I am using Ubuntu 16.04 (64 bits). 3GB memory.

Reading your first post you wrote ..

scram example9601-basic-event.xml example9601.xlm > result_file.xml

I see that you have typed “.xlm” instead of ".xml".
and you refer to “basic-event” and not "basic-events2.

Are they typing errors in posting? However you do generate result_file.xml

...

Now I tried my own scram command ...

scram CEA9601-basic-events.xml CEA9601.xml > result_file.xml

and my PC froze (exactly as you found).

I had to force reboot, not knowing if the scram command run had completed its course. Is there a command line option to monitor progress perhaps?

--verbosity int

After rebooting to cure the freeze I looked in the test folder and there was a file result_file.xml size 1.6 GB.

Also I inspected history of last commands using this command ..

history | tail

But there was no history of using last scram command as above. Terminal history is only updated at completionof any command. 

This suggests that the scram command did not complete before PC froze and I rebooted.

I tried opening result_file.xml with an XML editor .. XMLCopyEditor.

But again my PC froze.

Next I tried inspecting the file using this command

xmllint --noout your_test_file.xml; echo $?

But again my PC froze and I had to hard reboot.

I will post my development notes later on using Qt Creator.


Meanwhile, a question to Olzhas:

For debugging this particular test file set can the output results file be broken down into a number of smaller files which can later be concatenated?

[Olzhas Rakhimov]

I ran this case reluctantly because it is rather stress testing than a meaningful analysis exercise.

As I mentioned before, you should use the order limits for this input for analysis.

That is, run it with : scram --limit-order 3

Anyway, with limit order 20 (default), there are 130,281,976 products.
The RAM required for the analysis to succeed is around 2 GiB.

The analysis takes around 40 seconds.

Reporting 10 million products takes around 20 seconds and 2 GB of disk space,

so extrapolating to 13x,

the expected reporting time and disk-space usage is 260 seconds/26 GB.

No XML editor is capable of handling these large files (split or not).

You have to work with code or program a translator with SAX parsers (stream).

​As I said, if an analyst is interested in such large sets,

there's no better way but to work directly with ZBDD in C++ or Python or R.

Regards,

Olzhas Rakhimov​

[D L]

Development Notes

It certainly is a stress test but useful to understand what breaks.

XMLCopyEditor could not open the result_file.xml.

I searched for an editor to open large (1.6 GB) result_file.xml

https://askubuntu.com/questions/28847/text-editor-to-edit-large-4-3-gb-plain-text-file

I chose to install glogg 1.1.0-1 log viewer (which is in Ubuntu 16.04 software centre and easy to install) and used that to view result_file.xml. The glogg tool launched from terminal took a few minutes to index all the file.

glogg reported 53,688,649 lines indexed.

I navigated to the end of viewed result_ file.xml and line 53,688,649 was broken ..
so it seems that the scram process had not completed.

I saw errors in glogg terminal as reported here ...

https://github.com/nickbnf/glogg/issues/117

I have not yet built v1.1.4 which cures that bug.

Another approach tried was using split command which works.

split -b 100k result_file.xml

I will now look for a more representative model to test my ongoing workflow.

OP still has a problem using scram-gui.

Thanks for insights.

[Olzhas Rakhimov]

On Thu, Jul 6, 2017 at 11:30 AM, D L <dl6...@gmail.com> wrote:

Development Notes

It certainly is a stress test but useful to understand what breaks.

XMLCopyEditor could not open the result_file.xml.

I searched for an editor to open large (1.6 GB) result_file.xml

https://askubuntu.com/questions/28847/text-editor-to-edit-large-4-3-gb-plain-text-file

I chose to install glogg 1.1.0-1 log viewer (which is in Ubuntu 16.04 software centre and easy to install) and used that to view result_file.xml. The glogg tool launched from terminal took a few minutes to index all the file.

glogg reported 53,688,649 lines indexed.

I navigated to the end of viewed result_ file.xml and line 53,688,649 was broken ..
so it seems that the scram process had not completed.

I saw errors in glogg terminal as reported here ...

https://github.com/nickbnf/glogg/issues/117

I have not yet built v1.1.4 which cures that bug.

Another approach tried was using split command which works.

split -b 100k result_file.xml

I will now look for a more representative model to test my ongoing workflow.

OP still has a problem using scram-gui.

Thanks for insights.

​I didn't mean a plain text editor. You can use vi/less or other simple text editor/viewer without much problem.

Processing any large XML with XML specific tools like BaseX, libxml2 DOM, vim/emacs with XML plugins

pretty much futile.

I want to emphasize this once more. SCRAM is not broken.

I am pretty sure none of you waited 6 minutes with 26 GB of spare disk space for the reporting to finish.

I bet OP hasn't even installed scram-gui in the first place to start having problems.

[D L]

On Thursday, 6 July 2017 19:39:52 UTC+1, Olzhas Rakhimov wrote:

​I didn't mean a plain text editor. You can use vi/less or other simple text editor/viewer without much problem.

Processing any large XML with XML specific tools like BaseX, libxml2 DOM, vim/emacs with XML plugins

pretty much futile.

I want to emphasize this once more. SCRAM is not broken.

I am pretty sure none of you waited 6 minutes with 26 GB of spare disk space for the reporting to finish.

I bet OP hasn't even installed scram-gui in the first place to start having problems.

I should have written .. what breaks in my PC .. not in scram. 

I did not wait 6 minutes because my PC froze.

Probably because of limited free RAM.

OP can test required Qt dependency by running command ..

qmake --version

This is what I see in my Ubuntu 16.04 ..

qmake --version
QMake version 3.0
Using Qt version 5.5.1 in /usr/lib/x86_64-linux-gnu

Further advice on using Qt Creator can surely then follow.

 

[Andrianarivo Tantelitiana RAKOTOARIJAONA]

Hi.
In order to change the limits--product-order , i try to get the result by using the configuration mode ( scram --config config_CEA9601.xml>result.xml) and it give me a message like:

scram::ValidationError:
scram error:In file 'config_CEA9601.xml',
Validity error:
Expecting element config, got scram

And in my config file , i put :
<?xml version="1.0"?>
<scram>
  <input-files>
    <file>CEA9601.xml</file>
    <file>CEA9601-basic-events.xml</file>
  </input-files>
  <options>
    <analysis importance="true" uncertainty="false" ccf="true"/>
    <limits>
      <product-order>3</product-order>
      <mission-time>8760</mission-time>
      <cut-off>0.001</cut-off>
    </limits>
  </options>
</scram>

I don't understand why doesn't it work. By following the indication in the website (https://scram-pra.org/doc/config_file.html)  i think i did the configuration correctly .

[D L]

Comparing experiences ..

If you run the command ..

scram --help

you will see this ..

scram --help
Usage:    scram [options] input-files...


Options:
  --help                    Display this help message
  --version                 Display version information
  --config-file path        XML file with analysis configurations
  --validate                Validate input files without analysis
  --bdd                     Perform qualitative analysis with BDD
  --zbdd                    Perform qualitative analysis with ZBDD
  --mocus                   Perform qualitative analysis with MOCUS
  --prime-implicants        Calculate prime implicants
  --probability bool        Perform probability analysis
  --importance bool         Perform importance analysis
  --uncertainty bool        Perform uncertainty analysis
  --ccf bool                Perform common-cause failure analysis
  --sil bool                Compute the Safety Integrity Level metrics
  --rare-event              Use the rare event approximation
  --mcub                    Use the MCUB approximation
  -l [ --limit-order ] int  Upper limit for the product order
  --cut-off double          Cut-off probability for products
  --mission-time double     System mission time in hours
  --time-step double        Time step in hours for probability analysis
  --num-trials int          Number of trials for Monte Carlo simulations
  --num-quantiles int       Number of quantiles for distributions
  --num-bins int            Number of bins for histograms
  --seed int                Seed for the pseudo-random number generator
  -o [ --output-path ] path Output path for reports
  --verbosity int           Set log verbosity

In particular note the option ..

  --config-file path        XML file with analysis configurations

So I ran the command ...

scram --config-file config_CEA9601.xml > results.xml

note that --config-file is used and not --config.

This works for me (I thought at first that the full path to the config file was needed).

..

On with more learning.

[Andrianarivo Tantelitiana RAKOTOARIJAONA]

I've already used this command but it give me the same thing.
I pretty think that this problem is due to my debian distribution.
I've just talked about it to an other person who use SCRAM on
UBUNTU 16.04LST and a recent version of DEBIAN
 distribution and with the command line scram --config config.xml 
,it work very well with these linux distribution.

[D L]

I am on Ubuntu 16.04 LTS.

I see that you are still referring to option --config when the options list shows that you need to use --config-file path.

When I run this command

scram --version

I see ..

SCRAM 0.13.0 ()

Dependencies:
 Boost 1_58
 libxml++ 2.40.1
 libxml2 2.9.3
run scram

I tried using --verbosity option in command with --config-file but got this return ..

Option error: the argument ('--config-file') for option '--verbosity' is invalid

Usage: scram [options] input-files...

[Olzhas Rakhimov]

​You are using an old version. The documentation refers to the latest version.

The config format has slightly changed (the root tag 'config' is renamed to 'scram') in the recent releases.

All the changes are listed in the release notes.

Regards,

Olzhas Rakhimov