Handling output from simulations

[Alex]

Hi,

I would be grateful if anyone could help me with the following questions:

1. I've built a pretty big model and I am interested only in a subset of the statistics returned by the output report. Is there a way to control which statistics are returned by the output report as well as the format of the output report?

2. I am executing multiple replications of the model in batch mode and I would like to obtain directly the selected statistics averaged over all replications. Moreover, I'd like to combine some of the statistics in a single expression (e.g. the mean of the queue length average of several queues). Is there a way to do that (not manually)?

3. This is related to the 2nd question. Is there a way to call a Jaamsim simulation from a Java application so that I can control the input/output of the Jaamsim simulation from within the Java app?

Thanks in advance!!

 

[Harry King]

Alex,

1) Instead of trying to format the output report, your best bet is to write a spreadsheet macro to read the output file(s) and pull in the values you want into a summary spreadsheet. The report format was designed to make it relatively easy to parse in this way. We use a spreadsheet to both launch and summarize the results from runs performed over night in batch mode.

2) The procedure described in (1) also works for the replications. Calculate your replication statistics in the spreadsheet. Averages over several objects should also be calculated in the spreadsheet.

3) We have launched JaamSim directly from a Python program using a shell command. Java provides this ability as well but we haven't used it ourselves. We find Python to be the best language for writing a user friendly front-end.

Harry

[Alex]

Thanks a lot for your help Harry.

I'll give it a try and I'll let you know.

Hope I can get back to you if I have any problems :-)

[Yuriy Parfonov]

Hello Harry,

In my queueing model, I want to assess such performance measures as average waiting time, the busy period of the server, server utilization and so on.

If I get it right I can accomplish the task using EntityLogger, ExpressionLogger and Statistics objects or PrintReport keyword.

Then it necessary to parse the logs and use some program, for example, spreadsheet, to calculate the measures.

Is it so or I've  misunderstood something?

Thanks,

Yuriy

[Harry King]

Yuriy,

You can get many of the performance measures from the standard output file -- just set the PrintReport input to TRUE. This will print all the statistical outputs that you see in the Output Viewer to the report. The Queue object maintains quite extensive statistics on average time in queue, average queue length, queue length distribution, etc. The Server outputs will provide you with the total time it spends in the Working and Idle states. Utilization is just (Working Time)/(Total Time).

If you need more detailed outputs, say the distribution of waiting times, use the StateAssignment keyword for the Queue, Server, and EntityConveyors to set the state for the SimEntities as they pass through each object. Set the StateAssignment keyword to 'Waiting' for the Queue, 'Processing' for the Server, and 'Moving' for the first EntityConveyor after the Server. You can choose any names you want for the states. Test your model by selecting a generated SimEntity and watch its State output change as it passes through the model.

The last step is to add an EntityLogger just before the EntitySink to collect the state information for each SimEntity before it is destroyed. The EntityLogger will generate its own output report containing the outputs for each SimEntity that it processed. You can then use Excel to parse the log report and calculate whatever statistics you want.

Note - you must use a SimEntity to get the State output, not just a DisplayEntity.

Harry

[Yuriy Parfonov]

Hello Harry,

Thanks for your detailed explanation.

Meanwhile, I've conducted quite a lot of experiments with my model using PrintReport input, EntityLogger, ExpressionLogger and Statistics objects to collect data from it. Most of the experiments included multiple runs.

So I have several thoughts to share about my experience with JaamSim:

1) It is a nice feature to have all simulation outputs in the standard output file, but the one can be very large, especially with multiple runs. For instance, I used 50 replications in an experiment with a simple model that only consisted of about 15 objects and it resulted in the file with 5000 lines. Importing data to Excel was simple, however, extracting a subset needed to calculate performance measures was very time-consuming. Indeed, the process can be simplified with some script, but personally, I rather would write some program in Java or C# to parse the file with some standard parser and do the calculations.

Thus, I suggest changing plane text file by XML file as there are many standard XML parsers to use in the program. Also, I guess it simple to import XML in a spreadsheet too.

The XML file might look like:

<?xml version="1.0" encoding="UTF-8"?>

<simulation softname="JaamSim" softversion=... >

  <run rubnumber="1" index=...>  

    <customergen

        <statetimes idle="0.0" working="4.0 h"... />

        <numberadded>0.0</numberadded>

        <numberprocessed>74.0</numberprocessed>

    </customergen>

..................................................

  </run>

  ..................................................

  <run rubnumber="50" index=...>

  ...................................................

  </run>

</simulation>

And maybe all log files should be converted in XML format

2) In my opinion, it would be nice doing Statistics object output to its own file to avoid parsing the standard output file, for example if we need an only small portion of data to collect.

3) If someone just wants to display some average value in the model window, for instance in the Text object, a nice feature would have a "cumulative" output from a model object that accumulates the output from the object from multiple replications.

4) I thought about EntityLogger, ExpressionLogger, and Statistics to handle multiple runs, but as I understood building the source code from GitHub and running JaamSim yesterday the feature has already implemented.

5) It is great JaamSim feature to speed up real time of simulation. Maybe it would be useful to be able to slow down the time for simulating rapid processes.

P.S. Personally I consider Jaamsim as great simulation tool and appreciate your excellent work.

Yuriy

[Harry King]

Hi Yuriy,

It's great to get detailed feedback like this. Here are my initial thoughts on your comments:

1. I agree that the standard output report gets cumbersome for bigger models and/or multiple runs. I'm not sure about XML format - it's easy to parse but hard to read. I would rather give you better tools for creating customized outputs. Another user, Mark Manser, has suggested that we provide the ability to return the values of user-specified expressions to the command line -- one line for each run -- so that the outputs could be piped to other software for analysis. This would be a bit like the ExpressionLogger, except that the expressions would be evaluated only once at the end of each run. Would that work for your application?
2. See the previous item.
3. An object that evaluated an expression at the end of each run and recorded it to an array could be quite useful. There is an on-going debate on the JaamSim team on whether to provide the ability to carry out a series of simulation runs from a single copy of JaamSim or to write a program that launches multiple copies of JaamSim across multiple cores/computers. My feeling is that both approaches have merit and that they don't have to conflict with each other. The first approach is simpler and would be easier for people to use. The second approach is much more powerful but introduces greater complexity.
4. As you can see from GitHub, there has been lots of activity over the last month. We will be releasing a new build once some organisational changes have been completed.
5. I agree that it would be nice to run a simulation slower than real time. Up to now, I haven't been able to think of a good way to handle the user interface. I don't want to display a decimal value for the speed up value because ever increasing numbers of decimal digits would be needed as the speed is divided by two each time. Perhaps we should display the speed-up value as a ratio, e.g. 64:1 when the model is running 64 times faster, and 1:64 when the model is running 64 times slower.

Thanks again for the great feedback.

Harry

[Yuriy Parfonov]

Hi Harry,

Thank you very much for sharing your vision of the subject. In general, I agree with you maybe apart from not using XML.

As far as the issues are concerned I have such comments:

1. Ability to return the values of user-specified expressions to the command line how you've described it would be just fine. But what about a length of the line if a user wants several expressions to be evaluated?

3. Maybe using an object that evaluates an expression at the end of each run and records the value to some collection instead of an array would be more flexible? Concerning the approaches to carrying out a series of simulation runs from a single copy of JaamSim or from multiple copies, I guess an average user would choose the first. 

5. From my point of view displaying the speed-up value as a ratio would be OK.

Best regards,

Yuriy

[Yuriy Parfonov]

Hello,

Just for information (probably useful one).

To parse standard output file and calculate some statistics I've used Microsoft Access DBMS and its queries instead of Excel. From my point of view, it's more simple and less time-consuming solution than using Excel with no scripts.

Yuriy

[Harry King]

Yuriy,

The ability to run a simulation slower than real time has been added to release 2016-06.

Harry

[Harry King]

Yuriy,

The ability to print selected outputs for each run to the command line or a file has been added in release 2016-07.

Harry

[Yuriy Parfonov]

Harry,

Thank you very much again!

Yuriy

вторник, 5 апреля 2016 г., 3:29:39 UTC+3 пользователь Harry King написал: