Request for summarised outputs from entity logger

[BR]

Hi Harry and others

I have been racking my brain for a while now on how I might supersede the entity logger and output summarised values directly, so that the produced log file is not so very large. The size of the log file I produce can exceed 5gb when running at the capacity needed for my model. This information summarised into a json would be a fraction, which would greatly speed up downstream processes and storage for audit needs.

By summarised I mean instead of getting a row for each record, records are compiled and counted.

The absolute best I can do without trying to make a new java class for jaamsim myself (which I am woefully unprepared for), is to try and use the ExternalProgramServer object in tandem with a python server to read in the entity's attributes, send that information to a new thread which stores it in a json and summarises it every so often, then return a dummy to jaamsim to get it to continue. I have attached my python and an extremely reduced cfg example for inspection (you will obviously need to change the python version).

However, inherently, the I/O process and requirement to constantly flush the console adds significant overhead to the timing of the model run (1.5-2x) when tested on my model proper. There may be timing improvements possible but the bottleneck is the I/O and I'm not really sure how to go about improving it any further.

There may be improvements to be made with the overall model sequencing. For example, would having more ExternalProgramServer nodes where data is recorded split this up so it can run in parallel? Do I need to regulate how objects queue and hit the ExternalProgramServer object? 

Ultimately - 

1) Am I missing a way to do this in jaamsim? I looked at the statistics object but I would need to define thousands of combinations to count.

2) Am I missing an efficiency in my cfg or python code?

3) Would this feature be addable in the future?

[Harry King]

Hi Ben,

The ExternalProgramServer object is fast, but not fast enough to be called for every entity generated by your model.

I think the approach you should use is to summarize the data within your model and print out the final results at the end of the run. The summary can be generated by an Assign object that has attributes for the quantity you want to count. You should be able to calculate anything you need using the expression system.

If you want to log outputs during your run, you can use the ExpressionLogger object. It can create a log entry at regular intervals or whenever some condition is reached.

Harry

[BR]

Hi Harry,

This was the first thing I thought of. However, given that the number of individual level attributes that I need to record are numerous (6+ depending on the scenario), with many different combinations, the number of attributes I'd need to record will be limiting.

E.g. I have to record on an individual level the age of the simulated entity. The range of this is 25. Another attribute has 3 states. Another has 10. Even at the bare minimum i'm looking at 750 different assigns. I could do that by creating the combinations in a programming language and inserting them with an include or a grep or something. But this is inelegant. And may cause issues with run time...?

Any suggestions from here?

Ben

,

[Harry King]

Hi Ben,

If there are far too many combinations to summarize the data within the model, how can you do it by post-processing without the same amount of work?

One approach you might try is to set an attribute equal to a hashmap whose key is a string that combines all the variables you are tracking, and whose value is the number of times this combination has occurred. You can then construct an Assign object that looks up the number of times a combination has occurred and then increment and store the new value.

Harry