Running simulations concurrently in multiple threads

[Vikas J]

Hi experts,

Disclaimer: I am just a Java developer and have no knowledge of Modelling and Simulations so forgive me if I ask dumb questions or do not understand what is being said about simulations. 

I have this requirement where our Data Scientist designs a JaamSim model and this model is then used to run simulations in optimiser. The config file of a given model is used as a base config for all the runs. The simulations will be run for millions of records individually by changing the inputs accordingly. So I am looking at running them in parallel in multi threaded processes. Thanks to the new API released recently I have managed to run simulations in same JVM process. But when I try to run them in parallel I get following exception.

Unfortunately I cannot provide the model config file as it is proprietary work. I will try to reproduce this error with example files in your project.

com.jaamsim.events.ProcessError: Tried to schedule using an EventHandle already in use
	at com.jaamsim.events.EventManager.scheduleTicks(EventManager.java:820)
	at com.jaamsim.events.EventManager.scheduleTicks(EventManager.java:791)
	at com.jaamsim.ProcessFlow.Device.startStep(Device.java:141)
	at com.jaamsim.ProcessFlow.Device.restart(Device.java:65)
	at com.jaamsim.ProcessFlow.Device.thresholdChanged(Device.java:379)
	at com.jaamsim.Thresholds.Threshold$ThresholdChangedTarget.process(Threshold.java:134)
	at com.jaamsim.events.EventManager.executeTarget(EventManager.java:172)
	at com.jaamsim.events.EventManager.execute(EventManager.java:256)
	at com.jaamsim.events.Process.run(Process.java:101)

I am running simulations using following code in multiple threads, some statements are removed for brevity 

private void runTest(String item) throws InterruptedException {
        CountDownLatch countDownLatch = new CountDownLatch(1);

        JaamSimModel sim = new JaamSimModel("test-" + item);
        sim.autoLoad();

        InputAgent.readResource(sim, "./test.cfg");
        sim.setInput("Simulation", "PrintReport", "FALSE");

        //Set some input for the run

        EventTimeListener listener = new EventTimeListener() {
            @Override
            public void tickUpdate(long tick) {
            }
            @Override
            public void timeRunning() {
                if (EventManager.current().isRunning()) {
                    return;
                }
                countDownLatch.countDown();
            }
            @Override
            public void handleError(Throwable t) {
                log.error(t.getMessage(), t);
                countDownLatch.countDown();
            }
        };
        sim.setTimeListener(listener);
        sim.start();
        countDownLatch.await();
    }

I have encountered following other issues regarding running simulations in same JVM:

• Log file creation should be optional and the log messages should be redirected to source program's logging mechanism.
• The program in which JaamSim simulation is run using API cannot shutdown gracefully as there are GUI threads spawned. I know it will be hard, but API and engine code should be isolated from GUI.
• The run name should be a separate property, currently it is derived using config file name. If same config file is used for multiple runs it might be causing problems. 
  

Thanks

Vikas

[Harry King]

Vikas,

Can you and your Data Scientist prepare a simple input file that reproduces this error condition when executed through your API code? The error seems to be related to a Server or similar object that is controlled by an ExpressionThreshold. However, I am inclined to think that the problem is related to the way the multiple models are launched. If that is the case, then practically any non-trivial model should cause errors.

There should not be a GUI thread created when a model is launched through the API. There has been some additional attention to this in the latest release (2019-08) which might solve the problem. If it does not, can you provide an example that demonstrate the problem?

I noticed a few things in your code:

1. It would better if you used sim.configure instead of InputAgent.readResource to ensure that there are no errors in the input file.
2. There is no time-out mechanism in your runTest method. Look at the code in the unit test 'testAPI' in com.jaamsim.basicsim, specifically the WaitForPauseListener class.

Harry

[vickys...@gmail.com]

Thanks Harry for the prompt reply.

I will try to get a test model for you.

Meanwhile I am trying different models from this user group and I found one which produces following exception when run concurrently. The exception is different than what I get on my model file.

Resource1: Insufficient resource units: available=5, req'd=1

com.jaamsim.basicsim.ErrorException: Resource1: Insufficient resource units: available=5, req'd=1

at com.jaamsim.basicsim.Entity.error(Entity.java:613) ~[JaamSim.jar:na]

at com.jaamsim.resourceObjects.Resource.seize(Resource.java:129) ~[JaamSim.jar:na]

at com.jaamsim.ProcessFlow.Seize.seizeResources(Seize.java:221) ~[JaamSim.jar:na]

at com.jaamsim.ProcessFlow.Seize.startNextEntity(Seize.java:155) ~[JaamSim.jar:na]

at com.jaamsim.resourceObjects.AbstractResourceProvider.notifyResourceUsers(AbstractResourceProvider.java:132) ~[JaamSim.jar:na]

at com.jaamsim.ProcessFlow.Seize.thresholdChanged(Seize.java:105) ~[JaamSim.jar:na]

at com.jaamsim.Thresholds.Threshold$ThresholdChangedTarget.process(Threshold.java:134) ~[JaamSim.jar:na]

at com.jaamsim.events.EventManager.executeTarget(EventManager.java:172) [JaamSim.jar:na]

at com.jaamsim.events.EventManager.execute(EventManager.java:256) [JaamSim.jar:na]

at com.jaamsim.events.Process.run(Process.java:101) [JaamSim.jar:na]

With following code I can reproduce (even in `2019-08` release) the issue of process not shutting down gracefully. The test.cfg file is a copy of test file from unit tests.

import com.jaamsim.basicsim.JaamSimModel;
import com.jaamsim.events.EventManager;
import com.jaamsim.events.EventTimeListener;

import java.net.URISyntaxException;
import java.nio.file.Paths;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

public class JaamSimSimulator {

    public static void main(String[] args) throws URISyntaxException, InterruptedException {
        final JaamSimModel sim = new JaamSimModel("test");

        // Load the autoload file
        sim.autoLoad();

        sim.configure(Paths.get("./test.cfg").toFile());

      CountDownLatch countDownLatch = new CountDownLatch(1);

      //Set some input for the run

      EventTimeListener listener = new EventTimeListener() {
         @Override
         public void tickUpdate(long tick) {
         }
         @Override
         public void timeRunning() {
            if (EventManager.current().isRunning()) {
               return;
            }
            countDownLatch.countDown();
         }
         @Override
         public void handleError(Throwable t) {

            System.out.println("Simulation Failed:");
            t.printStackTrace();
            countDownLatch.countDown();
         }
      };
      sim.setTimeListener(listener);
      sim.start();
      boolean successful = countDownLatch.await(10, TimeUnit.MINUTES);

      if (successful) {
         System.out.println("Simulation finished successfully.");
      } else {
         System.out.println("TIMEOUT: simulation timed out");
      }

    }

}

Thanks

Vikas

[Vikas J]

Hi Harry,

I have created a demo project https://github.com/vikasjagtap/concurrent-jaamsim

When you run the JaamSimApplication class it will automatically run the concurrent execution of simulations. I have used config file from this post https://groups.google.com/d/msg/jaamsim-users/mtQKL4pBMG8/DDDEUU7oAAAJ

Thanks

Vikas

 

[Simon Taylor]

Hi Vikas,

I can help. We have a prototype cloud implementation that does what you want. It can run from Dropbox. 

Can you let me know roughly how many models and roughly how long a model runs for?

Best wishes

Simon

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Jaamsim Users Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jaamsim-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jaamsim-users/4d98ff26-98ef-4c1d-b7a7-5454ddbaaa9f%40googlegroups.com.

[Harry King]

Vikas,

With your help, I have been able to reproduce the error. It turns out that there are some static properties in the Threshold class that we missed. I will get back to you when we decide how to fix this problem.

Harry

[Harry King]

Vikas,

The problem with Threshold class is fixed in the HarryWIP branch on GitHub. The commit in question is 1e215c153cc6fc41fa179a892e49860fda79d67c (JS: move the updating of thresholds to JaamSimModel).

Your model should now run correctly with parallel runs. Let me know if you have any other problems.

Harry

[vickys...@gmail.com]

Harry,

Thanks for the fix, I can confirm that the simulations are running now concurrently. 

I have this problem where when I use JaamSimModel.configure method to load the config file, it tries to create, delete the log file with the same name and the process fails as there are multiple threads trying to work on the same log file. So to get myself up and running I added apiMode flag to JaamSimModel which does not create and open log file. 

Also GUIFrame and DisplayModel classes need GUI toolkit which spawns GUI thread and that is stopping the main process to be shutdown gracefully.

Thanks

Vikas

[vickys...@gmail.com]

Hi Simon,

There is only one model but inputs to the model differ per record. Number of records can be up to million.

The model runs vary between few seconds to 5 minutes depending on the record inputs which influences the SImulation.RunDuration

I hope it makes sense.

Thnaks

Vikas

To unsubscribe from this group and stop receiving emails from it, send an email to jaamsi...@googlegroups.com.

[Harry King]

Vikas,

I have this problem where when I use JaamSimModel.configure method to load the config file, it tries to create, delete the log file with the same name and the process fails as there are multiple threads trying to work on the same log file. So to get myself up and running I added apiMode flag to JaamSimModel which does not create and open log file. 

HK - I see what you mean about the configure method creating the .log file. Your use of readResource is fine. Also, I like your use of the CountDownLatch object and have adopted this ourselves in the latest version of the unit tests on the HarryWIP branch. You should still include a time-out feature, though. See the new version of 'waitForPause' in TestSimulation.java.

Also GUIFrame and DisplayModel classes need GUI toolkit which spawns GUI thread and that is stopping the main process to be shutdown gracefully.

HK - I don't see how the GUI thread is created when a JaamSimModel is started through the API. If the JaamSimModel's 'gui' property is null, there is no reference to the GUIFrame class. Are you sure that it is not just one of the threads running a JaamSimModel instance that is not shutting down? A time-out feature would solve this problem.

Harry

[Clever Alves]

Thank you for sharing such an interesting discussion, Harry and Vikas!

I'd like to learn a little bit more from you. Could you please address the questions below?

i) Does the multithreading you're talking about mean two JaamSim models working together and running on different machines? If so, how do you deal with time coordination/synchronisation?

ii) How does the API ("unit test 'testAPI' in com.jaamsim.basicsim") help in this process? 

iii) Could this API be used to interface JaamSim code with a Java external application-specific code?

Thank you for your attention.

Regards,

Clever.

[Harry King]

Clever,

i) Does the multithreading you're talking about mean two JaamSim models working together and running on different machines? If so, how do you deal with time coordination/synchronisation?

HK - No, we are talking about two independent models running simultaneously on one computer using separate threads, for example two replications with different random seeds.

 

ii) How does the API ("unit test 'testAPI' in com.jaamsim.basicsim") help in this process?

HK - This unit test shows how to launch two simultaneous models (instances of JaamSimModel).

 

iii) Could this API be used to interface JaamSim code with a Java external application-specific code?

HK - The API can be used to perform simulation analyses within a larger application. For example, if you had a model for a client's factory, you could write a program with a nice user interface that would allow management to forecast production based on the present state of the factory. This would involve updating model inputs from the factory's database, launching a series of simulation runs, collecting and analyzing the results, and presenting the forecast in terms of suitable key performance indicators.

Harry