Problems running DEMdemo_GRCPrep_Part1

[Heather Jones]

Dear Chrono Community,

I have encountered problems when running the DEM-Engine demo DEMdemo_GRCPrep_Part1. The first time I ran the GRCPrep Part1 demo, it crashed at frame 10 with the following terminal output:

terminate called after throwing an instance of 'std::runtime_error'
  what():  System max velocity is 52.22562, exceeded max allowance (15).
If this velocity is not abnormal and you want to increase this allowance, use SetErrorOutVelocity before initializing simulation.
Otherwise, the simulation may have diverged and relaxing the physics may help, such as decreasing the step size and modifying material properties.
This happened in unpackMyBuffer.

Aborted (core dumped)

I ran the DEMdemo_BallDrop2D, and it ran to completion with no errors. 

I then tried running the DEMdemo_GRCPrep_Part1 again and left it to run overnight. I saw no errors this time, but it ran through frame 60, and then the simulation hung. The process continued running, but no progress was made for hours.

I am running on a computer that has only one GPU. I was running code from the main branch of DEM-Engine as updated earlier this month, commit 56603feff39a8d4b02ae2b8a42d719dc4cb0c8d6. 

Thanks for any help on getting DEM-Engine to run more smoothly.

~Heather

[Ruochun Zhang]

Hi Heather,

First, I note that DEMdemo_GRCPrep_Part1 uses some randomness in generating each batch of particles instantiated into the simulation. Although I have yet to encounter it, I imagine it is possible that two large particles are generated so close together that there is an initial penetration, and the simulation crashes due to the subsequent large velocity. If your simulations did crash exactly after frame 10 and frame 60, then since they should be the frames right after a new batch of particles are added into the simulation, this appears to be more likely.

There are a few things you can do:

1. Increase the initial separation in the sampler. In line 103, you can change the separation to be something like 2.5*scales.at(0). This reduces the risk of initial penetration. Although I would indeed be a bit surprised that this consistently happens in your simulations twice.

2. Relax the physics as suggested. You can change the CoR of all the materials to something like 0.1. It increases the damping and should not change the final settling result, just makes the contact easier to calculate. Again, I feel doubt since it never happens to me.

3. The hanging problem could also be related to large velocities, or not enough memory. If everything in the simulation is moving large velocity (meaning the simulation probably already diverges), then a lot of false positive contacts could be detected and the long contact array could eat all the memory. This is by far the most common cause of hanging. I don't know if you indeed modified the max error-out velocity in your second try (using SetErrorOutVelocity) so it went longer than the first try: You can try doing that, but regardless, you should ensure enough memory is available. For example, do not run it on an ancient graphic card (I don't think you are though...), and if it is on a cluster, make sure the job you submitted requests enough (host) memory, like "--mem=10GB", or something like that. 

4. Due to what I said above, sharing the GPU you used and a rendering of the frames before the crash can help us understand what the exact problem is.

Thank you,

Ruochun

[Heather Jones]

Thanks Ruochan,

The computer has an NVidia RTX 3090 Ti graphics card. It should have had > 50GB system RAM available. Nothing appeared odd to me in the last frame before the system hung on my second attempt (screenshot attached), but there could be something I don't know to look for.

The DEMdemo_GRCPrep_Part1 did run all the way to completion with no errors on my third attempt. I did not change anything in the code (did not rebuild executables) between the times it failed and the time it succeeded.

Thanks,

Heather

[Ruochun Zhang]

Hi Heather,

By the look of it, it was a crash right after the instantiation of new particles. So yes, maybe they are instantiated in a way that could destabilize the simulation. I understood that the parameters chosen in the demo can theoretically be risky, but a smaller initial packing density creates more particles per batch and makes the simulation a bit shorter, and in my tests I've never seen a crash with this setup so I just kept it as is. But since you experienced it differently, I indeed recommend you relax the problem a bit like I suggested, if you need to rerun it in the future.

With all those said, if in the end you are looking to do a ground vehicle mobility test, the terrain you get with DEMdemo_GRCPrep_PartX is accurate (in terms of comparing performance against true GRC-1 simulant) and looks good if rendered with something like Blender; However, it is also a very numerically expensive simulation. My experience is that creating a terrain similar to the WheelDPSimplified demo, where using 3-sphere clumps only and potentially adjusting the size of the particles to match an expected terrain strength, is sufficient for simulation accuracy, and also an order of magnitude cheaper. That is probably something worth considering.

Thank you,

Ruochun