ChronoDEM Simulating Small Particles

62 views
Skip to first unread message

Mohammed Saffarini

unread,
Apr 11, 2025, 3:58:01 PMApr 11
to ProjectChrono

Dear Chrono-DEM Team,

I am trying to simulate a setup similar to the demo DEMdemo_Repose, but for particles that are much smaller in radius.

The original demo file ran normally to the end. However, when I tried to build my own setup with particles sizes that I need, the setup did NOT run. In fact, after so many trials that didn't work out to create a setup of my own, I finally decided to take the demo file and just use a scaling factor that can get me the particle sizes that I am looking for. However, I am still unable to get the simulation working for that range of particles sizes.

The particle sizes I am targeting is between 10 and 20 micrometers. However, since I am not able to simulate this size, I tried particles sizes in the range of 100 - 200 micrometer (Using a scale factor of 0.01 instead of the 2 that is in the original demo file). I made the necessary modification to get everything else properly scaled as well. However, the simulation takes an average of 8 - 12 minutes on 2 GPUs for every 0.1 seconds of simulation time. The system has 8,850 particles only.

The following warning is shown for small particles range:

WARNING! A type of clump is instructed to have near-zero (or negative) mass or moment of inertia (mass: 2.0000003e-10, MOI magnitude: 3.080585e-18). This could destabilize the simulation.
Please make sure this is intentional.


WARNING! A type of clump is instructed to have near-zero (or negative) mass or moment of inertia (mass: 1.00000015e-10, MOI magnitude: 1.5402925e-18). This could destabilize the simulation.
Please make sure this is intentional.


WARNING! A type of clump is instructed to have near-zero (or negative) mass or moment of inertia (mass: 4.00000061e-10, MOI magnitude: 6.16117001e-18). This could destabilize the simulation.
Please make sure this is intentional.


WARNING! A type of clump is instructed to have near-zero (or negative) mass or moment of inertia (mass: 2.0000003e-10, MOI magnitude: 3.080585e-18). This could destabilize the simulation.
Please make sure this is intentional.


WARNING! A type of clump is instructed to have near-zero (or negative) mass or moment of inertia (mass: 2.0000003e-10, MOI magnitude: 3.080585e-18). This could destabilize the simulation.
Please make sure this is intentional.


WARNING! A type of clump is instructed to have near-zero (or negative) mass or moment of inertia (mass: 2.0000003e-10, MOI magnitude: 3.080585e-18). This could destabilize the simulation.
Please make sure this is intentional.

 

And finally, the following error is obtained:

-------- Simulation crashed "potentially" due to too many geometries in a bin --------
The dT reported max velocity is 0

------------------------------------
If the velocity is huge, then the simulation probably diverged due to encountering large particle velocities.
Decreasing the step size could help, and remember to check if your simulation objects are initially within the domain you specified.

------------------------------------
If the velocity is fair, and you are using a custom force model, one thing to do is to SetForceCalcThreadsPerBlock to a small number like 128 (see README.md troubleshooting for details).
If you are going to discuss this on forum https://groups.google.com/g/projectchrono, please include a visual rendering of the simulation before crash.

terminate called after throwing an instance of 'std::runtime_error'
  what():  GPU Assertion: unspecified launch failure. This happened in /home/4h1/src/DEM-Engine/src/algorithms/DEMCubContactDetection.cu:384


srun: error: mod-condo-g28: task 0: Aborted (core dumped)

 

Here below, I am providing the latest code I used for the range of 10 - 20 micrometers in particle’s radius. Please let me know what the problem is here:

#include <core/ApiVersion.h>

#include <core/utils/ThreadManager.h>

#include <DEM/API.h>

#include <DEM/HostSideHelpers.hpp>

#include <DEM/utils/Samplers.hpp>

#include <cstdio>

#include <chrono>

#include <filesystem>

using namespace deme;

using namespace std::filesystem;

int main() {

    DEMSolver DEMSim;

    DEMSim.UseFrictionalHertzianModel();

    DEMSim.SetVerbosity(INFO);

    DEMSim.SetOutputFormat(OUTPUT_FORMAT::CSV);

    DEMSim.SetNoForceRecord();

    srand(42);

    float scaling = 0.001;

    int num_template = 6;

    int min_sphere = 1;

    int max_sphere = 5;

    float min_rad = 0.01 * scaling;

    float max_rad = 0.02 * scaling;

    float min_relpos = -0.01 * scaling;

    float max_relpos = 0.01 * scaling;

    auto mat_type_walls = DEMSim.LoadMaterial({{"E", 1e8}, {"nu", 0.3}, {"CoR", 0.3}, {"mu", 1}});

    auto mat_type_particles = DEMSim.LoadMaterial({{"E", 1e9}, {"nu", 0.3}, {"CoR", 0.7}, {"mu", 1}});

    DEMSim.SetMaterialPropertyPair("CoR", mat_type_walls, mat_type_particles, 0.3);

auto funnel = DEMSim.AddWavefrontMeshObject(GetDEMEDataFile("mesh/funnel.obj"), mat_type_walls);

funnel->Scale(0.01);

float funnel_bottom = 0.f;

std::vector<std::shared_ptr<DEMClumpTemplate>> clump_types;

for (int i = 0; i < num_template; i++) {

        int num_sphere = rand() % (max_sphere - min_sphere + 1) + 1;

        float mass = 0.1 * (float)num_sphere * std::pow(scaling, 3);

        float3 MOI = make_float3(2e-5 * (float)num_sphere, 1.5e-5 * (float)num_sphere, 1.8e-5 * (float)num_sphere) * 50. * std::pow(scaling, 5);

        std::vector<float> radii;

        std::vector<float3> relPos;

        float3 seed_pos = make_float3(0);

        for (int j = 0; j < num_sphere; j++) {

            radii.push_back(((float)rand() / RAND_MAX) * (max_rad - min_rad) + min_rad);

            float3 tmp;

            if (j == 0) {

                tmp.x = 0;

                tmp.y = 0;

                tmp.z = 0;

            } else {

                tmp.x = ((float)rand() / RAND_MAX) * (max_relpos - min_relpos) + min_relpos;

                tmp.y = ((float)rand() / RAND_MAX) * (max_relpos - min_relpos) + min_relpos;

                tmp.z = ((float)rand() / RAND_MAX) * (max_relpos - min_relpos) + min_relpos;

            }

            tmp += seed_pos;

            relPos.push_back(tmp);

            int choose_from = rand() % (j + 1);

            seed_pos = relPos.at(choose_from);

        }

 

        auto clump_ptr = DEMSim.LoadClumpType(mass, MOI, radii, relPos, mat_type_walls);

        clump_types.push_back(clump_ptr);

    }

    float spacing = 0.1 * scaling; //0.08 * scaling;

    float fill_width = 0.01f;

    float fill_height = 0.02f * fill_width;

    float fill_bottom = funnel_bottom + fill_width + spacing;

    PDSampler sampler(spacing);

    std::vector<std::shared_ptr<DEMClumpTemplate>> input_pile_template_type;

    std::vector<float3> input_pile_xyz;

    float layer_z = 0;

    while (layer_z < fill_height)

    {

        float3 sample_center = make_float3(0, 0, fill_bottom + layer_z + spacing / 2);

        auto layer_xyz = sampler.SampleCylinderZ(sample_center, fill_width, 0);

        unsigned int num_clumps = layer_xyz.size();

        for (unsigned int i = 0; i < num_clumps; i++) {

            input_pile_template_type.push_back(clump_types.at(i % num_template));

        }

        input_pile_xyz.insert(input_pile_xyz.end(), layer_xyz.begin(), layer_xyz.end());

        layer_z += spacing;

    }

    auto the_pile = DEMSim.AddClumps(input_pile_template_type, input_pile_xyz);

 

    DEMSim.InstructBoxDomainDimension({-10, 10}, {-10, 10}, {funnel_bottom - 1.f, funnel_bottom + 20.f});

    DEMSim.InstructBoxDomainBoundingBC("top_open", mat_type_walls);

    DEMSim.SetInitTimeStep(1e-7);

    DEMSim.SetGravitationalAcceleration(make_float3(0, 0, -9.81));

    DEMSim.SetMaxVelocity(25.);

    DEMSim.SetErrorOutAvgContacts(10000);

    DEMSim.SetForceCalcThreadsPerBlock(1);

    DEMSim.Initialize();

    path out_dir = current_path();

    out_dir += "/DemoOutput_Repose";

    create_directory(out_dir);

 

    std::chrono::high_resolution_clock::time_point start = std::chrono::high_resolution_clock::now();

    for (int i = 0; i < 140; i++) {

        char filename[200], meshfile[200];

        sprintf(filename, "%s/DEMdemo_output_%04d.csv", out_dir.c_str(), i);

        sprintf(meshfile, "%s/DEMdemo_funnel_%04d.vtk", out_dir.c_str(), i);

        DEMSim.WriteSphereFile(std::string(filename));

        DEMSim.WriteMeshFile(std::string(meshfile));

        std::cout << "Frame: " << i << std::endl;

        DEMSim.DoDynamics(1e-1);

        DEMSim.ShowThreadCollaborationStats();

    }

std::chrono::high_resolution_clock::time_point end = std::chrono::high_resolution_clock::now();

std::chrono::duration<double> time_sec = std::chrono::duration_cast<std::chrono::duration<double>>(end - start);

std::cout << time_sec.count() << " seconds (wall time) to finish the simulation" << std::endl;

DEMSim.ShowTimingStats();

DEMSim.ClearTimingStats();

std::cout << "DEMdemo_Repose exiting..." << std::endl;

return 0;

}

Dan Negrut

unread,
Apr 11, 2025, 4:43:44 PMApr 11
to Mohammed Saffarini, ProjectChrono

Hi Mohammed – unfortunately, I do not have time to dig deep in this, but looking at the values of MOI that you report, the “double precision” type that exists on a computer cannot do justice to what you are trying to do. The 64-bit size is not large enough to capture what you want.

 

That being said, one thing that I would look into is to change the units used to model the problem. Chrono DEM is unitless, as long as you are consistent with the units for stiffness, mass, time, etc. everything should be ok. For instance, use mm for position, and then amend the value of the stiffness, to have it expressed in these units as well. Also – one more thing. Be prepared to have some *tiny* time steps. When you have particles on this scale, and they are stiff, the natural frequency is super, super high. The step size is 1/10 of that. Please reflect on this, and understand what you sign up for. Your solution is going to take a long time. It’s not Chrono DEM that is slow, it’s going to be the nature of the problem you try to solve.

 

Also, as an alternative, depending on what problem you solve, you might want to look into a continuum representation of the granular problem. There is support for that in Chrono, it’s called CRM. Again, I don’t know if the problem you try to solve is amenable to a homogenization of the discrete material to a continuum. But if its, you’ll be orders of magnitude faster with CRM.

 

I hope this makes sense, perhaps it helps a bit.

Good luck.

Dan

---------------------------------------------

Bernard A. and Frances M. Weideman Professor

NVIDIA CUDA Fellow

Department of Mechanical Engineering

Department of Computer Science

University of Wisconsin - Madison

4150ME, 1513 University Avenue

Madison, WI 53706-1572

608 772 0914

http://sbel.wisc.edu/

http://projectchrono.org/

---------------------------------------------

--
You received this message because you are subscribed to the Google Groups "ProjectChrono" group.
To unsubscribe from this group and stop receiving emails from it, send an email to projectchron...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/projectchrono/ecd1307c-8f4f-4066-9add-1999d42015b1n%40googlegroups.com.

Ruochun Zhang

unread,
Apr 11, 2025, 4:52:13 PMApr 11
to ProjectChrono
Hi Mohammed,

Just to add to what Dan said and be a bit more specific...

First I'd like to address the speed issue. I see you used "SetForceCalcThreadsPerBlock(1)". Please remove it. As noted in the error message, it is for custom force model users (so that they can tweak it to prevent some CUDA failures when their model uses too much local memory per thread). It seems you are not using a custom model, so what it solely does here is launch 1 thread per CUDA block for force calculation, slowing things down some 100 times.

The bigger problem appears to be that everything in your simulation combined, probably occupies a cube in space not larger than several millimeters across. Yet the simulation world is around 10 meters in size (see the InstructBoxDomainDimension line). The solver needs to subdivide the domain to distribute the workload, and such a size discrepancy probably leads to a situation, where however the subdivision is created, one of the subdomains will contain almost all simulation entities, and bust its memory; it cannot create a finer subdivision either, as that busts the pool of available domain IDs. You should scale the simulation world with everything else, too, though I seem to have forgotten to put scaling in that line. I thought about a more dynamic subdivision strategy, but right now it's not a high development priority, and usually, domain sizes just somewhat physically meaningful should work fine.

After you make these changes it might run, but I wouldn't be sure if it represents anything like you wanted to do---this demo's clump templates are quite complex, and there's definitely no guarantee that a demo runs without some fine-tuning after you scaled a variable some 1000 times. You should consider using ParaView to create a rendering and include a screenshot (even if it fails at step 1, we can still have a look at the initial setup) for further discussion, which can help us rule out so many possibilities.

Thank you,
Ruochun

Reply all
Reply to author
Forward
0 new messages