Inquiry About Load Balancing and Manual Partitioning in Basilisk CFD with adaptwavelet

[Jinglei Xu]

Dear Basilisk Group,

I am working on a parallel CFD program implemented in C using Basilisk with MPI and the adaptwavelet library for adaptive mesh refinement (AMR). I have observed an interesting performance behavior and would appreciate your insights on whether manual repartitioning or specific load balancing mechanisms could improve efficiency.

In my simulations, I use AMR to dynamically refine the grid, but this impacts computational speed, likely due to load imbalance across MPI processes. To investigate, I set the simulation time to ( t = t_1 ), after which the grid no longer adapts. During this static grid phase, I noticed that the wall time per time step gradually decreases as the number of steps (( nstep )) increases. By the end of the simulation, the wall time is approximately three times faster than during the adaptive phase. The performance improvement slows down over time but continues to trend upward.

This behavior raises two questions:

1. Does the gradual performance improvement in the static grid phase indicate that Basilisk's MPI partitioning or load balancing (e.g., via adaptwavelet) is dynamically optimizing the workload distribution? If so, is this handled internally by Basilisk or adaptwavelet, and are there specific mechanisms I should be aware of?

2. Would it be beneficial to manually repartition the computational grid across CPU cores at ( t = t_1 ), when the grid becomes static, to achieve optimal MPI parallel efficiency? For example, could I use external tools like METIS or ParMETIS, or is there a way to leverage Basilisk's built-in load balancing functionality (e.g., through balance.h) to achieve this?

I am considering whether manually calling functions from balance.h or integrating an external partitioning tool could further optimize performance, especially given the observed trend. Any advice on whether Basilisk already handles such optimizations internally, or if manual intervention is recommended, would be greatly appreciated. Additionally, if there are specific parameters or functions in adaptwavelet or balance.h that I should explore, please let me know.

Thank you for your time and support. I look forward to your guidance.

Best regards,
Jinglei Xu