Multigrid access to data

[Victor Boniou]

Hello everyone,

To develop an algorithm to compute structure functions in 3D using a multigrid arrangement of data, I started from an existing algorithm of Antoon http://basilisk.fr/sandbox/Antoonvh/structure_function.h.

The main problem I face to adapt it to my needs is the data structure of the multigrid solver. To better understand how to access data, I did a very simple test where I run this simple loop after initializing the grid:

foreach()

    printf("%d,%g,%g,%g\n",pid(),x,y,z);

I used different number of processors in 3D on a 8*8*8 case (Level = 3) and obtain the following behaviour:

- 1 processor: returns 512 lines as expected -> all data are accessible

- 8 processor: each processor returns 64 lines as expected -> all data are accessible by compiling all processors data

- 27 processor: each processor returns 8 lines -> some of the data is not accessible as 27*8=212<512.

So the conclusion is that when Ncells/Nproc is not an integer, then I cannot access to all the data in a foreach loop. 

I would like to better understand why is it the case and how I could access to all the data points in the case of 27 processors.

Note that I am aware of the fact that the number of processors should be a cubic for multigrid (hence the choice of 3^3=27). But this restriction does not allow to always have Ncells/Nproc as an integer for larger cases (my target case is uses 512^3 points and 729 processors).

Does anyone have an explanation or a link to documentation to explain this behaviour?

I hope the question is clear,

Best,

Victor

[j.a.v...@gmail.com]

Hallo Victor

The 8x8x8 cells domain cannot be decomposed onto 3x3x3 processors. But the "smart" decomposition algorithm found 6x6x6 cells to be relatively close and divisable by 3x3x3 in each direction. Zo you should get 6x6x6=216 cells spaced over your cubic domain.

Antoon

Op donderdag 18 april 2024 om 11:18:36 UTC+2 schreef Victor Boniou:

[Victor Boniou]

Antoon,

Thank you for your prompt answer. I did not know that each processor should contain the same amount of cells. 

So, if I understand correctly, the number of cells is adjusted to be divisable by the number of processors? 

Then, it is not that data are not "available" as I thought but just that the number of cells have been decreased to match multigrid solver constraints?

Victor

[Victor Boniou]

Hello,

With further investigations I managed to clarify the behaviour of the multigrid solver:

1. The number of processors should be a cubic in 3D (8,27,64,125 ...). This information can be found here: http://basilisk.fr/src/Tips. Note that if you don't, the simulation will still run but with a non-cubic geometry.
2. The number of elements per direction sould be a power of 2 PER PROCESSORS. For example, if I ask for N = 256 elements per direction but I want to run it with Nproc = 125, it will modify the number of elements such that N/Nproc is a power of 2. Hence I will have a simulation with N=160=5*32 instead of the prescibred 256.

The second point might be obvious for people who are used to this type of data structure. However, I did not find the information in the documentation and I did the mistake, so it might be useful to remind people about this behaviour.

The second point also implies that using N=320 is totally okay as long as N/Nproc is a power of 2, which is not the case for octree (correct me if I'm wrong).

Best,

Victor