MPI processes on Intel's new Alder Lake hybrid CPUs?

[Patrik Král]

Hi, has anyone created any guideline for how to use MPI on Intel's new hybrid Alder Lake CPUs with P-cores and E-cores? How are the calculation threads distributed to the cores? Are P-cores used first? Would it automatically use the E-cores for smaller meshes and meshes with easier calculations (how would it predict?)? I suppose there would need to be some changes done in the compilation instructions for these. Is there anyone working on this? Any thoughts?

I can imagine this gives a headache to the coders from NIST.

Thanks.

[Kevin McGrattan]

Intel is transitioning away from its "classic" Fortran and C compilers to the so-called LLVM (https://llvm.org/) infrastructure. Rather than me trying to explain what this all means, I'll just say that we now compile with the oneAPI Intel compilers, but we've not yet transitioned from the "classic" Fotran to the new Fortran. The new Fortran will have the ability to exploit GPUs and CPUs with the current OpenMP directives in FDS. We plan to transition to the new Fortran this year, once Intel has built in complete functionality. It's still in "beta" mode.

All this being said, we will continue to develop FDS in standard Fortran (at the moment 2018). Same with MPI and OpenMP. Hopefully, the new compilers will exploit the new hardware. I would assume it would, given it's all Intel. 

Mac OSX is a different problem. Apple doesn't seem to be playing nice with Intel, and it is developing new chips that the Intel compilers cannot exploit. We'll see how that plays out, but it's a minor annoyance for us given that most of our users have Windows or linux OS, for which Intel, Gnu, etc will continue to support.

Anyone who's keen on testing all this stuff is more than welcome. FDS will compile with Intel Fortran beta, but I haven't had time to work with it. We do have a cluster at NIST with GPUs and CPUs mixed together, but that's another challenge, just getting code to run on a new platform.

[Patrik Král]

I wish you luck with all this. It seems as a lot of work and headache. Maybe, with Apple CPUs, possibly ARM CPUs on Linux servers and AMD CPUs, it could be interesting to also switch from Intel MKL to OpenBLAS (for AMD, the best option would be LibM in AOCL, but it is surely not desirable to keep and develop several solvers on several math libraries). Intel has been ordered by court several times to switch off the performance locks like e.g. AVX2 for only Intel CPUs, but we know they have not done it. (not saying AVX2 is any huge advantage in FDS) 

With the GPU utilisation, I am very curious about the results. From some scientific papers, it seems very pomising, but I wonder how effective it will be with FDS. From my experience from ANSYS, it is beneficial particularly on smaller tasks, otherwise I would need a real high-end with a lot of VRAM. 

I could help you with some testing if I am of any use, but I am no proper IT guy. If you need some testing, I have access to:

Old MacBook – Intel (Haswell, I think. 2 cores)

Linux – AMD Renoir / NVidia RTX (6GB VRAM)

Windows – AMD Renoir / NVidia RTX (6GB VRAM)

Windows – Intel 9th Gen (Coffee Lake Refresh, I think) / NVidia Quadro p620

Windows Server VR – Intel Xeon Gold 6148 (or 6152 I am not sure now)

Unfortunately, I don't have access to the new M1 from Apple as of now

...

Dne středa 12. ledna 2022 v 3:58:52 UTC+13 uživatel Kevin napsal:

[Kevin McGrattan]

Thanks. Our primary goal at the moment is to run on as many platforms, OS, chips, etc, as possible. To that end, we try as much as possible to use standard Fortran, MPI, OpenMP and avoid proprietary libraries or OS-specific compiler directives and features. We do use the MKL libs for some research work related to solving the pressure equation, but it is not absolutely necessary, at least not yet. We'll see how things evolve.

The good news about all this is, believe it or not, it is far easier now than it was 30 years ago when I started my career to build code that runs on multiple platforms. Fortran has come a long way in adding features, and parallel programming is no longer strictly a research activity, at least in the linux world. Mac OSX and Windows are still challenging because they are not designed for high performance computing, obviously. But the fact that we can run multiple CPU jobs across a Windows office network is a minor miracle.