PyFR on Xeon Phi

[Tom Vilsack]

Dear All,

I'm trying to run PyFR on Xeon Phi 7250 (Knight Landing). There is only one Xeon Phi processor, and no CPU on my computing node. Therefore, I log in to the Xeon Phi from front-end node via ssh, and execute command on Xeon Phi directly.

In this case, how I can write "backend-mic" section in .ini file? Could you tell me sample description of "device-id" and "mkl-root" parameters?

I could run PyFR using "backend-openmp" settings on Xeon Phi. However, I think simulation speed will be faster if I can run PyFR with mic native functions.

Best regards,
Tom

[Freddie Witherden]

Hi Tom,

For KNL you want to be using the OpenMP backend. The key here is to
make sure that you get the number of OpenMP threads correct, have MKL
available, use the ICC compiler, and have libxsmm installed. Together,
these should result in rather good performance for non-pathological test
cases.

The MIC backend is for the legacy Knights Corner.

Regards, Freddie.

[Tom Vilsack]

Hi Freddie,

Thank you for your reply.

I have installed libxsmm according to commands as described below.

  make STATIC=0 BLAS=0
  make PREFIX=<libxsmm install path> STATIC=0 BLAS=0 install

The make process finished without error. Then, I set parameter in "backend-openmp" section in ini file.

  [backend-openmp]
  cc = icc
  cblas = <libxsmm install path>/lib/libxsmm.so

Are the setting parameters correct? Xeon Phi 7250 have 68 physical cores, and each core can run 4 threads. Therefore, I set $OMP_NUM_THREADS=272.

Could you tell me meanings of "gimmik-max-nnz", "libxsmm-block-sz" and "libxsmm-max-sz" in "backend-openmp" section in ini file?

Best regards,
Tom

[Freddie Witherden]

Hi Tom,

On 06/06/17 15:47, Tom Vilsack wrote:
> Thank you for your reply.
>
> I have installed libxsmm according to commands as described below.
>
> make STATIC=0 BLAS=0
> make PREFIX=<libxsmm install path> STATIC=0 BLAS=0 install
>
> The make process finished without error. Then, I set parameter in
> "backend-openmp" section in ini file.
>
> [backend-openmp]
> cc = icc
> cblas = <libxsmm install path>/lib/libxsmm.so

So libxsmm is not a cblas library (that is MKL). All you need to do is
ensure that libxsmm is somewhere where PyFR can find it. If need be you
can point PyFR to the library by doing

export PYFR_XSMM_LIBRARY_PATH=/path/here/to/lib.so

> Are the setting parameters correct? Xeon Phi 7250 have 68 physical
> cores, and each core can run 4 threads. Therefore, I set
> $OMP_NUM_THREADS=272.

For best performance you want one thread per core.

> Could you tell me meanings of "gimmik-max-nnz", "libxsmm-block-sz" and
> "libxsmm-max-sz" in "backend-openmp" section in ini file?

The first parameter decides the cut-off point at which we will no longer
use the GiMMiK matrix multiplication library. The second parameter
controls the block size for libxsmm, although performance does not
appear to be too sensitive to its value. The final parameter decides
the cut-off point at which we will no longer use libxsmm for matrix
multiplications. The hierarchy is:

1. libxsmm (if available and size < libxsmm-max-sz)
2. GiMMiK (else if nonzeros < gimmik-max-nnz)
3. cblas (else)

The defaults in PyFR are not too bad, but you may be able to get 20-30%
out for certain test cases by playing around with these parameters. The
key is to ensure that libxsmm is available, this is especially important
at lower polynomial orders.

Regards, Freddie.

[Tom Vilsack]

Hi Freddie,

Thank you for the detail information.

I set the environment variable "PYFR_XSMM_LIBRARY_PATH" in .bashrc

  export PYFR_XSMM_LIBRARY_PATH=$HOME/FR/libxmm/lib

In libxmm/lib, there are dynamic libraries like libxsmm*.so.

Then, I set "[backend-openmp]" section in .ini file as described below.

  [backend-openmp]
  cc = icc
  cblas =  /opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin_mic/libmkl_rt.so

I got an error "KeyError: "'mul' has no providers"" when I execute PyFR. The error could be related to wrong cblas library pass. In mkl lib directory, there are many libraries. What dynamic library I should set in cblas section?

  libmkl_ao_worker.so             libmkl_cdft_core.so    libmkl_intel_thread.so     libmkl_sequential.a
  libmkl_blacs_intelmpi_ilp64.a   libmkl_core.a          libmkl_lapack95_ilp64.a    libmkl_sequential.so
  libmkl_blacs_intelmpi_ilp64.so  libmkl_core.so         libmkl_lapack95_lp64.a     libmkl_tbb_thread.a
  libmkl_blacs_intelmpi_lp64.a    libmkl_intel_ilp64.a   libmkl_rt.so               libmkl_tbb_thread.so
  libmkl_blacs_intelmpi_lp64.so   libmkl_intel_ilp64.so  libmkl_scalapack_ilp64.a   locale
  libmkl_blas95_ilp64.a           libmkl_intel_lp64.a    libmkl_scalapack_ilp64.so
  libmkl_blas95_lp64.a            libmkl_intel_lp64.so   libmkl_scalapack_lp64.a
  libmkl_cdft_core.a              libmkl_intel_thread.a  libmkl_scalapack_lp64.so

Best regards,
Tom

[Freddie Witherden]

Hi Tom,

On 07/06/17 14:32, Tom Vilsack wrote:
> Thank you for the detail information.
>
> I set the environment variable "PYFR_XSMM_LIBRARY_PATH" in .bashrc

The path needs to be the absolute path to the library itself as opposed
to the directory in which it is located.

You want libmkl_rt.so. This will resolve the provides problem.

Regards, Freddie.

[Tom Vilsack]

Hi Freddie,

I set the environment variable "PYFR_XSMM_LIBRARY_PATH" in .bashrc

  export PYFR_XSMM_LIBRARY_PATH= <path to home>/FR/libxmm/lib/libxsmm.so

I also changed [backend-openmp] section according to your information.
  [backend-openmp]
  cc = icc
  cblas = /opt/intel/compilers_and_libraries_2017.2.174/linux/mkl/lib/intel64_lin/libmkl_rt.so

It works well! PyFR start to run without error. Thank you!

Best regards,
Tom