How to enable multi-threaded solving

[Bryan W. Weber]

Hello all,

I'm trying to solve a 1-D freely propagating adiabatic flame using cantera 2.1 on Ubuntu 12.04.3. I have one of the more recent Intel compilers installed with Intel MKL. I used the Intel compilers and MKL to compile Cantera with the Boost thread-safe libraries and Sundials. Both Sundials and Boost were compiled with the Intel compiler. I also compiled the flamespeed C++ sample with the Intel compilers and linking to the Intel MKL, but the application only ever uses one thread (core). I have 24 cores, so I tried exporting MKL_NUM_THREADS=12 and MKL_DYNAMIC=FALSE (to force MKL to use multi-threads), and my mechanism is several hundred species and several thousand reactions. How can I enable Cantera to use multiple threads (cores)?

Thanks!

Bryan

[Bryan W. Weber]

I have also tried using the Python interface, with Numpy, Scipy, and the Cantera python interface compiled by the Intel compilers. Again, only one core is used.

Bryan

[Moffat, Harry K]

 

Hi Bryan

 

    Sorry about your trying to get multithreading working. Cantera isn’t coded specifically for multithreading at all. In mp codes, Cantera is used in an embarrassingly parallel mode, where each processor gets its own instantiation of Cantera. Works great. A 1D code with this approach may be available soon as open source using trilinos to distribute and solve the resulting linear systems, depending upon the needs of the project that’s funding it.

 

However, in order to use multiple threads, you need to specifically code towards that goal, which creates its own problems with conditional compilation depending on your toolset. I notice that you are using the intel compilers which have robust support for multithreading. Other compilers don’t have the support. The priority for this type of approach hasn’t been there for Cantera, as it doesn’t support the creation of capabilities which can be used to support new applications.

 

I’ll soon be creating a module for Cantera that will document the speed of calculations as best as it can be done. I’ve done some initial work with trying to use pthreads to speed up Cantera. It wasn’t promising. I’ll put that work in the module.

 

Best wishes,

Harry Moffat

Cantera Head

--
You received this message because you are subscribed to the Google Groups "Cantera User's Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cantera-user...@googlegroups.com.
To post to this group, send email to canter...@googlegroups.com.
Visit this group at http://groups.google.com/group/cantera-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Bryan W. Weber]

Hi Harry,

Thanks for getting back to me. It's too bad that Cantera doesn't support multi-threading, but I understand why it might not be a development priority. Nonetheless, I have some questions about Cantera's architecture that your reply brings to mind. My (admittedly very basic) understanding of the solution procedure is that Cantera "sets up" the problem and "exports" it to the numerical routines in LAPACK and BLAS. Therefore, if LAPACK and BLAS support threading, the parts of the problem that are solved by LAPACK and BLAS should be able to run in multi-threaded mode. As far as I know, the Intel MKL should support multi-threading, so at least portions of the solution should run in multi-threaded mode. It is certainly possible that the time spent in the LAPACK and BLAS routines is minimal and mult-threading is never activated, or that the particular routines used by Cantera don't support multi-threading. Leaving those two points aside, is my understanding correct? And if so, do you have any idea why I wouldn't see multiple threads being used?

Regards,

Bryan

[Ray Speth]

Bryan,

To be precise, Cantera does not use multiple threads anywhere internally, but it is safe to use in multithreaded applications, as long as each thread that accesses Cantera objects (e.g. ThermoPhase, GasKinetics) has its own copies. An example of a code that uses Cantera in this way is my 1D solver, Ember. I would disagree with Harry's statements about compiler support for multithreading. All the major compilers -- GCC, Clang, MSVC, and Intel, support multithreading. Compiler-independent implementations can be written using libraries like the Intel "Thead Building Blocks" library (used by Ember) or Boost Thread.

In the other direction, linking Cantera to a parallel BLAS or LAPACK implementation should also work, and I'm not sure what's going wrong in your case. Cantera's 1D solver ultimately relies on calling LAPACK functions for banded matrices, specifically dgbtrf and dgbtrs (see BandMatrix::factor and BandMatrix::solve). I don't know if the MKL provides parallel implementations of those particular functions. For a system with several hundred species, I would expect the time to factor the Jacobian to be at least as much as the time spent formulating it, which is the step that won't be parallelized.

Regards,

Ray

[Bryan W. Weber]

Ray,

Thanks for the clarification. It turns out the mistake was on my end. I had compiled Sundials using the mkl_sequential
library. When I compiled it with mkl_intel_thread *AND* iomp5, portions of the solution were run in multi-threaded mode. I imagine the other portions were the steps you mentioned about formulating the Jacobian. Multithreading was enabled in both a C++ test and a Python test.

To anyone who comes across this thread in the future, if only mkl_intel_thread is used to build Sundials, Cantera will build fine and (in my case) will pass all of the tests except Cython2 (and since I was building the Python3 module, Cython3). Cython2 just failed with no error message (actually, Python threw a segmentation fault), but Cython3 has the error message "Undefined symbol __kmpc_reduce_nowait" when it tries to import ._cantera. This symbol is found in libiomp5, so -liomp5 *MUST* be included in the libraries when you build Sundials. 

Bryan