Trilinos Amesos Parallel Direct Solver

[Chad Balen]

Hey Guys,

Since I last used the mailing list, we got our software to use PETScWrappers::SparseDirectMUMPS Class for solving our problems in parallel. However, installing deal.ii with PETSc can be difficult as pointed out in your FAQ (and I'm having that issue on some people's computer currently), so we started to look at possibly using Trilinos instead. We noticed that one of Trilinos' packages, Amesos, has the capabilities of using MUMPS as well as its own parallel direct solver Paraklete. When I was looking at the deal.ii classes I do not see any specific class for MUMPS through Trilinos nor anything mentioning Paraklete. I did find the TrilinosWrappers::SolverDirect Class which is for Trilinos KLU direct solver (using the Amesos package) but that one is a serial solver. I also found TrilinosWrappers::PreconditionAMG::AdditionalData Struct but that just lists most of the Amesos solvers (except Paraklete).

So I was just wanting to make sure before I tried to get Trilinos working with MUMPS is this possible with deal.II? Or more preferably so I don't have to install MUMPS is it possible to use Paraklete with deal.ii? If any of this is possible which class am I going to have to use in deal.II?

Thanks for the help,
Chad

[Martin Kronbichler]

Dear Chad,

> I did find the TrilinosWrappers::SolverDirect Class which is for
> Trilinos KLU direct solver (using the Amesos package) but that one is
> a serial solver.

You are right that the interface to Amesos does not currently support
other solvers than KLU (you really should go over SolverDirect, not
PreconditionAMG). However, it should be easy to add support for other
solvers.

Would you mind trying to add a new "std::string" variable to encode the
solver type? That would be great. This data should then by passed to the
Amesos().Create() method found in source/lac/trilinos_solver.cc, line
440 (current development version). The default should be "Amesos_Klu" to
maintain backward compatibility. With these options, you should be able
to access any solver you configured Amesos with. (E.g. for MUMPS or
Superludist, Trilinos must be configured properly with these third-party
libraries.)

Best,
Martin

[Wolfgang Bangerth]

Any patch in this direction would indeed be fantastic. For the record, there
is already an enhancement request with a couple of links:
https://code.google.com/p/dealii/issues/detail?id=217

Best
W.

--
------------------------------------------------------------------------
Wolfgang Bangerth email: bang...@math.tamu.edu
www: http://www.math.tamu.edu/~bangerth/

[Uwe Köcher]

Chad,

the main problem for you should be to get Trilinos/Amesos to work with your solver.

For the deal.II support, I've added this functionality for you by giving the TrilinosWrappers::SolverDirect::AdditionalData 

a field std::string solver_type.

The current svn diff is submitted to Issue 21 (see the link from Wolfgang above).

If you like to try it instantly, download the diff from there and recompile deal.II.

Best

  Uwe

[Chad Balen]

Sorry for the long wait, must have messed up when I posted because your responses are not linking to my email. Thank you for the suggestions good to know that it should be possible I'll have to try that and see how it goes. I'm interested to see how the Paraklete compares to MUMPS.

Best,
Chad

[Uwe Köcher]

Hi Chad,

the patch for the TrilinosWrappers::SolverDirect is checked into the development version of deal.II today.

Also the documentation (hints for using third party libs with trilinos) was updated:

http://www.dealii.org/developer/external-libs/trilinos.html

Best

  Uwe

[Uwe Köcher]

Chad,

I have some additional hints for you.

- MUMPS (system package from fedora20 against openmpi) and the cmake settings for Trilinos the SolverDirect now works

- The same for Scalapack

- SuperLU_DIST is a litte more work since Trilinos 11.8.1 has to be patched for the current version 3.3.,

I've updated my deal.II installer (git clone https://github.com/koecher/candi)

Unfortunately, I do not get the Paraklete solver to compile (Trilinos cmake: " -D Amesos_ENABLE_PARAKLETE "),

since I get a lot of compilation errors (with gcc-4.9/openmpi 1.6.5) from the current Trilinos 11.8.1

For this, an older compiler and/or Trilinos version is maybe required; if you want to use it, you should file a bugreport

to the Trilinos guys).

[Chad Balen]

Hi Uwe,

I think I might try to stay away from MUMPS if I can for now (bad memories). I have a few questions regarding my Trilinos Installation I was hoping to get your opinion on:

i) I managed to get SuperLU and SuperLUDist (v2.5) to install however when Trilinos uses SuperLUDist (v2.5) one of the tests failed:

408 - Stratimikos_test_amesos_thyra_driver_MPI_1 (Failed)

This doesn't occur when I just have SuperLU. Do you get this as well, and do you think it will possibly cause future issues for SuperLUDist (the Amesos_Test_SuperLU_Dist_MPI_4 passed fine), or should I be fine?

ii) I tried SuperLUDist v3.3 but Trilinos gives the error:

Amesos_Superludist.cpp:477:82: error: ‘EXTRA’ was not declared in this scope

I take it that this is what you were talking about when you said, "SuperLU_DIST is a little more work". Did you manage to get Trilinos to work with v3.3?

iii) You mentioned Trilinos v11.8.1, but deal.ii recommends to only use v11.4.1. Have you had any issues with v11.8.1?


As well, I also get issues with Paraklete, specifically:

amesos_t_cholmod_dense.c:23:1: error: unknown type name ‘cholmod_dense’
amesos_t_cholmod_dense.c:26:5: error: unknown type name ‘cholmod_sparse’
amesos_t_cholmod_dense.c:28:5: error: unknown type name ‘cholmod_common’
amesos_t_cholmod_dense.c:128:1: error: unknown type name ‘cholmod_sparse’
amesos_t_cholmod_dense.c:131:5: error: unknown type name ‘cholmod_dense’
amesos_t_cholmod_dense.c:134:5: error: unknown type name ‘cholmod_common’
amesos_t_cholmod_dense.c:225:5: error: unknown type name ‘cholmod_dense’
amesos_t_cholmod_dense.c:227:5: error: unknown type name ‘cholmod_dense’

I might send them a bug report, but I'm hesitant with how between all the information on Amesos and Amesos2 it looks like Amesos is not being maintained and will probably be replaced with Amesos2 (which does not include Paraklete).

Thanks for the help,
Chad

[Timo Heister]

> I might send them a bug report, but I'm hesitant with how between all the
> information on Amesos and Amesos2 it looks like Amesos is not being
> maintained and will probably be replaced with Amesos2 (which does not
> include Paraklete).

Amesos seems pretty dead to me too. I tried about a year ago and even
wrote to the developers, but nothing ever came out of it:

./ The Amesos help page still listed autoconf configuration options
and I couldn't figure out a way to enable support for superlu/mumps
with cmake
./ I was also unable to even configure amesos2 back then.

--
Timo Heister
http://www.math.clemson.edu/~heister/

[Wolfgang Bangerth]

> ii) I tried SuperLUDist v3.3 but Trilinos gives the error:
>
> Amesos_Superludist.cpp:477:82: error: ‘EXTRA’ was not declared in this scope
>
> I take it that this is what you were talking about when you said,
> "SuperLU_DIST is a little more work". Did you manage to get Trilinos to work
> with v3.3?

I think you should open a bug report with trilinos.sandia.gov (link at the
bottom of that page). They may not want to further develop Amesos, but they
may still be interested in keeping it working with current packages as a fallback.

> iii) You mentioned Trilinos v11.8.1, but deal.ii recommends to only use
> v11.4.1. Have you had any issues with v11.8.1?

Martin Kronbichler may remember the exact details, but we have found bugs in
pretty much every version of Trilinos. We typically try to work around them,
sometimes to the detriment of performance. I think you will probably be fine
with 11.8.

> As well, I also get issues with Paraklete, specifically:
>
> amesos_t_cholmod_dense.c:23:1: error: unknown type name ‘cholmod_dense’
> amesos_t_cholmod_dense.c:26:5: error: unknown type name ‘cholmod_sparse’
> amesos_t_cholmod_dense.c:28:5: error: unknown type name ‘cholmod_common’
> amesos_t_cholmod_dense.c:128:1: error: unknown type name ‘cholmod_sparse’
> amesos_t_cholmod_dense.c:131:5: error: unknown type name ‘cholmod_dense’
> amesos_t_cholmod_dense.c:134:5: error: unknown type name ‘cholmod_common’
> amesos_t_cholmod_dense.c:225:5: error: unknown type name ‘cholmod_dense’
> amesos_t_cholmod_dense.c:227:5: error: unknown type name ‘cholmod_dense’

As above.

[Uwe Köcher]

Hi Chad,

the problem with SuperLU_DIST v3.3 is known to me; I've wrote a patch, which is automatically applied when you use my installer.

I did not tested the older versions, since they are out of build.

MUMPS was pretty easy to include for me, I've installed the packages (in Fedora20) for MUMPS compiled against the openmpi

compiler provided also by Fedora and set the TPL_ENABLE_MUMPS, as it is now in the description for the third party libs for

Trilinos (see Developer Documentation/Install Notes)

The Paraklete errors looks the same for me, I've tried to patch them as well, but failed after an hour (you patch a error and get another one).

I do not have any issues in using Trilinos 11.8.1 for my parallel codes.

I do not use the testsuite for Trilinos, maybe I find the time next week to test this.

Today, I've read a little in the Amesos2 documentation; looks as dead as Amesos, the C++ interface does not give a useful extension to the Amesos package for me,

so I decided not to use/tryout Amesos2. Also, as you mentioned, Amesos2 seems to support less TPL solvers.

Best

  Uwe

[Uwe Köcher]

Timo,

On Friday, 11 July 2014 19:34:12 UTC+2, Timo Heister wrote:

Amesos seems pretty dead to me too. I tried about a year ago and even
wrote to the developers, but nothing ever came out of it:

./ The Amesos help page still listed autoconf configuration options
and I couldn't figure out a way to enable support for superlu/mumps
with cmake
./ I was also unable to even configure amesos2 back then.

yes, looks pretty bad to me to, but it works so far.

To find the cmake options to enable something, I use "cmake-gui ."

in the build directory (after a first compile), then I can simply search for the enable configuring options.

Then, if Trilions has trouble to find headers and or libs, it will show you a message in the next run of cmake.

Not nice, but it works.

 

Best

  Uwe