Multithreading alternatives to OpenMP

[Chris Sweeney]

For anyone using Clang (majority of Mac users probably) you'll know that OpenMP is not supported without a custom version of Clang so sadly clang users cannot utilize the multithreading of Ceres easily. Has there been any thought into moving away from OpenMP for an alternative multithreading library? It seems that OpenMP functionality is mostly some simple "#pragma parallel for" instructions, so it would not be terribly difficult to replace.

One candidate would be TinyThread++. This gives nearly all of the C++11-like threads features so one could probably write a thread pool or a parallel_for function quite easily using those tools. A thread pool class may actually be quite useful for other parts of Ceres as well.

I am a bit busy at the moment, but at some point in the future I could probably help lead the refactoring.

Chris

[Sameer Agarwal]

Hi Chris,

This has come up a couple of times in the past. Yes lack of threading on macos is a pain.

We have thought about moving away from openmp, but the resulting increase in code complexity (the pragmas are really quite simple) and the expected integration of openmp in clang has always kept us away from going down this path.

I'd be curious to see a patch (say for the evaluator) which shows the impact of such a change. Also if this mostly targeting clang, why not just use c++11 ? why introduce a dependency?

I am sure Keir and Alex have something to say about this too. 

Sameer

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9456fc38-a83d-4792-80cf-fbe9284d58bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Sweeney]

On Sun, Mar 1, 2015 at 2:32 PM 'Sameer Agarwal' via Ceres Solver <ceres-...@googlegroups.com> wrote:

Hi Chris,

This has come up a couple of times in the past. Yes lack of threading on macos is a pain.

We have thought about moving away from openmp, but the resulting increase in code complexity (the pragmas are really quite simple) and the expected integration of openmp in clang has always kept us away from going down this path.

I hear you, but I have found myself waiting for openmp to be integrated into clang for several years now, always hoping that the next one will support it only to be disappointed. Further, Apple's clang is always a few steps behind the open source clang so it is likely that the default mac compiler will not have openmp for quite some time :(

 

I'd be curious to see a patch (say for the evaluator) which shows the impact of such a change.

I can probably make an example patch that would demonstrate how the code would change. I'll work on that.

 

Also if this mostly targeting clang, why not just use c++11 ?

Currently ceres does not require c++11... Supporting c++11 threads with OpenMP would really muck up the code. 

 

why introduce a dependency?

It keeps Ceres from having to support yet another compiler configuration, for one. You could even include TinyThreads++ (or library of choice) in Ceres the way that gtest and gmock are. That shouldn't be too difficult. It would remove the dependency on OpenMP entirely.

 

I am sure Keir and Alex have something to say about this too. 

Sameer

On Sun, Mar 1, 2015 at 2:22 AM, Chris Sweeney <kip...@gmail.com> wrote:

For anyone using Clang (majority of Mac users probably) you'll know that OpenMP is not supported without a custom version of Clang so sadly clang users cannot utilize the multithreading of Ceres easily. Has there been any thought into moving away from OpenMP for an alternative multithreading library? It seems that OpenMP functionality is mostly some simple "#pragma parallel for" instructions, so it would not be terribly difficult to replace.

One candidate would be TinyThread++. This gives nearly all of the C++11-like threads features so one could probably write a thread pool or a parallel_for function quite easily using those tools. A thread pool class may actually be quite useful for other parts of Ceres as well.

I am a bit busy at the moment, but at some point in the future I could probably help lead the refactoring.

Chris

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9456fc38-a83d-4792-80cf-fbe9284d58bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABqdRUAMyZZO4Ch%2BnCsO2-Qy-RpKoOZqv6Xit6ubkjtfCGmDhg%40mail.gmail.com.

[Sameer Agarwal]

I hear you, but I have found myself waiting for openmp to be integrated into clang for several years now, always hoping that the next one will support it only to be disappointed. Further, Apple's clang is always a few steps behind the open source clang so it is likely that the default mac compiler will not have openmp for quite some time :(

It is not that far :) 

http://llvm.org/releases/3.6.0/tools/clang/docs/ReleaseNotes.html

3.7 is when it should happen.

 

I'd be curious to see a patch (say for the evaluator) which shows the impact of such a change.

I can probably make an example patch that would demonstrate how the code would change. I'll work on that.

SGTM.

 

Also if this mostly targeting clang, why not just use c++11 ?

Currently ceres does not require c++11... Supporting c++11 threads with OpenMP would really muck up the code. 

on android/ndk it does, as that is the only way you get std::shared_ptr.

but I understand the complication. That said, I am trying to convince Keir about moving to c++11, maybe in the 1.12 timeframe.

Sameer

 

why introduce a dependency?

It keeps Ceres from having to support yet another compiler configuration, for one. You could even include TinyThreads++ (or library of choice) in Ceres the way that gtest and gmock are. That shouldn't be too difficult. It would remove the dependency on OpenMP entirely.

 

 

 

I am sure Keir and Alex have something to say about this too. 

Sameer

On Sun, Mar 1, 2015 at 2:22 AM, Chris Sweeney <kip...@gmail.com> wrote:

For anyone using Clang (majority of Mac users probably) you'll know that OpenMP is not supported without a custom version of Clang so sadly clang users cannot utilize the multithreading of Ceres easily. Has there been any thought into moving away from OpenMP for an alternative multithreading library? It seems that OpenMP functionality is mostly some simple "#pragma parallel for" instructions, so it would not be terribly difficult to replace.

One candidate would be TinyThread++. This gives nearly all of the C++11-like threads features so one could probably write a thread pool or a parallel_for function quite easily using those tools. A thread pool class may actually be quite useful for other parts of Ceres as well.

I am a bit busy at the moment, but at some point in the future I could probably help lead the refactoring.

Chris

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9456fc38-a83d-4792-80cf-fbe9284d58bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABqdRUAMyZZO4Ch%2BnCsO2-Qy-RpKoOZqv6Xit6ubkjtfCGmDhg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O7OvSkQ0Y%3DH7q3oR6m%2BcbMs1us0RmXn6ab5F1JML%3DyhAw%40mail.gmail.com.

[Sameer Agarwal]

Chris,

Keir and I talked offline. 

We would like to pursue this avenue seriously.

How soon will you be able to produce an example of TinyThread++ usage in ceres?  

I would recommend threading program_evaluator.h using it. 

Sameer

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAK0oyErdwzQpv9rUVHFb%3D_9_itj%2Ba%3D89GbLFQ2xmEZpsYBTfiw%40mail.gmail.com.

[Chris Sweeney]

Hi Sameer,

Just took a look at program_evaluator.h. I don't think it should be too difficult to get it working with TinyThreads++ (which has basically the same API as c++11 threads). I'll give it a crack and send you guys a patch this weekend.

Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CABqdRUCLE--1Bueq%2Bu0czJvTZOQ_qkOzMKSSrrAssgizHXvc7w%40mail.gmail.com.

[Sameer Agarwal]

Fantastic.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O7DYi8C8DD9%2BF8%2BQ5NGh%2BEORTxyJDLcb0vcb%2BJLFLYM2A%40mail.gmail.com.

[Keir Mierle]

Nice! This would be great.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O7DYi8C8DD9%2BF8%2BQ5NGh%2BEORTxyJDLcb0vcb%2BJLFLYM2A%40mail.gmail.com.

[Chris Sweeney]

Just wanted to follow up on this thread.

I talked to Sameer and Keir a bit ago and it seems that it would be difficult to implement a pre-c++11 multithreading module that would not add significant overhead to the code. If anybody has a compact, cross-platform solution for thread pooling or parallel for loops please feel free to post it here.

One thing I would suggest is that Ceres disables the CHECK-fails for multithreading. Currently the program evaluator in Ceres will crash if OpenMP is not detected but the number of threads requested > 1. This means that projects that use Ceres are then responsible for also detecting OpenMP and disabling threads to Ceres if no OpenMP was detected. For projects that use multithreading via other libraries (e.g., c++11, TBB, etc.) it seems unnatural to have to look for OpenMP when the project itself may not use it. If these CHECK-fails are disabled then perhaps a very obvious logging message (or output during CMake) can inform the user that only 1 thread is being used.

-Chris

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CADpYijH3mQdQG4TMHSRvA2r4vPskVO%2Bk-2rVFpB12Cni0PJ6fw%40mail.gmail.com.

[Sameer Agarwal]

Thanks chris.
I am okay making the checks less severe. Send me a patch that does the needful and I am happy to review.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O6CxPO5q6TrEPOSsj1XdiJDkd9a9vPMnwbLBwtgQatuKw%40mail.gmail.com.

[Chris Sweeney]

Another follow-up to this thread. I was able to successfully and relatively painlessly install Clang-OMP using homebrew for Mac OS X based on these formulas. Here are the needed commands

brew tap marcoesposito1988/homebrew-openmp

brew install --HEAD llvm-omp

brew install openmp-rt

This will install a version of clang with openmp as clang-omp, so it does not replace/overwrite the system clang. One slightly funky thing I had to do was add "-lc++" as a command line argument for basically anything using clang-omp. But this allows Ceres multithreading to work with clang with a slight modification of the CMakeLists.txt file.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/9456fc38-a83d-4792-80cf-fbe9284d58bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.

[Keir Mierle]

That's great! I have a thread pool implementation on the way but it is not there yet :(

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/07e56f9d-d68b-4b3e-b564-eb4236c8dbb3%40googlegroups.com.

[Johannes Schönberger]

I am not sure if this is still open for discussion, but OpenMP support is finally coming with Clang/LLVM 3.7. See http://lists.cs.uiuc.edu/pipermail/cfe-dev/2015-May/042845.html.

Depending on when 3.7 will be released, we should see OpenMP relatively soon on OSX, since Apple is usually quick about integrating the newest versions.

[Sameer Agarwal]

Thanks for the heads up Johannes.

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/cdc2a78a-63d3-4990-85dc-e268f07546d3%40googlegroups.com.

[Keir Mierle]

Thanks for this Johannes! Unfortunately it will be a long time before this trickles down across platforms. I'll be working on the threading implementation in Ceres starting sometime next week.

--

You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.

[Chris Sweeney]

Just to follow up here, OpenMP has been fully integrated into newly-released LLVM 3.7. Who knows how long it will be before Apple incorporates it into their Clang, but this is progress in the right direction!

http://clang-omp.github.io/

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CADpYijE0bw79jt%2BmHjrpZns32UtV8zaAPRQj1LoPYVexYZfkXg%40mail.gmail.com.

[Sameer Agarwal]

woohoo! I look forward to adding more ifdefs around clang versions in the ceres code :)

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/CAB_h4O6VfyMNjOijRLCevvYVRQFkhRDpsNtKX2BUiu_Ef6UTwg%40mail.gmail.com.