Re: [ceres-solver] Re: speeding Ceres up

[Sameer Agarwal]

Brent,

Can you tell us more about the problem you are trying to solve. What are its dimensions? where is the time being spent? The output of GradientProblemSolver::Summary::FullReport() would also be useful.

Sameer

On Sun, Jan 21, 2018 at 3:49 PM Brent Morgan <brent.tay...@gmail.com> wrote:

(it seems as though the gradient_problem_solver.h doesn't have any thread options like other parts of Ceres code)

Best,

Brent

On Sunday, January 21, 2018 at 5:31:09 AM UTC-7, Brent Morgan wrote:

Hello,

I'm using the general minimization solver in Ceres.  It's great; I'm wondering how to speed it up, however.  I have the capability to daisy-chain multiple CPU's together- will this benefit me with the capability of Ceres?  If not, is there any other way I can speed it up using hardware and the proper configurations?  I tried browsing the documentation but couldn't find anything...  Will keep trying, thank you.

Best,

Brent

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/a8130043-827a-42ab-9ca0-4c35e871066e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sameer Agarwal]

Brent,

It would really help if you shared the output of the FullReport().

That said, if I am understanding your problem correctly, you are solving a large non-linear regression of some sort for 12 parameters?

From Ceres' perspective, thats relatively small problem and very little if any of the 380 seconds should be spent inside Ceres proper. Given the size of your data, almost all of your time is being spent in the function and gradient evaluation. This should be evident from the output of FullReport().

You should thread/optimize your function and gradient evaluation any which you like, it has no bearing upon Ceres.

As for dumping out the parameters, there is a bug open for this.

https://github.com/ceres-solver/ceres-solver/issues/246 

That I have not had a chance to work on.

Sameer

On Wed, Jan 24, 2018 at 12:45 PM Brent Morgan <brent.tay...@gmail.com> wrote:

Hi Sameer,

This optimizaiton problem is based on data I'm inputting into it- for 12 parameters to solve, I'm inserting 12 columns of data from text files, each column having ~200,000 data points. Obviously I read in my data before the solve() call so the reading the data part isn't an issue for time consumption.  But there are loops in my objective function and gradient...  There's no way around the loops- but it is efficient as possible.  For 12 parameters currently it takes about ~380 seconds to meet convergence criteria, from the FullReport().  This is OK for now but when I add many many many more columns, I'm just curious on how I can utilize some hardware to boost speed...  I imagine Ceres is dependent on CPU so I can cluster a bunch of CPU's together....

Question #2: is there a way to output the parameter solution into a .txt file for every iteration of the solver?  I'll keep looking into this as well.

Best,

Brent

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/113c847c-5cf8-437e-99ae-d36b5c71c428%40googlegroups.com.

[Sameer Agarwal]

Hi Brent,

as can be seen from the report:

Time (in seconds):

  Cost evaluation                      0.0000 (0)
  Gradient evaluation                337.1690 (4169)
  Polynomial minimization              0.0397
Total                                338.4850

All the time is being spent in the gradient evaluation. OpenMP is usually the first thing I would try to parallelize your evaluation.  But the exact details would depend on how your objective function is structured.

Sameer

On Wed, Jan 24, 2018 at 1:40 PM Brent Morgan <brent.tay...@gmail.com> wrote:

Hi Sameer,

Your understanding of the problem is correct; 12 parameters for now, but I'm preparing for many many more.  Attached is an example of my experiment I mentioned in earlier message, perhaps it's just my cost/gradient functions that are inefficient here.

You should thread/optimize your function and gradient evaluation any which you like, it has no bearing upon Ceres.

Yes!  How are people "threading" their function/gradient?  Any direction for me to figure this out would significantly help my speed, I think.  

Ceres is amazing, thank you.

Best,

Brent

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/79795735-b6e3-44eb-b5b7-c9bd882bf0af%40googlegroups.com.

[Sameer Agarwal]

Sounds great!

Sameer

On Wed, Jan 24, 2018 at 8:37 PM Brent Morgan <brent.tay...@gmail.com> wrote:

Hi Sameer,

I used OpenMP commands for my loops (simple "#pragma omp for" statement above loops that allow independent iteration calculations) and the time went down to ~20 seconds, a factor of ~20x improvement in speed.

When it comes to printing the solution for every iteration to a text file, I can add a this within Evaluate(), but will this add too much delay of writing to the file every call of Evaluate()?  I'll try to avoid opening/closing for every call of Evaluate().  Thank you,

Best,

Brent

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/34f9a6ad-20e8-4bbc-9562-91cd1d9815de%40googlegroups.com.

[Sameer Agarwal]

No there is most likely a bug in your gradient calculation.

On Thu, Jan 25, 2018 at 11:19 AM Brent Morgan <brent.tay...@gmail.com> wrote:

Hi Sameer,

Actually I caused a bug which made it faster; I resolved.

When I try using #pragma omp parallel { #pragma omp for      on top of my for loops, it gives incorrect answers....  Is it possible it's clashing with threads already embedded in Ceres?

Best,

Brent

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/e5fdff8a-ce0d-4529-9d50-507cb1b13688%40googlegroups.com.

[Sameer Agarwal]

Brent,

You can control the convergence/termination by changing the tolerance parameters. Have a look at the three tolerances described here

http://ceres-solver.org/gradient_solver.html#_CPPv2N21GradientProblemSolver7Options18function_toleranceE 

Sameer

On Fri, Jan 26, 2018 at 8:47 AM Brent Morgan <brent.tay...@gmail.com> wrote:

Hi Sameer,

I was able to speed this up a bit, but I noticed that it spent ~90%+ on the last ~200 iterations when the optimal solution was already found, it just tried to optimize it one iota further.  So since I just saw that the logging option for outputting the parameters after every iteration was submitted/closed/solved this morning, I will use that to help me grab my solution faster in the event it gets focused on the very last few iterations to make it most optimal (which may not be required for my project).

Thank you!!!!!!!!!!!!!!!!!!!!!!!!!  I have no further questions on this, Ceres is awesome.

Best,

Brent

To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/48e90c38-8a83-4edd-843e-d40fd0a0d273%40googlegroups.com.