Ceres Python Bindings

[Nikolaus Mitchell]

Hey everyone,

I know there has been some discussion in the past regarding having some python bindings. Here is a project I have been hacking on over the past couple of weeks. It uses Pybind11 to wrap a good portion of Ceres.

https://github.com/Edwinem/ceres_python_bindings.

I'll slowly add more functionality to it over the coming weeks. For basic stuff it seems to work quite well.  If people have specific things they want wrapped let me know, and I''ll try to target that featureset.

This is also a post to generate some discussion on if something like this should be added to main Ceres.

Reasons for:

- Get access to a state of the art non linear least squares solver in Python.

- Allows one to unify Python and C++ code on a single solver.

Reasons against:

- Added maintenance costs

- For now: If you want any speed then the cost function needs to be defined in C++. This decreases the ease of use of this library, as the end user is now expected to write their own wrapping code for their cost function. (Note at my work this is not a problem as we codegen Python Wrappers for Cost Functions. However, this is not a tenable solution for other users)

- The main problem with Python defined cost functions is the GIL. As it means running Ceres with multiple threads is useless.

[Sameer Agarwal]

Hi Nikolaus,

This is very interesting! Have you looked at using Jax for doing automatic differentiation and using that to define cost function but also firstorderfunction objects?

https://github.com/google/jax

Sameer

--
You received this message because you are subscribed to the Google Groups "Ceres Solver" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/3f185882-fcd7-44a7-8711-55e3579bf0f6%40googlegroups.com.

[Nikolaus Mitchell]

For now I actually have a basic autodiff example working with Jax. However, it works through Python, and thus the GIL is still a problem I think.

This is actually the direction I was thinking of in regards to improving the CostFunction capability. Somehow define the CostFunction in PyTorch or Jax. Then when you add it to the Ceres problem it detects this special case and directly calls the say PyTorch C++ code without ever entering into Python. Still a very hypothetical idea.

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-...@googlegroups.com.

[Nikolaus Mitchell]

I can add the code for FirstOrderFunction Objects pretty easily, it is similar enough to a CostFunction. The problem lies with GradientProblem as it takes ownership of the FirstOrderFunction, and is responsible for deleting it. As it is setup right now, and generally how Pybind11/Python operates is that anything created in Python is owned by Python. So a double free is going to happen as Python deletes the FirstOrderFunction* and the GradientProblem* also tries to delete it.

[Sameer Agarwal]

you can make ceres::Problem not take ownership of the cost functions, by configuring it using ceres::Problem::Options.

GradientProblem does not have this feature yet, but can easily be added by adding an options struct.

Sameer

To unsubscribe from this group and stop receiving emails from it, send an email to ceres-solver...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ceres-solver/8436cad1-724c-44c3-b94d-42d84112fe1a%40googlegroups.com.

[Nikolaus Demmel]

Hi Nikolaus,

nice name :-).

This would be super useful, in particular if it were possible to define cost functions in python, get autodiff and decent optimization performance.

I recently was looking for an NLSQ optimization library that I can use with Python and that support local parameterizations for on-manifold GN. I found minisam [1], which probably is not as mature and powerful as ceres, but it does support defining residuals in both python and C++. I worked fine for me for a small usecase optimizing over rotations. I used the provided sophuspy binding for SO3. Could possibly be helpful to see how they do things. Interestingly there is also an issue with multithreading an Python [2]. Might be related to what you mentioned.

Best,

Niko

[1] https://github.com/dongjing3309/minisam

[2] https://github.com/dongjing3309/minisam/issues/8

[Nikolaus Mitchell]

Thanks. :)

Yeah I know of minisam. Didn't know though that it can do Python based Costfunctions so I will definitely take a deeper look.

My implementation does support custom cost functions, loss functions, parameterizations etc , essentially all that Ceres has to offer(note only tested cost functions). You can then use some autodiff software with something like JAX to define the jacobians.

Performance via threading is sadly just a fundamental limitation of Python defined functions. In C++ it can be as parallelized as possible, but they all must go through GIL as soon as they touch Python. So it wouldn't surprise me if minisam is suffering from a similar issue. The only way I believe to avoid it is to use something like PyTorch and maybe JAX in the future. Where the computation is defined in Python, but then executed within C++ bypassing any need to touch Python.

That or do everything with multiprocessing instead.