Pymanopt - manifold optimization in Python, with automatic differentiation!

[Jamie Townsend]

I noticed that a key bottleneck and cause of headaches when using manopt is the derivation of gradients, and particularly of hessians (which are usually fourth order tensors). I realised that this work doesn't need to be done by the human, it can be done automatically, and there are a number of packages which are able to do this (e.g. Theano, Stan, and the recently released Google TensorFlow).

In an effort to ease people's headaches, I've started a project, which can be found here:

https://github.com/j-towns/pymanopt

The idea is to use optimization techniques from manopt, combined with the automatic differentiation power of the Python package Theano. You need to define a smooth cost function using Theano, and then pymanopt will compute the gradient (and hessian if necessary) automatically. Having computed these it then uses them to perform optimization using a solver ported from manopt.

Here is an exampe usage to give you an idea of what this might look like:

import theano.tensor as T
import numpy as np

from pymanopt.solvers.steepest_descent import SteepestDescent
from pymanopt.manifolds.stiefel import Stiefel

# ---------------------------------
# Define cost function using Theano
# ---------------------------------
# Note, your cost function needs to have one (matrix) input and one (scalar) output.
X = T.matrix()

# Cost is the sum of all of the elements of the matix X.
cost = T.sum(X)

# ---------------------------------
# Setup solver and manifold objects
# ---------------------------------
solver = SteepestDescent()
manifold = Stiefel(5,2)

# --------------------
# Perform optimization
# --------------------
# Currently the solve function requires three inputs: the cost and input variable
# (both defined using theano) and the manifold to optimise over.
Xopt = solver.solve(cost, X, manifold)

print Xopt

Currently I have only implemented the Stiefel and Grassmann manifolds and the steepest-descent solver. But I'm planning on porting more of manopt's manifolds and solvers in the coming weeks. I hope the project in its current state will function as an effective proof of concept, and something fun to play around with.

For now, the package forces you to just supply a cost (i.e. you aren't allowed to give it gradients and hessians calculated by hand), my belief is that this will eliminate the possibility of human error in gradient and hessian calculations. However this constraint could be relaxed if people wish.

See the project homepage on github for installation instructions.

You will need to know the basics of numpy (Python's answer to MATLAB) and Theano in order to use this package. A helpful page for MATLAB users wishing to learn numpy can be found at http://docs.scipy.org/doc/numpy-dev/user/numpy-for-matlab-users.html. A tutorial for setting up functions in Theano can be found at http://deeplearning.net/software/theano/tutorial/index.html.

Any feedback and contributions are welcome. Bear in mind it is currently only me working on this project so forgive me if it takes time for me to respond to you.

Jamie

[Nicolas Boumal]

Hello Jamie,

This is a great initiative! Looking forward to see what comes of it :).

At some point, we should include automatic differentiation or symbolic differentiation in Manopt as well.. I agree that this would be a big help to our users.

All the best,

Nicolas