Automatic symbolic matrix differentiation... again ?

[Valentin Z]

Hi everyone,

I have seen an ancient thread on this forum about automatic symbolic
matrix differentiation, to have sympy compute expressions like:

∂(XY) = (∂X)Y + X(∂Y)
d( A' ) = d ( A ) ' (transpose)
etc...

Are there still projects in this direction ? What has been decided
about a symbolic matrix class (where one matrix = one letter ) ?

I just implemented my own matrix differentiator method but it is
something built kind of "beside" the rest of sympy (with its own
printing and latex methods). However it could be easily added to the
package:

http://zulko.wordpress.com/2012/04/15/symbolic-matrix-differentiation-with-sympy/

Hope it helps until an 'official' solution comes to light :)

Cheers,

Valentin

[Aaron Meurer]

Hi.

We do plan to do this. See
http://code.google.com/p/sympy/issues/detail?id=2759. No present work
is being done on it now, though.

We have MatrixSymbol objects, which already implement all the
boilerplate stuff like transposes (it's been implemented since the
last release, so you'll have to use the git master to use it). All
that needs to be done is to properly implement the ._eval_derivative
methods.

If you can help start implementing these rules, that would be great.
Let us know if you need any help with the git workflow. We have an
extensive guide at
https://github.com/sympy/sympy/wiki/development-workflow.

Aaron Meurer

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To post to this group, send email to sy...@googlegroups.com.
> To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/sympy?hl=en.
>

[Matthew Rocklin]

That's really cool work. It would be great to see these sorts of things integrated back into SymPy. I'm not sure any of the other major CASs have anything like this. Definitely check out the MatrixExpr classes in the development branch. This work is new and could really use your contribution.

I suspect that you could easily accomplish this if you put the rules you state in your MATRIX_DIFF_RULES dict into a new  _eval_derivative method in each of the relevant MatrixExper classes (MatAdd, MatMul, Transpose, Inverse) as Aaron suggests.

Do we have a class to represent (∂X)?

I'm happy to help if you have questions about MatrixExprs. 

[Kjetil brinchmann Halvorsen]

see below.

This is cool, and very welcome, but some questions:
--- What is meant exactly above with "matrix derivatives" There are multiple cases: Derivative of matrix as a matrix function of a scalar argument, as matrix funtion of a vector argument, or as matrix function of a matrix argument.  Concentrating on the last case: The matrix derivative can be seen as a collection of tyhe set of all partial derivatives, and this can be packaged into a matrix in various ways. In the following I am following the book by Kollo and von Rosen, "Advanced Multivariate Statistics with Matrices", Springer. They give various forms of "the" Matrix derivative, in two main classes: Organized by Kronecker products or by using the vec operator, and ends up favouring the last.

One definition (the one they favour) is
   \frac{d Y}{d X} = \frac{\partila}{\partial \vec{X}} \vec^T {Y}
  which, when X is a p\times q - matrix, Y a r\times s - matrix, is a (pq)\times (rs) - matrix.

An alternative definition (Neudecker, 1969) is
\frac{d Y}{d X} = (\frac{\partia}{\partial \vec X} )^T \otimes \vec Y

where the \otimes symbol representa a Kronecker product.

An example of a Matrix differentiation result using the first of this definitions, is

\frac{d A^T X}{d X} = I_q \otimes A

where A is a constant matrix.

Which of (all) these definitions should be used in sympy (should it be possible to ask for which def to use?

Kjetil
 

--
"If you want a picture of the future - imagine a boot stamping on the human face - forever."

George Orwell (1984)

[Valentin Z]

Hi,

Thanks Matthew for the support, I'll let you know If I have any
question !


Kjetil : This is a good question. What I describe in my blog is
actually not "derivation", but rather "differentiation". Here is the
difficulty:

If you consider any numerical (non-matrix) function F , then its
"derivative" w.r.t. x, dF (x) /dx , will be the number such that for
a small value of ∂x

F(x+∂x) = F(x) + dF(x)/dx * ∂x + o(∂x)

You see that the ∂x is multiplied "after" the dF(x)/dx.

Now what happens with matrices ? Consider the product X * X, and ∂X
a "small" matrix (i.e. a matrix of small norm).

Then (X +∂X) * (X +∂X) = X∂X + (∂X)X + o(∂X)

Which cannot be written under the form [something]*∂X + o(∂X) as it
was the case for F.


For a function G of a matrix X, and a "small" matrix ∂X (i.e. a matrix
of small norm):

G(X+∂X) = D_G(∂X) + o(∂X)

But here D_G(∂X) is a "function" and ∂X is "inside" an expression.
However it is interesting to find the expression of D_G (these
computations can be the first steps to come to more meaningful
objects, like Jacobians and so on).

So what we would like is something like:
diff( X*X , X ) >>> X∂X + (∂X)X
diff( X*Y , X ) >>> (∂X)Y

diff( X*Y , (X,Y) ) >>> X∂Y + (∂Y)X
or the same (more in sympy's spirit ? )
diff( diff( X*Y , X) ,Y ) >>> X∂Y + (∂Y)X

On 16 avr, 04:06, Kjetil brinchmann Halvorsen <kjetil1...@gmail.com>
wrote:

> see below.
>
>
>
>
>
>
>
>
>
> On Sun, Apr 15, 2012 at 17:44, Matthew Rocklin <mrock...@gmail.com> wrote:
> > That's really cool work. It would be great to see these sorts of things
> > integrated back into SymPy. I'm not sure any of the other major CASs have
> > anything like this. Definitely check out the MatrixExpr classes in the
> > development branch. This work is new and could really use your contribution.
>
> > I suspect that you could easily accomplish this if you put the rules you
> > state in your MATRIX_DIFF_RULES dict into a new  _eval_derivative method in
> > each of the relevant MatrixExper classes (MatAdd, MatMul, Transpose,
> > Inverse) as Aaron suggests.
>
> > Do we have a class to represent (∂X)?
>
> > I'm happy to help if you have questions about MatrixExprs.
>

> >>http://zulko.wordpress.com/2012/04/15/symbolic-matrix-differentiation...

[Alan Bromborsky]

You always start with a directional derivative so that if F(A) is a
matrix function of a matrix define the directional derivative in the
direction B where B is a Matrix (vector in a n**2 dimensional vector
space of matrices) as -

(B*\partial_{A})F = \lim_{h->0} (F(A+h*B)-F(A))/h

so that (B*\partial_{A}) is a scalar operator that is then defined in
any direction B and * in this context is a scalar product of two
matrices. From knowing the directional derivative in any direction the
operator \partial_{A}F can be determined.

See section 11.1 of "Geometric Algebra for Physicists" by Doran and
Lasenby for more details including taking the derivative of a linear
transformation with respect to a linear transformation.

[Alan Bromborsky]

You might also want to look at - ftp.ra.phy.cam.ac.uk/~clifford/publications/ps/lag_field.pdf

[Kjetil brinchmann Halvorsen]

You might also want to look at - ftp.ra.phy.cam.ac.uk/~clifford/publications/ps/lag_field.pdf

Firefox says that link does' nt exist.

Kjetil

--
You received this message because you are subscribed to the Google Groups "sympy" group.
To post to this group, send email to sy...@googlegroups.com.
To unsubscribe from this group, send email to sympy+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sympy?hl=en.

[Alan Bromborsky]

I have attached the paper.

[Matthew Rocklin]

I put up a quick and dirty implementation of this here

https://github.com/sympy/sympy/pull/1275