Checking numerical gradients

[David Reichert]

Hi,

Another quick question: is there already a way to check the numerical values of the gradients during training (using SGD, to check that my cost is doing the right thing), or what would be the non-hacky way to implement this?

Cheers,
David

[Ian Goodfellow]

I think theano.gradient.verify_grad does what you want. There's
another version, theano.tests.unittest_tools.verify_grad. I always
forget how they differ from each other.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "pylearn-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pylearn-dev...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[David Reichert]

Hm I think I should have been clearer: in my case I don't actually need to verify that the gradients are correctly derived from a cost, rather I need to check that two gradients computed for two different parameters are the same numerically (a special case for my cost, just to check that my code is the doing right thing). So details aside, the point is that I need to read out the actual numerical values of the gradients and then do something with them (something that isn't verifying the gradient in the sense of verify_grad as far as I can see)...

[Ian Goodfellow]

Ah, just write your own script:

my_grad_1 = theano.tensor.grad(my_cost_1, my_param)
my_grad_2 = theano.tensor.grad(my_cost_2, my_param)
assert numpy.allclose(my_grad_1.eval(), my_grad_2.eval())

On Wed, May 15, 2013 at 4:43 PM, David Reichert

[David Reichert]

yeah ok... I guess I should write some proper dedicated tests anyway (rather than just putting asserts everywhere in the code :) )

[Ian Goodfellow]

A test in time saves nine.

On Wed, May 15, 2013 at 4:57 PM, David Reichert