How do you debug keras models?

[Mustg Oplay]

Even with debug mode in PyCharm I'm not able to see data or shapes or anything useful.

How do people debug?

Thanks

[Isaac Gerg]

I use wingIDE, best I've ever used.

What do you mean by seeing the data or shapes? Can you be more specific?

I also like to use quiver_engine and matplotlib.

[Mustg Oplay]

Thank you for replying.

Sure.  The vast majority of my time is trying to fix shape errors and trying to figure out if calcs are right.  It's very frustrating.  For example, right now I'm working on a custom layer and it trains well if the batch size is one.  That in and of itself was a big accomplishment as I had to randomly work my way through a bunch of shape errors.  If I increase the batch size it still give a similar error to the previous errors:

ValueError: total size of new array must be unchanged
Apply node that caused the error: Reshape{4}(Elemwise{true_div,no_inplace}.0, TensorConstant{[32  1 24 24]})
Toposort index: 160
Inputs types: [TensorType(float32, matrix), TensorType(int32, vector)]
Inputs shapes: [(32, 14400), (4,)]
Inputs strides: [(57600, 4), (4,)]
Inputs values: ['not shown', array([32,  1, 24, 24])]
Outputs clients: [[DimShuffle{1,0,2,3}(Reshape{4}.0)]]

Knowing this I try to get the batch_size from the input shape, K.get_shape(x) and you can't index off that - i.e. can't do K.shape(x)[0] and you can't use K.get_value(K.shape(x)).  If I manually input the batch size, then it works when training but it turns out keras is automatically using a batch_size of 50 during testing.  Then when I try and run values through to try and visualize the layer output, it's using a batch size of 32 so always using a batch size of 50 or 32 doesn't work.

So in this particular case I can't manipulate the shape the way I need to.

In other situations, when I'm in debugging mode I can see the tensor variables but I can't see the shape of the tensor.  It just shows the function Shape.  It seems like a significant portion of the questions on these boards are about shape mismatches.  It'd be nice to know if there was a way to just look at it.  You can't just print it because it only goes through the layer code when compiling and even then you can't get_value of the K.shape.

I found a trick that works sometimes but not all the time and that's to purposely put in a variable assignment to a nebis2images because the error shows as above to give the mismatch.

I have the theanorc flags for debugging on (per the debugging section of theano site) and that helps a little but not always:

exception_verbosity=high
traceback.limit=N
#optimizer=fast_compile
optimizer=None

The other problem is that I can't see the data so anytime I want to see if my calculations are correct I have to do it in numpy, then try to convert it and pray it's right.

I understand that keras is a wrapper but if you can get data out to save it I don't understand why you can't get data out to look at it.  If you have suggestions on how to see the values in the tensors as well please that would be appreciated.

Sorry for letting my frustration seep through.

Thanks

[yuv...@gmail.com]

With tensorflow backend, you can inspect the output of a layer activation using session.run()
For example:
Inspecting the output of layer[2]:

sess = K.get_session()

sess.run([model.layers[2].output], feed_dict={'input_1:0':X_val})

Where  'input_1:0' is the placeholder for the model inputs

[ro...@cgl.ucsf.edu]

Keras seems like it has a long way to go in usability for novices.