Constant loss for D and G with GANs

[Rafael Valle]

Hey all,

I'm training a GAN using the traditional generator and discriminator loss functions, being -log(D(G(z)) and -log(D(x)) -log(1 - D(G(z)) respectively.

For some reason the loss for the discriminator and the generator are always constant, with the generator loss being close to 0 and the discriminator loss being proportional to the batch size.

Changing the loss functions to categorical cross-entropy on two labels, real or fake, the network behaves normally.

Does anyone have an idea how to solve this? The code in on the link below.

https://github.com/rafaelvalle/neural_network_control_improvisation/blob/master/CRGAN-MultiStep-PianoRoll.py#L448

[Rafael Valle]

Just realized that I was using softmax as nonlinearity instead of sigmoid.

In this case it would be very useful to have a warning message saying that softmax is being applied to a single unit layer :-)

[Jan Schlüter]

Just realized that I was using softmax as nonlinearity instead of sigmoid.

In this case it would be very useful to have a warning message saying that softmax is being applied to a single unit layer :-)

Good idea. Theano often doesn't know the shape of softmax inputs, but with Lasagne, it can tell whether there's a single unit (because that dimension will be broadcastable).