New GANs example with Julia: request feedback
209 views
Skip to first unread message
Manuel Morales
Dec 12, 2017, 5:21:41 PM12/12/17
to knet-users
Hi,
I am trying to replicate the Generative Adversarial Network (GAN) example found here with Julia. I think I manged to make it work (see here), but I was wondering if I could get some feedback on one issue. I've read in a few post about the dangers training two networks at once, such as longer training times due to gradients with respect to another network or unknowingly updating the weights of the other network. On the original example in MXNet they use a function called .detach() to remove a variable from a graph (to prevent the errors previously mentioned). It seems to me that this is unnecessary with knet since Grad only takes gradients with respect to the first input.
Could anyone take a look at my code (see here) and comment on any improvements or if my assumption is correct?
Thanks!
I am trying to replicate the Generative Adversarial Network (GAN) example found here with Julia. I think I manged to make it work (see here), but I was wondering if I could get some feedback on one issue. I've read in a few post about the dangers training two networks at once, such as longer training times due to gradients with respect to another network or unknowingly updating the weights of the other network. On the original example in MXNet they use a function called .detach() to remove a variable from a graph (to prevent the errors previously mentioned). It seems to me that this is unnecessary with knet since Grad only takes gradients with respect to the first input.
Could anyone take a look at my code (see here) and comment on any improvements or if my assumption is correct?
Thanks!
Manuel Morales
Dec 12, 2017, 6:10:37 PM12/12/17
to knet-users
This is a condense version of the same code to avoid jargon.
ikes...@ku.edu.tr
Dec 14, 2017, 11:15:44 AM12/14/17
to knet-users
Hi Manuel,
Your assumption is correct. In one training iteration, people update discriminator weights (gradients obtained by using discriminator loss) first, then update generator weights (by using generator loss). The training process in your code and their code are identical to each other and this is how it is defined in the original paper. I don't know if there exists another training strategy. I've also implemented a vanilla GAN network on MNIST data [0], you can take a look at my implementation (see this procedure for training [1], also noe that this code needs to be ported to Julia v0.6). It works on MNIST but they are all MLPs not conv/deconv networks. What's more, you can look through this beautiful generative models collection repository [2] which contains many different VAE and GAN models in TensorFlow and PyTorch (code is very clean!).
[0] https://github.com/ilkerkesen/adversarial
[1] https://github.com/ilkerkesen/adversarial/blob/master/vanilla-gan.jl#L180
[2] https://github.com/wiseodd/generative-models
Your assumption is correct. In one training iteration, people update discriminator weights (gradients obtained by using discriminator loss) first, then update generator weights (by using generator loss). The training process in your code and their code are identical to each other and this is how it is defined in the original paper. I don't know if there exists another training strategy. I've also implemented a vanilla GAN network on MNIST data [0], you can take a look at my implementation (see this procedure for training [1], also noe that this code needs to be ported to Julia v0.6). It works on MNIST but they are all MLPs not conv/deconv networks. What's more, you can look through this beautiful generative models collection repository [2] which contains many different VAE and GAN models in TensorFlow and PyTorch (code is very clean!).
[0] https://github.com/ilkerkesen/adversarial
[1] https://github.com/ilkerkesen/adversarial/blob/master/vanilla-gan.jl#L180
[2] https://github.com/wiseodd/generative-models
Reply all
Reply to author
Forward
0 new messages