GAN Showcase Project using TensorFlow.js

[Yingtao Tian]

Hey everybody,

I would like to share https://alantian.net/ganshowcase/ , my weekend project which is a showcase of a deep GAN (Generative Adversarial Network) that generates (or dreams) high resolution (256x256) photo realistic images, running completely on browser thanks to TensorFlow.js . 

Technically, the network architecture is similar to the residual network (ResNet) based generator (Gulrajani et al.), as well as the classical DCGAN generator (Radford et al.) and the GAN training uses DRAGAN (Kodali et al.) style gradient penalty for better stability. Training code is written in Chainer. The trained model is then manually converted to a Keras model, which in turn is converted to a web-runnable TensorFlow.js model. The dataset used for training is CelebAHQ, an dataset for Karras et al. which can be obtained by consulting its github repo. 

(More info follows in the project page)

Note that one issue is that the model files are large (good model occupies ~200+MB) due to (1) ResNet-like architecture and (2) use of Conv2DTranspose with manually constructed weights to mimic unpooling (not supported in TensorFlow.js yet), both leads to large (de)convolution in the network, especially in early layers. As Nikhil Thorat suggests, using Separable (De)convolutions like ones in MobileNet may mitigate this issue when supports are available.

Feedback are welcome!

[Shanqing Cai]

Very cool GAN demo, Yingtao! Thanks for sharing. 

[Daniel Smilkov]

This is amazing. Btw the demo gives 404.

2) use of Conv2DTranspose with manually constructed weights to mimic unpooling (not supported in TensorFlow.js yet),

Looks like TF Python doesn't have unpooling yet also. Unpooling seems to be the gradient of maxPool, which we have implemented, but haven't exposed it yet to the public API. Instead of unpooling, can you try using bilinearResize , after the conv2d, to upscale the input.

 

As Nikhil Thorat suggests, using Separable (De)convolutions like ones in MobileNet may mitigate this issue when supports are available.

We already have separable conv  in tfjs. Try it out!

Cheers!

Daniel

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/04333ad6-ef73-4a4e-bb91-5fc0b80d63e7%40tensorflow.org.

[Stan Bileschi]

So excellent!  Thanks for sharing!

[Loreto Parisi]

Thanks for sharing I have tried the tool with both the GAN 64 and ResNet 128 and 256.
Few questions:
- Why did you use Chainer's GAN?
- Are the model's chunks i.e. the files named shards the only way to handle these files i.e. multiple files rather than a single file (or is the latter possible?)

Some other concerns about TFJS and security to the TF JS dev team. In few steps it's easy to download the models files and reproduce the model locally. Now, supposed that a TFJS model running in the browser is copyright protected by the author, and that the javascript can be obfuscated wit uglify etc, is there a way to protect at least the model's files in some way?

Thank you

[Daniel Smilkov]

Some other concerns about TFJS and security to the TF JS dev team. In few steps it's easy to download the models files and reproduce the model locally. Now, supposed that a TFJS model running in the browser is copyright protected by the author, and that the javascript can be obfuscated wit uglify etc, is there a way to protect at least the model's files in some way?

Great q. Short version of an answer: We've been thinking about this a lot. There is no way to protect a model at the moment. For now, if you want to protect the model for all but a small set of users, you can do what the web does, use auth tokens and require users to authenticate. You'd have to trust those users.

 

Daniel

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/1a846f66-5c6d-4c74-b247-eed2b0a678a4%40tensorflow.org.

[Loreto Parisi]

Thanks for the clarification, do you mean oauth 2.0 with user login on the web site or an authentication token for each file request if so, there should be something in the model's hub mechanism I think that would be a better solution. Of course that will be optional (I have seen around some trusted model file sharing architecture based on blockchain, but very complex indeed). By the way, could you please address me to some examples of approach you describe i.e. auth tokens, etc. - if any?
Thanks a lot!

Great q. Short version of an answer: We've been thinking about this a lot. There is no way to protect a model at the moment. For now, if you want to protect the model for all but a small set of users, you can do what the web does, use auth tokens and require users to authenticate. You'd have to trust those users.

 

Daniel

On Wed, Apr 25, 2018 at 10:51 AM Loreto Parisi <loreto...@gmail.com> wrote:

Thanks for sharing I have tried the tool with both the GAN 64 and ResNet 128 and 256.
Few questions:
- Why did you use Chainer's GAN?
- Are the model's chunks i.e. the files named shards the only way to handle these files i.e. multiple files rather than a single file (or is the latter possible?)

Some other concerns about TFJS and security to the TF JS dev team. In few steps it's easy to download the models files and reproduce the model locally. Now, supposed that a TFJS model running in the browser is copyright protected by the author, and that the javascript can be obfuscated wit uglify etc, is there a way to protect at least the model's files in some way?

Thank you

Il giorno martedì 24 aprile 2018 02:55:07 UTC+2, Yingtao Tian ha scritto:

Hey everybody,

I would like to share https://alantian.net/ganshowcase/ , my weekend project which is a showcase of a deep GAN (Generative Adversarial Network) that generates (or dreams) high resolution (256x256) photo realistic images, running completely on browser thanks to TensorFlow.js . 

Technically, the network architecture is similar to the residual network (ResNet) based generator (Gulrajani et al.), as well as the classical DCGAN generator (Radford et al.) and the GAN training uses DRAGAN (Kodali et al.) style gradient penalty for better stability. Training code is written in Chainer. The trained model is then manually converted to a Keras model, which in turn is converted to a web-runnable TensorFlow.js model. The dataset used for training is CelebAHQ, an dataset for Karras et al. which can be obtained by consulting its github repo. 

(More info follows in the project page)

Note that one issue is that the model files are large (good model occupies ~200+MB) due to (1) ResNet-like architecture and (2) use of Conv2DTranspose with manually constructed weights to mimic unpooling (not supported in TensorFlow.js yet), both leads to large (de)convolution in the network, especially in early layers. As Nikhil Thorat suggests, using Separable (De)convolutions like ones in MobileNet may mitigate this issue when supports are available.

Feedback are welcome!

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+unsubscribe@tensorflow.org.

Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/1a846f66-5c6d-4c74-b247-eed2b0a678a4%40tensorflow.org.

--

Dott. Ing. Loreto Parisi
Parisi Labs

Company: in...@parisilabs.com

Personal: loreto...@gmail.com

Twitter: @loretoparisi

Web: http://parisilabs.com
LinkedIn: http://www.linkedin.com/in/loretoparisi

[Yingtao Tian]

On Tuesday, April 24, 2018 at 10:13:35 AM UTC-4, Daniel Smilkov wrote:

This is amazing. Btw the demo gives 404.

 

Thanks! Maybe it was time when I was updating the model --- I re-deployed and it should be working.

 

2) use of Conv2DTranspose with manually constructed weights to mimic unpooling (not supported in TensorFlow.js yet),

Looks like TF Python doesn't have unpooling yet also. Unpooling seems to be the gradient of maxPool, which we have implemented, but haven't exposed it yet to the public API. Instead of unpooling, can you try using bilinearResize , after the conv2d, to upscale the input.

Yeah.. I use unpooling due to its success in many deep generative model's intermediate layers.  I now convert to TensorFlow.js from Keras model. Would converting from Tensorflow saved model better since unpooling can be implemented as a chain of transpose/expend_dims/reshape/tile (I did in my tfjs code to show 64x64 image in a 256x256 canvas) etc?  Also it seems that bilinearResize can be try in place of unpooling layers as you mentioned.

 

 

As Nikhil Thorat suggests, using Separable (De)convolutions like ones in MobileNet may mitigate this issue when supports are available.

We already have separable conv  in tfjs. Try it out!

One issue is that I want a separable version of conv2DTranspose --- would it be possible do to it with separableConv2d?

[Yingtao Tian]

On Wednesday, April 25, 2018 at 10:51:22 AM UTC-4, Loreto Parisi wrote:

Thanks for sharing I have tried the tool with both the GAN 64 and ResNet 128 and 256.
Few questions:
- Why did you use Chainer's GAN?

Because training GAN is hard and I only managed to train successfully using Chainer where I customized the training loop by using DRAGAN-style gradient penalty and some hack-ish weight clipping. 

I attempted using Keras for training but the training dynamics seemed to be wired.

 

- Are the model's chunks i.e. the files named shards the only way to handle these files i.e. multiple files rather than a single file (or is the latter possible?)

Some other concerns about TFJS and security to the TF JS dev team. In few steps it's easy to download the models files and reproduce the model locally. Now, supposed that a TFJS model running in the browser is copyright protected by the author, and that the javascript can be obfuscated wit uglify etc, is there a way to protect at least the model's files in some way?

Good q and Daniel's answer is great.

[Yingtao Tian]

Sorry for my typo, I mean "DRAGAN-style gradient penalty and some hack-ish learning rate annealing. "

[Daniel Smilkov]

One issue is that I want a separable version of conv2DTranspose --- would it be possible do to it with separableConv2d

You can use dilation >1 to simulate de-separableConv2d. tf.separableConv2d takes dilation as a param. We just need to expose it to tf.layers.separableConv2d. I filed an issue.

Daniel

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/eb22bdf4-e119-4ab6-ba8c-2b23b5b55292%40tensorflow.org.

[Yingtao Tian]

The source code for the whole pipeline that makes this showcase is available on GitHub now: https://github.com/alantian/ganshowcase

[Nikhil Thorat]

Thanks for opening up the code!

To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+unsubscribe@tensorflow.org.

Visit this group at https://groups.google.com/a/tensorflow.org/group/tfjs/.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/a5220365-305c-4350-9bb9-c0201f222746%40tensorflow.org.