Alternative approaches for using 3D conv/pooling in TFJS

[Anish Khazane]

Hi,

I'm currently working  on a project that requires a TF Keras model to be translated into TFJS, but I ran into a few issues with layer compatibility issues with 3D max pool/cnv, which I've already created an issue for here: https://github.com/tensorflow/tfjs/issues/1035

While this issue is being examined, does the community have any suggestions on alternative approaches for translating 3D layers (specifically 3D conv and 3D max-pooling) to a JS-supported framework? I found another framework, Keras.js, that does support these layers, but its author is no longer supporting the project. I couldn't find any other frameworks that seem to support 3D layers...

Or, is it possible to re-create my entire model via Tensorflow (and NOT Keras) and convert it to TFJS? Would this run into the same issue / are the only supported layers, regardless of framework, the ones listed in the TFJS API reference? 

Another approach that I thought about was re-creating my model within tensorflow.js itself, and using the recently added 3D conv layer and N-D pooling layer to mimic my Keras model's 3D conv / 3D max pooling layers. However, the N-D pooling layer only returns a 3D or 4D tensor, whereas my model needs to take in / return a 5D tensor (batch_size, channels, height, width, depth), so I'm not sure if I can even do this in-house with the lastest TFJS version.

Thanks for reading my post. Just wanted to look at alternative solutions for my use-case before considering a jump at the 3D max pooling / tfjs-converter implementation myself.

Thanks!

[Shanqing Cai]

Hi, Anish, 

Thanks for reporting the issue regarding missing 3D max pooling layers in TensorFlow.js. Looking at the code for MaxPooling3D in Keras (https://github.com/keras-team/keras/blob/master/keras/layers/pooling.py#L424), it depends on a backend function called pool3d. But such a function seems to be still missing at the level of tfjs-core (ops) API. So it needs to be added before we can implement the MaxPooling3D layer. The good news is that thanks to an external contributor, we've recently added conv3d (https://js.tensorflow.org/api/0.14.1/#conv3d). pool3d shouldn't be much harder to implement than conv3d. 

So, contribution will be welcome, while we at the TensorFlow.js team will also be happy to do that ourselves. But beware that given the things on our plate, we can't promise any concrete time frames for the completion of the task.

Are you missing any other layer types?

Best regards,

Shanqing

[Anish Khazane]

Hi Shanqing, 

Thanks for your reply, and pointing out the exact function that Keras is missing. Just to clarify though, does the tfjs-core API also interface with a model that is implemented purely with tensorflow layers API? I think the answer to this question, because I noticed that the tf.layers.max_pooling3d function in https://github.com/tensorflow/tensorflow/blob/r1.12/tensorflow/python/layers/pooling.py just ends up creating a MaxPooling3D class, which lacks the pool3d function that you mentioned earlier.

Fortunately, I don't think I'm missing anything else other than the pooling layer. Do you have any tips (for example, most relevant files) that might be helpful to look at before tacking a pool3d implementation? 

Thank you!

Anish