Fusion of two Conv2D layers

[Dimitra Karatza]

Hello everyone,

While diving into kernel fusion in TensorFlow, I noticed that the main goal is to fuse convolution layers with following element wise operations, but never fuse two convolutions with each other. I was wondering if you plan on doing so. I suppose it will be a very interesting and useful feature, to which I would like to contribute to.

Thank you,

Dimitra Karatza

[Wei D. Shao]

The proposed fusion is not feasible though, mathematically speaking. Conceptually it results in a quadratic operations on the first level input which cannot be modeled as a conv. 

--
You received this message because you are subscribed to the Google Groups "TensorFlow Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/developers/af074533-e12c-4713-9604-35ba6cda5117n%40tensorflow.org.

[Dimitra Karatza]

I see. Why does it result in a quadratic operation though?

[Wei D. Shao]

Sorry I was wrong in stating on the quadratic relationship. But the Activation (eg Relu) and output regularization for the conv2d would introduce nonlinear effects. So it is not possible to model 2 consecutive conv into one. Even if there is no Relu or other nonlinear effects, when you try to force fuse of 2 pure Conv ops, you would have an expanded kernel size, defeating the purpose of fuse. If your intention is to reduce MAC counts, consider a new model architecture and retrain the network. 

On Fri, Feb 26, 2021 at 10:08 AM Dimitra Karatza <dimitra....@gmail.com> wrote:

I see. Why does it result in a quadratic operation though?

On Friday, 26 February 2021 at 19:22:55 UTC+2 weido...@gmail.com wro

The proposed fusion is not feasible though, mathematically speaking. Conceptually it results in a quadratic operations on the first level input which cannot be modeled as a conv. 

On Fri, Feb 26, 2021 at 8:52 AM Dimitra Karatza <dimitra....@gmail.com> wrote:

Hello everyone,

While diving into kernel fusion in TensorFlow, I noticed that the main goal is to fuse convolution layers with following element wise operations, but never fuse two convolutions with each other. I was wondering if you plan on doing so. I suppose it will be a very interesting and useful feature, to which I would like to contribute to.

Thank you,

Dimitra Karatza

--
You received this message because you are subscribed to the Google Groups "TensorFlow Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@tensorflow.org.