Model-Parallel Layout in Mesh-Tensorflow

Skip to first unread message

Suhas Shekhar

Jul 1, 2019, 8:39:07 PM7/1/19
to tensor2tensor

I am trying to understand the mesh shape and layout rules. The paper talks about splitting the model in the hidden layer for model parallelization. And the example provided has one hidden layer. How to specify the layout for multiple hidden layers? My assumption is, it is enough to provide one layer of the model for the layout rules and the rest of the layer will be split accordingly. Is it a right assumption? Also, could anyone provide an example of the layout for CNN with multiple layers?

Thank you.


Jul 30, 2019, 3:56:23 PM7/30/19
to tensor2tensor
+1, looking for answer/
Thanks you
Reply all
Reply to author
0 new messages