ScopedAllocatorOptimizer & MultiWorkerMirroredStrategy

11 views

Skip to first unread message

Bili Sun

unread,

Feb 21, 2021, 8:20:04 PM2/21/21

to TensorFlow End Users - GETTING STARTED, TUTORIALS & HOW-TO'S

Hi all,

I am using distributed training with MultiWorkerMirroredStrategy in Tensorflow 1.14 (currently in the progress of moving to Tensorflow 2.4), and I wanted to understand if there is a need to backport any changes from ScopedAllocatorOptimizer, and if so, which changes would be critical. Looking at https://github.com/tensorflow/tensorflow/commits/master/tensorflow/core/grappler/optimizers/scoped_allocator_optimizer.cc it seems like there are issues with control_dependencies for example, which makes me think that some issues I've been seeing with MultiWorkerMirroredStrategy might be due to my version of Tensorflow. Thank you!

Best,

Bili Sun

unread,

Feb 21, 2021, 8:37:30 PM2/21/21

to TensorFlow End Users - GETTING STARTED, TUTORIALS & HOW-TO'S

Overall I'm most concerned about correctness of gradient calculations / potential undefined behavior.

Reply all

Reply to author

Forward

0 new messages