Truncated BPTT

222 views
Skip to first unread message

Fernando Quivira

unread,
Jan 17, 2017, 11:29:29 PM1/17/17
to Chainer User Group
Hi

I'm reading a bit more on backpropagation throught time (BPTT) and its variations, and I came across the following blog post: http://r2rt.com/styles-of-truncated-backpropagation.html

If I understand correctly, when we use unchain_backward(),  we are using the following paradigm, right? (graphic from the blog post)


Diagram of Tensorflow Truncated Backpropagation

The unchaining would occur right when we compute the "final state." Then, after unchaining, we use the previous state and continue computing with the RNN.

The blog post mentions this other approach: 

Diagram of True Truncated Backpropagation

This is another variation on BPTT (different parameters k1 and k2 as explained in http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf)

My question is: is there a clean way of implementing these BPTT variations in Chainer? 

Thanks!

Fernando Quivira

unread,
Feb 26, 2017, 2:45:15 PM2/26/17
to Chainer User Group
Does anybody have an idea how to do this? or am I incorrect in my understanding of BPTT?

Thanks
Reply all
Reply to author
Forward
0 new messages