Future of the forward mode differentiation

53 views
Skip to first unread message

Vojta Molda

unread,
Feb 17, 2021, 12:35:15 PMFeb 17
to Swift for TensorFlow
The differentiable programming proposal being pitched for integration into Swift is scoped only to the reverse mode differentiation. I understand that it makes sense to reduce the breadth because the project is primarily aiming at deep learning applications. And DL almost exclusively uses the reverse mode.

I've used S4TF for physics and simulations and it frequently makes sense to use my personal favorite, the forward mode :) I know it's been a bit less tested and some features implemented for reverse were missing, but in my experience it worked well enough.

Can someone here, please, comment on what is the planned future of the forward mode? Is it going to be included in nightly builds of Swift under experimental _Differentiable import after the proposal is accepted? Or is it part of the tensorflow/swift fork and it's going go into "maintenance" unless someone comes over and picks it up?

Vojta


PS1: Thanks to the S4TF team at Google and to all contributors. I appreciate Brennan's courage to hold the last meeting and answer all the questions. I feel that most people in the same situation would just post a message on the forums and then bury their head in the sand like the proverbial ostrich. In some ways it is harder you than the community, because you spend a good few years of your life working on it full (over)time.

PS2: I'm still puzzled about the reasons why the project was stopped. But, I guess, the only way to find out this secret is to go work for Google...


Dan Zheng

unread,
Feb 17, 2021, 12:50:49 PMFeb 17
to Vojta Molda, Swift for TensorFlow
Hi Vojta,

Thank you for your question and kind words!

The development of differentiable programming in Swift continues in open-source: you may have more luck getting useful responses and interactions by posting on forums.swift.org – either creating a standalone #autodiff thread or adding a comment to this differentiable programming Swift Evolution pitch thread.

My understanding is that development of forward-mode differentiation has been delayed in the initial Evolution pitch, to keep the scope small. The prototype forward-mode differentiation used an approach similar to reverse-mode differentiation [0] – but there are alternative approaches too, like dual numbers [1] and Enzyme [2]. There are tradeoffs regarding UX and flexibility vs performance & optimization potential.

[0]: The JVP (Jacobian-vector product) function type for an original (T) -> U function type looks like (T) -> (value: U, differential: (T.TangentVector) -> U.TangentVector).

[1]: Dual number formulation of (forward-mode) autodiff is quite easier to implement: (value: T, tangent: T.TangentVector) -> (result: U, resultTangent: U.TangentVector). It's possible to formulate reverse-mode using a dual number approach. And it may be theoretically possible to unify multiple differentiation modes (including reverse-mode) under forward-mode-differentiation-first typing rules.

[2] Enzyme performs autodiff on optimized LLVM IR. In the space of autodiff approaches, Enzyme's approach and starting point (focused on performance?) seems quite different from Swift's (focused on good user experience, idiomatic-ness within Swift, good differentiable function ABI design, etc). Enzyme is being considered for incubation as an LLVM project and is being integrated with Julia and Fortran - it may be possible to hook up Enzyme with Swift as well.

See you on the forums!

Dan

--
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

Michelle Casbon

unread,
Feb 17, 2021, 1:27:45 PMFeb 17
to Dan Zheng, Vojta Molda, Swift for TensorFlow
Hi Vojta,

I'd like to express my gratitude for PS1. I am also appreciative of our last meeting and Brennan's responsibility to our team and the community. We've all invested significant efforts in this project and his leadership in turning things down responsibly is indeed notable. Thank you, Vojta for recognizing this.

--Michelle

Dan Zheng

unread,
Feb 17, 2021, 1:44:34 PMFeb 17
to Michelle Casbon, Vojta Molda, Swift for TensorFlow
Seconded Michelle!

Some more clarifications on Vojta's question:

Can someone here, please, comment on what is the planned future of the forward mode? Is it going to be included in nightly builds of Swift under experimental _Differentiable import after the proposal is accepted? Or is it part of the tensorflow/swift fork and it's going go into "maintenance" unless someone comes over and picks it up?

Forward-mode differentiation is currently available with nightly builds of Swift (swift.org/download#trunk-development-main) via import _Differentiation and invocations of swiftc -Xfrontend -enable-experimental-forward-mode-differentiationHere are some examples from the compiler test suite.

There is no meaningful Swift for TensorFlow fork of the Swift compiler: all differentiable programming support exists on master and is available via trunk development toolchains. Regarding further development and plans for forward-mode differentiation: the Swift forums are a great place to ask.

Dan

Vojta Molda

unread,
Feb 17, 2021, 4:41:36 PMFeb 17
to Swift for TensorFlow, fangda...@gmail.com, Vojta Molda, Swift for TensorFlow, Michelle Casbon
Thanks Dan (and Michelle). The links are exactly what I was looking for!

Gaston Seneza

unread,
Feb 17, 2021, 5:41:48 PMFeb 17
to Dan Zheng, Vojta Molda, Swift for TensorFlow
Thank you Dan!
 -Gaston S.

"Either I will find a way or I will make one" ~Philip Sidney

Reply all
Reply to author
Forward
0 new messages