Future of the forward mode differentiation

[Vojta Molda]

The differentiable programming proposal being pitched for integration into Swift is scoped only to the reverse mode differentiation. I understand that it makes sense to reduce the breadth because the project is primarily aiming at deep learning applications. And DL almost exclusively uses the reverse mode.

I've used S4TF for physics and simulations and it frequently makes sense to use my personal favorite, the forward mode :) I know it's been a bit less tested and some features implemented for reverse were missing, but in my experience it worked well enough.

Can someone here, please, comment on what is the planned future of the forward mode? Is it going to be included in nightly builds of Swift under experimental _Differentiable import after the proposal is accepted? Or is it part of the tensorflow/swift fork and it's going go into "maintenance" unless someone comes over and picks it up?

Vojta

PS1: Thanks to the S4TF team at Google and to all contributors. I appreciate Brennan's courage to hold the last meeting and answer all the questions. I feel that most people in the same situation would just post a message on the forums and then bury their head in the sand like the proverbial ostrich. In some ways it is harder you than the community, because you spend a good few years of your life working on it full (over)time.

PS2: I'm still puzzled about the reasons why the project was stopped. But, I guess, the only way to find out this secret is to go work for Google...

[Dan Zheng]

Hi Vojta,

Thank you for your question and kind words!

The development of differentiable programming in Swift continues in open-source: you may have more luck getting useful responses and interactions by posting on forums.swift.org – either creating a standalone #autodiff thread or adding a comment to this differentiable programming Swift Evolution pitch thread.

My understanding is that development of forward-mode differentiation has been delayed in the initial Evolution pitch, to keep the scope small. The prototype forward-mode differentiation used an approach similar to reverse-mode differentiation [0] – but there are alternative approaches too, like dual numbers [1] and Enzyme [2]. There are tradeoffs regarding UX and flexibility vs performance & optimization potential.

[0]: The JVP (Jacobian-vector product) function type for an original (T) -> U function type looks like (T) -> (value: U, differential: (T.TangentVector) -> U.TangentVector).

[1]: Dual number formulation of (forward-mode) autodiff is quite easier to implement: (value: T, tangent: T.TangentVector) -> (result: U, resultTangent: U.TangentVector). It's possible to formulate reverse-mode using a dual number approach. And it may be theoretically possible to unify multiple differentiation modes (including reverse-mode) under forward-mode-differentiation-first typing rules.

[2] Enzyme performs autodiff on optimized LLVM IR. In the space of autodiff approaches, Enzyme's approach and starting point (focused on performance?) seems quite different from Swift's (focused on good user experience, idiomatic-ness within Swift, good differentiable function ABI design, etc). Enzyme is being considered for incubation as an LLVM project and is being integrated with Julia and Fortran - it may be possible to hook up Enzyme with Swift as well.

See you on the forums!

Dan

--
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

[Michelle Casbon]

Hi Vojta,

I'd like to express my gratitude for PS1. I am also appreciative of our last meeting and Brennan's responsibility to our team and the community. We've all invested significant efforts in this project and his leadership in turning things down responsibly is indeed notable. Thank you, Vojta for recognizing this.

--Michelle

[Dan Zheng]

Seconded Michelle!

Some more clarifications on Vojta's question:

Can someone here, please, comment on what is the planned future of the forward mode? Is it going to be included in nightly builds of Swift under experimental _Differentiable import after the proposal is accepted? Or is it part of the tensorflow/swift fork and it's going go into "maintenance" unless someone comes over and picks it up?

Forward-mode differentiation is currently available with nightly builds of Swift (swift.org/download#trunk-development-main) via import _Differentiation and invocations of swiftc -Xfrontend -enable-experimental-forward-mode-differentiation. Here are some examples from the compiler test suite.

There is no meaningful Swift for TensorFlow fork of the Swift compiler: all differentiable programming support exists on master and is available via trunk development toolchains. Regarding further development and plans for forward-mode differentiation: the Swift forums are a great place to ask.

Dan

[Vojta Molda]

Thanks Dan (and Michelle). The links are exactly what I was looking for!

[Gaston Seneza]

Thank you Dan!

 -Gaston S.

"Either I will find a way or I will make one" ~Philip Sidney

[Dane christian Neilson]

MT103/202 DIRECT WIRE TRANSFER
PAYPAL TRANSFER
CASHAPP TRANSFER
ZELLE TRANSFER
LOAN DEAL
TRANSFER WISE
WESTERN UNION TRANSFER
BITCOIN FLASHING
BANK ACCOUNT LOADING/FLASHING
IBAN TO IBAN TRANSFER
MONEYGRAM TRANSFER
IPIP/DTC
SLBC PROVIDER
CREDIT CARD TOP UP
DUMPS/ PINS
SEPA TRANSFER
WIRE TRANSFER
BITCOIN TOP UP
GLOBALPAY INC US
SKRILL USA
UNIONPAY RECEIVER

Thanks.


NOTE; ONLY SERIOUS / RELIABLE RECEIVERS CAN CONTACT.

DM ME ON WHATSAPP
+14234753476