Minor Syntax Question

27 views
Skip to first unread message

Xander Dunn

unread,
Jan 28, 2021, 9:56:29 PM1/28/21
to Swift for TensorFlow

In the BERT model in swift-models, the TransformerEncoderLayers are initialized with parameters { $0 } and I’m having difficulty understanding what that value turns out to be. See here:

self.encoderLayers = (0..<hiddenLayerCount).map { _ in
TransformerEncoderLayer(
hiddenSize: hiddenSize,
attentionHeadCount: attentionHeadCount,
attentionQueryActivation: { $0 },
attentionKeyActivation: { $0 },
attentionValueActivation: { $0 },
intermediateSize: intermediateSize,
intermediateActivation: intermediateActivation,
hiddenDropoutProbability: hiddenDropoutProbability,
attentionDropoutProbability: attentionDropoutProbability)
        }

What are the attentionQueryActivation, attentionKeyActivation, and attentionValueActivation values? The value from the .map { is ignored via the _ in, but even if it weren’t, Ints wouldn’t make sense there? This is probably just some facet of Swift syntax I haven’t encountered yet.

Thanks,
Xander

Dan Zheng

unread,
Jan 29, 2021, 8:59:46 AM1/29/21
to Xander Dunn, Swift for TensorFlow
Hi Xander,

> What are the attentionQueryActivationattentionKeyActivation, and attentionValueActivation values?

These "xxActivation" parameters are activation functions, which take some previous layer output and apply some typically non-linear transformation. Here, the activation functions are just passthrough identity functions, so it is as if no activation is applied at all.

> The value from the .map { is ignored via the _ in, but even if it weren’t, Ints wouldn’t make sense there?

(0..<n).map { _ in value } is simply a way to create an Array containing n duplicates of value.

A more idiomatic way to write the code would be: Array(repeating: TransformerEncoderLayer(...), count: hiddenLayerCount).

Dan

--
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

Xander Dunn

unread,
Jan 29, 2021, 9:05:30 AM1/29/21
to Swift for TensorFlow, danie...@google.com, Swift for TensorFlow, Xander Dunn

Ah, this closure is itself the activation function. { $0 } is a function that returns the input. I was expecting to see something like identity but this makes sense now.

Dan Zheng

unread,
Jan 29, 2021, 9:12:56 AM1/29/21
to Xander Dunn, Swift for TensorFlow
Yep, { $0 } is the identity function as a closure.

Dan
Reply all
Reply to author
Forward
0 new messages