Minor Syntax Question

[Xander Dunn]

In the BERT model in swift-models, the TransformerEncoderLayers are initialized with parameters { $0 } and I’m having difficulty understanding what that value turns out to be. See here:

self.encoderLayers = (0..<hiddenLayerCount).map { _ in
TransformerEncoderLayer(
hiddenSize: hiddenSize,
attentionHeadCount: attentionHeadCount,
attentionQueryActivation: { $0 },
attentionKeyActivation: { $0 },
attentionValueActivation: { $0 },
intermediateSize: intermediateSize,
intermediateActivation: intermediateActivation,
hiddenDropoutProbability: hiddenDropoutProbability,
attentionDropoutProbability: attentionDropoutProbability)
        }

What are the attentionQueryActivation, attentionKeyActivation, and attentionValueActivation values? The value from the .map { is ignored via the _ in, but even if it weren’t, Ints wouldn’t make sense there? This is probably just some facet of Swift syntax I haven’t encountered yet.

Thanks,
Xander

​

[Dan Zheng]

Hi Xander,

> What are the attentionQueryActivation, attentionKeyActivation, and attentionValueActivation values?

These "xxActivation" parameters are activation functions, which take some previous layer output and apply some typically non-linear transformation. Here, the activation functions are just passthrough identity functions, so it is as if no activation is applied at all.

> The value from the .map { is ignored via the _ in, but even if it weren’t, Ints wouldn’t make sense there?

(0..<n).map { _ in value } is simply a way to create an Array containing n duplicates of value.

A more idiomatic way to write the code would be: Array(repeating: TransformerEncoderLayer(...), count: hiddenLayerCount).

Dan

--
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.

[Xander Dunn]

Ah, this closure is itself the activation function. { $0 } is a function that returns the input. I was expecting to see something like identity but this makes sense now.

​

[Dan Zheng]

Yep, { $0 } is the identity function as a closure.

Dan