In the BERT model in swift-models, the TransformerEncoderLayer
s are initialized with parameters { $0 }
and I’m having difficulty understanding what that value turns out to be. See here:
self.encoderLayers = (0..<hiddenLayerCount).map { _ in
TransformerEncoderLayer(
hiddenSize: hiddenSize,
attentionHeadCount: attentionHeadCount,
attentionQueryActivation: { $0 },
attentionKeyActivation: { $0 },
attentionValueActivation: { $0 },
intermediateSize: intermediateSize,
intermediateActivation: intermediateActivation,
hiddenDropoutProbability: hiddenDropoutProbability,
attentionDropoutProbability: attentionDropoutProbability)
}
What are the attentionQueryActivation
, attentionKeyActivation
, and attentionValueActivation
values? The value from the .map {
is ignored via the _ in
, but even if it weren’t, Int
s wouldn’t make sense there? This is probably just some facet of Swift syntax I haven’t encountered yet.
Thanks,
Xander
attentionQueryActivation
, attentionKeyActivation
, and attentionValueActivation
values?.map {
is ignored via the _ in
, but even if it weren’t, Int
s wouldn’t make sense there?--
To unsubscribe from this group and stop receiving emails from it, send an email to swift+un...@tensorflow.org.
Ah, this closure is itself the activation function. { $0 }
is a function that returns the input. I was expecting to see something like identity
but this makes sense now.