VSAONLINE. SEASON 12. May 25, 20:00GMT. Travis Pence
Evgeny Osipov
Dear all,
Welcome to the next talk of Season 12 on VSAONLINE. Travis Pens from University of Wisconsin-Madison, USA
will give a talk
”Composing Linear Layers from Irreducibles”
Date: May 25, 2026
Time: 20:00 GMT
Zoom: https://ltu-se.zoom.us/j/65564790287
Abstract: Contemporary large models often exhibit behaviors suggesting the presence of low-level primitives that compose into modules with richer functionality, but these fundamental building blocks remain poorly understood. We investigate this compositional structure in linear layers by asking: \textit{can we identify/synthesize linear transformations from a minimal set of geometric primitives?}
Using Clifford algebra, we show that linear layers can be expressed as compositions of bivectors---geometric objects encoding oriented planes---and introduce a differentiable algorithm that decomposes them into products of rotors.
This construction uses only O(log^2 d) parameters, versus O(d^2) required by dense matrices. Applied to the key, query, and value projections in LLM attention layers, rotor-based layers match the performance of strong baselines such as block-Hadamard and low-rank approximations. Our findings provide an algebraic perspective on how these geometric primitives can compose into higher-level functions within deep models.
Evgeny Osipov
Dear all,
Welcome to the next talk of Season 12 on VSAONLINE. Travis Pens from University of Wisconsin-Madison, USA
will give a talk
”Composing Linear Layers from Irreducibles”
Date: June 1, 2026