The Transformers have arrived in PicoChess!
Dirk
Meet chess‑transformers: a playful twist on your classic chess engines, but instead of evaluating millions of positions via search, it reads the move list as text, learns how humans actually play, and predicts the next move in a single transformer go. It’s kind of like training a GPT for chess moves—no brute-force tree search, just pure pattern learning from human games
Ok, after having ported the new maia2 models (https://groups.google.com/g/picochess/c/_zGh_KyDwds) which offer only a decent playing experience because of its restricted opening and endgame knowledge and which are also based on the new transforms architecture, I finally got the transformers-chess project running on PicoChess!
This engine is based on the brilliant chess-transformers project by sgrvinod. I’ve just wrapped it in a friendly UCI shell so we can all have some fun with it.
So what is difference to „old“ lc0/maia1 models (in the meantime newer lc0 models are also transformer based)?
Chess Transformers - The Linguists
Projects like Maia2 or chess-transformers treat chess not as a game of positions, but as a language, where games are sentences and moves are words.
- Core Concept and How It Works:
Based on the Transformer architecture (famous from Large Language Models like GPT), the model learns the patterns, rules, and strategy of chess simply by reading the sequence of moves in thousands of games. - Network Architecture:
- Type: A Transformer network. This architecture is specifically designed to understand dependencies in long sequences (e.g., the relationship between words in a long sentence).
- Input: The sequence of all previous moves in the game, for example, in Standard Algebraic Notation (SAN) like ["1. e4", "1... e5", "2. Nf3", "2... Nc6"].
- Output: A probability distribution for the very next move in the sequence.
- Learning Phase (Training):
The training is also Supervised Learning, but it closely resembles the training of a language model:- Data: A large database of games in PGN format.
- Learning Goal: The model is given an incomplete game (a sequence of moves) and is trained to predict the next move in the game. It learns the "grammar" and "semantics" of chess by "reading" master games and trying to continue them. The rules are not explicitly taught; the model infers them from the patterns.
- Playing Phase (Inference):
Similar to Maia, the playing phase is a direct query of the model:- The sequence of moves played so far in the game is given to the Transformer model as input.
- The model calculates the probabilities for all possible continuations (next moves).
- The move with the highest probability is selected and played. Here too, there is generally no deep tree search. The strength of the move relies solely on the deep contextual understanding of the move sequence that the Transformer has learned.
Installation
Like with Maia2 and typical for q pure PyTorch implementation several steps are needed and even some modifications of the original GitHub files were necesscarry in order to run the transformers models with PicoChess 3.3.
pip3 install einops tqdm gdown markdown tabulate Ipython colorama
pip3 install torch --index-url https://download.pytorch.org/whl/cpu
sudo apt install libhdf5-dev liblzo2-dev libblosc-dev
pip uninstall -y numpy tables numexpr blosc2 setuptools wheel Cython
pip install --upgrade pip
pip install --upgrade setuptools wheel Cython
pip install numpy==1.26.4
pip install --no-cache-dir --extra-index-url https://www.piwheels.org/simple tables==3.9.2
pip install regex
Copy the transformers_uci folder to the engines/script_engines directory (containing my uci wrapper transformers_uci.py) and the bash and uci files to the arch64/script folder.
Add this favorites.ini entry:
[script/transformers]
name = chess-transformers by MIT UCI version by Molli
small = trans
medium = transf
large = Transform.
web = Chess-Transformers
elo = 1500
ponder/brain = n
What’s under the hood?
The project ships with a few transformer architectures:
- CT‑E‑20 (~20 M parameters): encoder-only, predicts the next half‑move in UCI notation.
- CT‑EFT‑20 (~20 M): encoder-only but predicts both the source and target squares for that move.
- CT‑ED‑45 (encoder‑decoder, ~45 M): predicts sequences of half‑moves.
- CT‑EFT‑85 (~85 M): encoder-only, next from‑to square prediction with larger capacity (GitHub).
Since version v0.2.0 (Dec 17, 2023), CT‑EFT‑20 became the star: compact, smart, and surprisingly effective .
Important!
- At the first time you select a new model the model will be downloaded (so you need an active internet connection). After this happened the offline models will be used.
- The "Transformation": The One-Time Warm-Up!
When you first load the engine, it needs to "transform" from its dormant state (the raw model file) into its ready-to-fight "robot mode."
- What's Happening? Under the hood, PyTorch (the deep learning/net framework it runs on) is performing a Just-In-Time (JIT) compilation. It analyzes the model and optimizes it for your specific hardware (our Pi!).
- What You'll See: The first time you ask the engine to think (either via the isready command or the first go), it will pause for a bit. On a Raspberry Pi 4, this "transformation" might take anywhere from 30 to 60 seconds. It might look like it's thinking really hard, but it's really just getting its battle armor on.
- Then... Instant Moves! After this one-time warm-up is complete, the engine is lightning-fast. Since it doesn't search, it will return its move almost instantly for the rest of the game!
My UCI wrapper includes a Warmup uci option (enabled by default) to handle this initial compilation gracefully before the first move directly after having loaded the model. But this means it will last up to 1 minute until the engine is ready (or if you have the engine selected as last engine the PicoChess boot up will last as long as the engine is ready). After this initialization the engine answers immediately and indeed plays an interesting game...
Enjoy
Dirk
Dirk
P.S.
1) Forgot to mention before you do the pip installation commands you must activate the corresponding virtual environment: eg. pyenv activate picochess-3.9.2
2) Don’t feel like dealing with a complex installation?
Just wait a little bit.
My human-style Leela Chess Zero enhancements are coming soon — drop-in nets, a few engine settings and that’s it - no headaches. 😄
