SWE-Glu SF: Entropy Sampling and Parallel CoT Decoding

[Cheikh Fiteni]

Glu afternoon,

Our meeting will be Saturday, October 12th, 2:30 PM @ 848 Divisadero Street. If you’ve spent any time on the interwebs lately, you’ll be unsurprised to learn that this weekend we’ll reading, running, and discussing through Entropix, xjdr’s entropy sampling repo. The repo is being actively contributed to, so we’ll try to stay scoped to commit e55e9a3.

There are a lot of goodies here, so we really encourage people not to miss. The repo will be the main source of authority, but (unironically) reading through @_xjdr, and taking a look at Chain-of-Thought Reasoning without Prompting and a quick reskim of parts of  Quiet-STaR, will explain a lot of the thought process.

 Please read and run the codebase before coming, and this colab (able to run on Google’s free TPU tier!) is a great immediate starting point. If you have any questions about JAX or getting situated feel free to reach out. Look forward to seeing you all there.

Why understanding entropy sampling is cool:

1. Might solve hallucination

2. Doesn’t break the bank with pretraining

3. Strong underlying theory, with intuitive relationship to basic concepts from statistics and information theory. This quick blog post is excellent: Entropy-based sampling

Find out what these colors mean 🧐

Best,
Cheikh and Sasha

P.S. if you are somehow reading this email but not on our listserv join it here. If you are on our listserv, send it to your friends.