LLM From scratch?

20 views
Skip to first unread message

Gmail

unread,
Apr 26, 2025, 2:58:27 PM4/26/25
to RSSC-list
Has anyone created an LLM from scratch? I am going to do just that and I was hoping i could find someone that had done it before who could give me some pointers. 


Thomas



Thomas

-  

Need something prototyped, built or coded? I’ve been building prototypes for companies for 15 years. I am now incorporating generative AI into products.

-

Need a great hardworking engineer? I am currently looking for a new job opportunity in robotics and/ or AI. 

Contact me directly or through LinkedIn:   

Alan Timm

unread,
Apr 26, 2025, 5:26:06 PM4/26/25
to RSSC-List
Hey Thomas, having a pretty good idea about what's involved, I just have two questions:
  • Why?
  • How?
If you're talking about training a non-toy LLM from scratch (and not fine tuning) you are going to need literal terabytes of source material.  I'd start by taking a look at ThePile.  It's the first, and only, open source collection of text that's been published specifically for training LLMs.  https://pile.eleuther.ai/

Chris Albertson

unread,
Apr 26, 2025, 5:45:29 PM4/26/25
to Gmail, RSSC-list
An LLN from scratch?

The first thing you need is training data.  Many terabytes of it.   Then you need enough computing hardware so that the training is completed within your lifetime.   This usually means that you have a data center-sized facility.      There was a Chinese lab a couple of months ago that astonished the world by creating an LLM from scratch with a budget of “only” a few tens of millions of dollars.

I don’t think you want to start from zero.  

The best way is to download an LLM model from Hugging Face and then fine-tune it to fit your needs.  That is a realistic project that can be done by only one person working alone.   You will still need to buy hardware.  What you need depends on the number of parameters in your downloaded model.   A reasonable size for private use is about 8 billion parameters.   For that you would need as a minimum a GPU with more than 8GB VRAM.  Apple Macs with their unified RAM work well for 8 billion parameter models if the Mac has at least 16GB or RAM.  I’ve run one at decent speed on mine.

Going larger then 8 billion gets expensive because you will need a corresponding larger GPU.  There is a limit to what you can install in a home computer and that limit is very much smaller than the current state-of-the-art 400B parameter LLMs.

I found that an 8B parameter Llama modal could understand most English input and had very basic “smarts”.  It did not compare to ChatGPT but I think it could handle questions like “How many legs does a dog have?” or “how many beers in a 6-pack?”    These small models will “BS" badly.  For example I asked for a review of Bethoven’s 13th symphony and got a detailed three paragraph easy, all made up.  Then I asked for short biography of George Washington’s daughter, “Kate" and got another nicely written fictional story.

You have to understand the limits of an LLM that wil run on personal-sized computers.   8B is the current sweet spot.    I would go with Facebook's “Llama” whatever the current version is.  As said, it runs reasonably well on my Apple M2-Pro Mac mini with 16GB RAM.   A Linux PC with new 16GB Nvidia card would be better but the Mac is half the price and I already had it.  Do not even try using a Raspberry Pi.  A gamer-style PC would be best.

Here is where I got my copy



--
You received this message because you are subscribed to the Google Groups "RSSC-List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rssc-list+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rssc-list/E7ECD6BE-7D2E-45B9-BBDD-E13B0D7AD5B9%40gmail.com.

Chris Albertson

unread,
Apr 26, 2025, 6:05:30 PM4/26/25
to gestalt73, RSSC-list
My guess is that Thomas simply mistyped and used the wrong words.  I think all he wants is to run an LLM directly on his own hardware.

I just wrote an email discussing my experiance.  But just realized I did not give any practical pointers with no download links.   SO, see below.  Get it from Github.

If this is the case them I 100% recommend looking at the Open Source runtime engine called “llama.cpp”.  It lets you set up a server that looks a lot like a mini OpenAI server and uses about the same API.

This is good because then you can point your local software at either your own local server running llama.cpp ot to Open AI and not have to change much.

Llama.cpp will run models other than Llama.   As long as they are in the “correct” file format.

Llama.cpp is likey the highest perfooaming software you can get and it will run on ”anything”.  But you will want the biggest Nvidia GPU can can find and afford.



Actually, maybe read Wiki first, then the README.MD on GitHub.







-- 
You received this message because you are subscribed to the Google Groups "RSSC-List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rssc-list+...@googlegroups.com.

Gmail

unread,
Apr 26, 2025, 7:38:06 PM4/26/25
to Alan Timm, RSSC-List
Alan,

Why? Because I want to learn all the ins and outs of how one is created and add that to my resume. One learns best by doing. 

How? I saw a book on the subject. It says that a chatgpt2-kind of llm can be built using a standard PC, although I do have 2 gaming computers. Amazon link to the book: https://a.co/d/3eF5rS4

I assume by your answer that you have not yet created one from scratch. 


Thomas

-  

Need something prototyped, built or coded? I’ve been building prototypes for companies for 15 years. I am now incorporating generative AI into products.

-

Need a great hardworking engineer? I am currently looking for a new job opportunity in robotics and/ or AI. 

Contact me directly or through LinkedIn:   

On Apr 26, 2025, at 2:26 PM, Alan Timm <gest...@gmail.com> wrote:

Hey Thomas, having a pretty good idea about what's involved, I just have two questions:
--

Gmail

unread,
Apr 26, 2025, 7:48:03 PM4/26/25
to Chris Albertson, RSSC-list
Chris, 

Thanks 🙏 for your suggestions. Still, you didn’t answer my first question. 🤔 Have you, yourself built one from scratch? 🧐If so how did it go❓



Thomas

-  

Need something prototyped, built or coded? I’ve been building prototypes for companies for 15 years. I am now incorporating generative AI into products.

-

Need a great hardworking engineer? I am currently looking for a new job opportunity in robotics and/ or AI. 

Contact me directly or through LinkedIn:   


On Apr 26, 2025, at 2:45 PM, Chris Albertson <alberts...@gmail.com> wrote:

An LLN from scratch?

The first thing you need is training data.  Many terabytes of it.   Then you need enough computing hardware so that the training is completed within your lifetime.   This usually means that you have a data center-sized facility.      There was a Chinese lab a couple of months ago that astonished the world by creating an LLM from scratch with a budget of “only” a few tens of millions of dollars.

I don’t think you want to start from zero.  

The best way is to download an LLM model from Hugging Face and then fine-tune it to fit your needs.  That is a realistic project that can be done by only one person working alone.   You will still need to buy hardware.  What you need depends on the number of parameters in your downloaded model.   A reasonable size for private use is about 8 billion parameters.   For that you would need as a minimum a GPU with more than 8GB VRAM.  Apple Macs with their unified RAM work well for 8 billion parameter models if the Mac has at least 16GB or RAM.  I’ve run one at decent speed on mine.

Going larger then 8 billion gets expensive because you will need a corresponding larger GPU.  There is a limit to what you can install in a home computer and that limit is very much smaller than the current state-of-the-art 400B parameter LLMs.

I found that an 8B parameter Llama modal could understand most English input and had very basic “smarts”.  It did not compare to ChatGPT but I think it could handle questions like “How many legs does a dog have?” or “how many beers in a 6-pack?”    These small models will “BS" badly.  For example I asked for a review of Bethoven’s 13th symphony and got a detailed three paragraph easy, all made up.  Then I asked for short biography of George Washington’s daughter, “Kate" and got another nicely written fictional story.

You have to understand the limits of an LLM that wil run on personal-sized computers.   8B is the current sweet spot.    I would go with Facebook's “Llama” whatever the current version is.  As said, it runs reasonably well on my Apple M2-Pro Mac mini with 16GB RAM.   A Linux PC with new 16GB Nvidia card would be better but the Mac is half the price and I already had it.  Do not even try using a Raspberry Pi.  A gamer-style PC would be best.

Here is where I got my copy

Alan Timm

unread,
Apr 26, 2025, 8:40:11 PM4/26/25
to RSSC-List
Eh, no, I have not.  That's one rabbit hole that I've thankfully avoided so far.
But I'm grateful that there are people out there that live for challenges like that.

Probably the best place to start would be to purchase a copy of the book.  It looks like Sebastian lays out (a) path in 368 pages.

I also see there's a companion github (with python notebooks) that's probably worth checking out if you haven't already.

Oh, and I saw in a reddit thread that someone recommended a 43 video youtube playlist from Vizuara titled "Build LLMs from Scratch" that might be worth checking out.

I'd be interested in hearing more about the process.  Let me know if you decide to tackle it.

Alan

Subhobroto Sinha

unread,
May 9, 2025, 5:24:18 PM5/9/25
to Gmail, RSSC-list
Thomas,

How's your progress going? How do you like that book?


I'm personally extremely motivated to learn about your progress, so do share even if it's a 1:1 email instead of a group update.

Gmail

unread,
May 9, 2025, 9:08:20 PM5/9/25
to Subhobroto Sinha, RSSC-list
Hi,

Unfortunately at this point I am just doing a small amount of research. I won’t have time to dedicate to the project until perhaps mid June. I’ll keep you posted. I will document my work on my YouTube channel. 



Thomas


-  

Need something prototyped, built or coded? I’ve been building prototypes for companies for 15 years. I am now incorporating generative AI into products.

Contact me directly or through LinkedIn:   


On May 9, 2025, at 2:24 PM, Subhobroto Sinha <subho...@gmail.com> wrote:



Alan Timm

unread,
Jun 19, 2025, 12:23:38 PM6/19/25
to RSSC-List
Hey Thomas,

In case you're still looking to make your own GPT I came across this video from a few years back.  It's a 2 hour video with accompanying google collab and github repo from Andrej Karpathy.


5,859,914 views Jan 17, 2023
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video. Links:Supplementary links:Suggested exercises:
  • EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT).
  • EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a+b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you're hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of train.bin, val.bin. You may want to mask out the loss at the input positions of a+b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of +-*/. Not an easy problem. You may need Chain of Thought traces.)
  • EX3: Find a dataset that is very large, so large that you can't see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?
  • EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?
Chapters:00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespearebaseline language modeling, code setup00:07:52 reading and exploring the data00:09:28 tokenization, train/val split00:14:27 data loader: batches of chunks of data00:22:11 simplest baseline: bigram language model, loss, generation00:34:53 training the bigram model00:38:00 port our code to a scriptBuilding the "self-attention"00:42:13 version 1: averaging past context with for loops, the weakest form of aggregation00:47:11 the trick in self-attention: matrix multiply as weighted aggregation00:51:54 version 2: using matrix multiply00:54:42 version 3: adding softmax00:58:26 minor code cleanup01:00:18 positional encoding01:02:00 THE CRUX OF THE VIDEO: version 4: self-attention01:11:38 note 1: attention as communication01:12:46 note 2: attention has no notion of space, operates over sets01:13:40 note 3: there is no communication across batch dimension01:14:14 note 4: encoder blocks vs. decoder blocks01:15:39 note 5: attention vs. self-attention vs. cross-attention01:16:56 note 6: "scaled" self-attention. why divide by sqrt(head_size)Building the Transformer01:19:11 inserting a single self-attention block to our network01:21:59 multi-headed self-attention01:24:25 feedforward layers of transformer block01:26:48 residual connections01:32:51 layernorm (and its relationship to our previous batchnorm)01:37:49 scaling up the model! creating a few variables. adding dropoutNotes on Transformer01:42:39 encoder vs. decoder vs. both (?) Transformers01:46:22 super quick walkthrough of nanoGPT, batched multi-headed self-attention01:48:53 back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF01:54:32 conclusions
Corrections:00:57:00 Oops "tokens from the future cannot communicate", not "past". Sorry! :)01:20:05 Oops I should be using the head_size for the normalization, not C
Reply all
Reply to author
Forward
0 new messages