Attn. VIG SIG: I managed to get over 80 tokens per second from my local 4 GB LLM!
5 views
Skip to first unread message
Thomas Messerschmidt
Sep 8, 2025, 5:27:53 AM (13 days ago) Sep 8
to hbrob...@googlegroups.com
With a small but powerful change to my LLM code, I went from about 10 tokens per minute on my laptop to over 80! 8 times faster? YES PLEASE!
I also set up my computer for object identification /classification (Owlv2) —and it works! And it’s FAST as well!
And now EVERYTHING is local—I no longer need an internet connection.
(And, yes, I fixed the bug stopping my LLM to Arduino code from running.)
We can discuss this more at our meeting on Thursday.
Thomas
I also set up my computer for object identification /classification (Owlv2) —and it works! And it’s FAST as well!
And now EVERYTHING is local—I no longer need an internet connection.
(And, yes, I fixed the bug stopping my LLM to Arduino code from running.)
We can discuss this more at our meeting on Thursday.
Thomas
Reply all
Reply to author
Forward
0 new messages