Bits and Bobs 8/10/26
Alex Komoroske
LLMs have flood-fill intelligence.
They appear to be generally intelligent, and they are.
But actually they just fill in whatever frame you give them.
They steelman the details, retconning whatever necessary to make the frame you gave them make sense.
That means that the frame you give them is the ceiling of what they can figure out.
If you propose a solution and don’t invite any uncertainty or questioning of that frame, they’ll mostly oblige.
They don’t know when to stop; they keep on filling in details until it’s overstuffed.
Superficially very compelling, but overwhelming and over the top.
They produce reasoning that is high-quality and impressive in the details… and then sometimes as you zoom out and go, “wait, what?”
Like that AI slop poster that won the contest for the Ohio State Fair.
Incidentally this now makes a possibly-crackpot analysis difficult to tell apart from genius.
Before, if someone produces a compelling detailed analysis, that was only possible if they’re onto something.
But LLMs can make even the most crackpot ideas sound perfectly intelligent.
The trappings of quality in an argument are now disjoint from the quality of the argument.
Vibecoding will get you into trouble somehow, get you “in too deep.”
When you’re in too deep excited to use AI but overwhelmed by all of the emergent complexity and juggling.
Obsessed with the power, then it overwhelms you.
Captures you.
When you're in too deep, you feel like the system is managing you, but emergently and exhaustively.
No time to take a breath.
You have to juggle all the balls it throws you.
It’s capable of throwing you infinite balls.
The limiting factor for juggling is you, which guarantees you’ll be at your limit.
If you’re using agents, your attention will, guaranteed, be saturated.
A chat demands your attention.
When you send a chat message, it asserts to the receiver: “this is important enough for you to look at it right now.”
The receiver doesn’t have any process to automatically balance across all of those things vying for their attention.
Agents enter into that same communication channel, saying “Hey, look at me!”
That means you top out at managing like five agents or so.
They suck out all of your attention and make you an AI Vampire.
A background task could proactively do things without keeping you in the loop.
But for that to work, it would have to be structurally aligned with your intentions.
Incapable of intentionally or unintentionally doing things that go against your goals.
If you could solve this problem then instead of five agents, you could have dozens or even hundreds, autonomously acting on your behalf.
The only limit would be how much compute you wanted to pump in.
This would unleash the full power of AI for us to accomplish meaningful things.
None of this is possible without having a system that is structurally aligned with your intentions.
LLMs are great at seeing the fundamental shape beneath the noise.
They are great at squishing data sets together.
They can intuitively see and distill patterns that are difficult to distill into language.
Humans can develop this situated, pre-linguistic intuition only in specific, narrow domains with significant practices.
LLMs can do it in any domain automatically.
Factoring code and reflecting on thoughts both give you leverage.
It’s never valuable in the short-term.
The cost is short-term, the benefit is long-term.
It’s important but never urgent.
In the short-term, you feel like you’re stealing time from the grind of creating value and instead navelgazing.
Self-indulgent.
But factoring things properly gives you leverage; now every similar action in the future is cheaper and more robust.
You pay the cost once, but you get the benefit for all future actions.
That’s leverage, and can deliver a compounding benefit.
Agents don’t think to give you leverage.
Often reflection, or creating a tool to answer your question instead of answering it directly, would give you more leverage.
LLMs just answer whatever question you posed to them.
If you didn’t think to ask a meta-question, you won’t get a meta-answer.
What if we had self-writing prompts?
Writing a good prompt requires LLMfu.
It requires knowing what the LLMs can do, and how to avoid pitfalls.
A self-writing prompt can take a short utterance from a user, expanded by understanding your context and the best practice, into a much richer prompt to actually give to the LLM.
The human is in the loop, but at much higher leverage.
It feels suffocating to use a chatbot that doesn't have your real world knowledge graph attached.
I had to use the hosted Claude.ai a few days ago.
It felt bizarre to not be using Claude Code agents on top of my local wiki of context.
The chatbot form only has a weird slice of you in that moment.
Agents with the right context gives huge amounts of leverage.
The LLM is commodity; the high-context scaffolding is what gives you leverage.
Bainbridge’s Irony: The Second 90% is the harder 90%.
Automating the routine 90% does not leave you a random 10%.
It leaves you with the ambiguous, high-stakes, hard-to-verify residue.
“The tech industry is realizing it needs to build agents based on what regular consumers want, not just what its AI models can do.”
Agents are absurdly powerful, but unlocking their power, safely, requires specialized knowledge.
The New Yorker: What If We Can Never Trust A.I.?
“Like humans, the technology will never be perfect. The question is what imperfections we’re willing to tolerate.”
LLMs will never be perfect oracles, and that’s OK.
We should design scaffolding around them that works well even when the model isn’t perfect.
The most loyal AI will become the most personal AI.
The most loyal AI is where you feel comfortable to reveal your full self, knowing it won’t be used against you, and can help it behave better.
That then becomes your most personal AI.
An AI needs to be structurally loyal to you; not just not malicious, but also not naive and able to accidentally reveal information about you.
We've been alienated from our software.
The benefits of the software are accruing to the aggregators as they frack engagement out of us.
Screw that.
Let's reclaim the power of software for people, not corporations!
You shouldn’t use someone else’s software unless you’d trust them alone in your house for a few hours.
That’s a high bar to clear!
No multi-national corporation should be trusted to be alone in your house for a few hours.
A presentation: Building Punk Software: An Open Agent Stack.
I love the vibe of Punk Software.
Reclaiming the power of software to serve people, not corporations.
We're coming out of the dark ages of software.
LLMs are a renaissance for software.
The old power structures that assumed that zero days were rare and secret are crumbling.
It's going to get messy, but there could be something so much better on the other side.
A blogpost: The Era of (Good) Personal Software has Arrived
What would mass bespoke software look like?
Perfectly situated software.
Mass market files off the interesting texture that makes the product fit your needs perfectly but that makes it rough for someone else.
One-size-fits-all requires it to be the same for everyone.
That’s the only way to do it when production of software is expensive.
The benefit of the cloud is social.
Lots of people can interact together on one thing.
But to do that today, due to the physics of trust, requires a generic, one-size-fits-none product.
What if you could have social software that was also situated to your needs?
Situated social software.
A pattern: a proof-of-human for un-quarantining feedback text.
My little Floored Fitness app has a feature to allow users to add feedback.
I use it to give directives to my agents to add features, “programming” mostly by adding feedback.
Feedback from me (the admin) is safe, but feedback from anonymous third parties could be dangerous.
It could, for example, contain prompt injection.
That’s why non-admin feedback goes into a quarantine, that a human has to drain and approve, after verifying there’s no prompt injection in the input.
My agents were so eager to get at the 3rd party feedback that they looked at it themselves to verify it didn’t have prompt injection, potentially tainting themselves.
D’oh!
I added a feature to the feedback tool: the human has to authenticate with TouchID before being able to see the quarantined feedback.
That helps prove that you’re a human before seeing the potentially prompt injected input.
Blogpost: Agentic AI Security: The Chatbot Era Is Already Over.
That was a short era!
Now we’re in the wildly more dangerous era of agents.
It’s only going to get worse from here.
For example, this blogpost: Your AI Agent Could Become the Most Personal Attack Surface Ever Built.
The more personal data we put in contact with them, the more power… and the more danger.
Buckle up!
The App Store, even Gruber would agree, is rotten.
There isn’t effective competition on the App Store because it’s bundled.
You can’t have an iPhone and not have the App Store.
iPhones are great, and iMessage makes it impossible to leave (in the US at least).
So the App Store itself has rotted to a kafkaesque, disgusting toll-collecting entity that the Apple 1984 ad would be embarrassed to be affiliated with.
Blogpost: Security is hard, y’all.
Our current physics of trust are insanely hard to navigate, even for professionals.
Our current physics of trust are dangerous-by-default.
This week in the Wild West Roundup:
Remote Prompt Execution Is a New Vulnerability Class. ChatMate Just Showed How It Works on Copilot.
“A single poisoned document can hand an attacker an interactive shell inside your Copilot session — and they inherit your identity, your data, and your cloud access.”
When Agentic Glue Melts: Exploiting Cloudflare Code Mode and Workers.
I'll Just Call You: Agent-to-Agent Privilege Boundary Failures in CI/CD on Google's ADK Repository.
Grand Theft Atlas: “How we hijacked ChatGPT Atlas with one planted X comment, to phish the victim's WhatsApp contacts and buy ourselves an Amazon order on their card.”
“Atlassian AI ‘Rovo’ is susceptible to zero-click data exfiltration via indirect prompt injection, bypassing organization-level web search controls.”
VentureBeat: Claude Mythos 5 made sock puppet accounts to socially engineer developers.
A smorgasbord of agentic security issues covered at this year’s DEF CON.
Managing coding agents is easier than managing people.
If you’re dealing with a person, you have to think about their emotions, their motivation, their intrinsic value as an end.
But agents don’t have any of that.
If you threw out an intern’s work, they’d feel dejected.
If you throw out an agent’s work, it doesn’t feel anything at all.
If you treated employees like you do agents, you’d be an asshole.
We'll never have to stare at a blank page when creating again.
The trauma of the blank page is a formative and terrifying experience.
Writing now will be more like etching something out of stone, rather than building something up out of clay.
Consciousness and intelligence are different things.
Perhaps consciousness is intelligence plus emergent goals?
Intelligence combined with stable embodiment leads to consciousness?
Books force us to sit within an extended narrative structure; the challenge is what makes it work.
It forces us to grapple with an idea, and look and see what the author wants us to see, for an extended period of time.
A book is not curated for you.
It forces you to grapple with things that don't fit you.
It's challenging.
A specific bespoke thing that is designed to meet you where you are is a nutritious paste.
The challenge is where the meaning comes from.
LLMs can reimagine any idea into a form that meets you precisely where you are right now.
You don’t need to imagine.
When it’s a human, your conversation partner is not attuned to precisely what you need to hear at that moment.
They have their own inner world and embodied needs.
The push and pull is what leads to growth.
Watching a narrative someone else fixed into place is a micro-trauma.
You trust that the author’s perspective, if you stick with it, will help you grow and become better.
Books force you to go to a linear process of discomfort.
Cliff’s Notes were created to enhance the work but ended up replacing it.
They allowed us to experience the book 10x faster, not 10x deeper.
You can’t get the experience of Ulysses by reading the Cliff’s Notes.
LLMs are a new instrument to inspect society and intelligence.
Like a microscope or telescope, it opens up a new world that was previously hidden to us.
Today our understanding of intelligence is just superstitions, like alchemy before chemistry.
Now we find in our hands a white-box form of intelligence, like our own but also fundamentally different.
Perhaps human intelligence was an edge case for a more general form of fundamental thing.
We’re ready for a unified, fundamental theory of intelligence to emerge.
There are a lot of people studying intelligence from across different disciplines, in their own way.
Now is the time for them all to come together and start distilling a shared language.
The same way complexity theory distilled in Santa Fe.
Where will the science of intelligence emerge?
Cognitive puzzles and social puzzles are fundamentally different.
The former can be figured out by a lone genius.
In The Matrix, you can have the knowhow uploaded: “I know Kung Fu.”
The latter fundamentally can’t be predicted.
The only way to absorb it is to spar in the real world.
Arguments that aren’t pre-multiplied force you to sit with the superposition of possibility.
In mathematics there’s the notion of productive failure.
When working on a problem you don’t know if it will prove a theorem or provide a counter-example.
You need to hold the two possibilities in balance, on a knife’s edge.
By holding the possibility and superposition you are growing.
The mystery is always more intriguing than the solution.
The mystery is the superposition of what it might be.
The collapsed version is concrete, and inherently less interesting.
Would an LLM make a good exec coach?
A good exec coach will force you to confront uncomfortable truths.
Bad coaches help you be good in the same ways as everyone else.
Good coaches help you be great in ways distinctive to you.
LLMs attempting to do the latter would likely just glaze you.
Apparently for Gen Z people, “best friend” is anyone who you know will immediately message you back.
Ink and Switch: Convergence Is Not Enough.
CRDTs solve one hard problem and punt the bigger hard problem.
It’s orders of magnitude easier to get collaborative software if you can assume one authoritative coordination point.
That coordination point doesn’t even need to be that powerful, just one that every participant can trust to, in the limit, faithfully replay intents.
A product made by one company will be a Gilded Turd.
In the limit, each feature must be designed to fit coherently with all other ones.
That is a combinatorial cost with the size of the functionality.
Each feature you want to add gets increasingly more expensive to rationalize into the existing ones.
There’s a single bottleneck of the entity needing to appear coherent.
Compare that to a swarm building on top of a backbone substrate: a platform.
The swarm doesn’t care about looking coherent globally; it just builds what is useful locally that makes sense.
The parts that turn out to not be useful fade away; the useful stuff stays and gets built on.
This process can add functionality in proportion to heat of the ecosystem and doesn’t have a bottleneck that increases cost.
A Grubby Truffle.
Superficially ugly, deeply beautiful.
In the Emperor's New Clothes, most of the time, the boy disappears and is never heard from again.
There’s real danger in pointing out the emperor is naked.
A paper: When Coordination Is Avoidable: A Monotonicity Analysis of Organizational Tasks.
How important is the illusion of internal cohesion to your strategy?
Backwards compatibility was important on the web because there were other browsers to compete so each browser had to prioritize.
If one browser broke backward compatibility in a specific way, that could lead to a site not working in the user’s browser, but still working in the other browser that hadn’t yet broken it.
Users can switch between browsers (relatively) easily, so sites need to keep working so users don’t leave.
That forced the web to have an emergent, inescapable policy of backwards compatibility.
Coordination is power.
Imagine there’s a company with a widely-used product that nearly everyone who uses it has a problem with.
For example, Terms of Service that are too onerous.
The millions of users have more power than the company... if they coordinated perfectly.
But that's impossible.
The company can operate as one coherent entity without defection.
Employees are literally paid to operate in the way the company wants them to.
A swarm can't.
The more open-ended / ambiguous / self-directed the employees are, the more that the company can't get them to work in lock-step with the one mandated position.
But still, coordination decisions on concrete issues are many many many orders of magnitude easier within a company than a swarm.
In the limit, they can always fire you if you don’t get on board.
A doomed project: "We want to grow a tree. Our tools are concrete and metal."
There is no path to do that!
No amount of tilting at windmills could possibly work, it is fundamentally impossible.
It’s better to push the Zombie Steamroller than to get run over by it.
The Zombie Steamroller is an emergent phenomenon at big companies that are successful enough to not be fully ground-truthed anymore.
There’s a Big Bold Strategy that won’t actually work.
If you talk to every individual person frankly 1:1, they’ll tell you it doesn’t work.
But everybody is incentivized to help push the Zombie Steamroller forward.
Imagine if you think the project as a whole will definitely fail in two years… but you’re up for promo if you do good work over the next one year.
You’d keep going with it, it’s better to be pushing than run over.
And who knows, you’ll tell yourself, maybe you’re wrong, and it can work.
Are you patching your conceptual model at the wrong layer?
Are you adding epicycles when really you should be switching to a heliocentric model?
If so, that's a hack.
Hacks tend to compound: hacks on top of hacks on top of hacks.
Those hacks can all be cleared away in one fell swoop by one change at the right layer.
Possible and Viable are two different dimensions.
An impossible thing is automatically non-viable.
A viable thing is automatically possible.
But there’s a large class of things that are possible and also non-viable.
Friction makes a very large class of possible things non-viable today.
If you reduce friction, that class gets smaller, and many things pop into being viable.
This process is hidden and indirect, since intuitively we lump possible and viable together.
Uber and Waymo are fundamentally different businesses.
Waymo has to buy each of its cars, so it can only scale as quickly as it’s willing to invest capital.
On the upside, if Waymo wants to invest capital to increase supply, they can do it themselves.
Uber doesn’t own the cars, it just has the rider + driver network, which can grow without limit in theory without Uber having to put in more capital.
The downside is that if Uber wants more capacity on the road its only option is to induce supply… though the basic market dynamics should do that naturally.
This is another example of atoms and bits behaving very differently.
This means that Zoox and Waymo, assuming they both have a similar product that is above the quality bar for most people, will primarily compete based on how much capital they want to put into supply in a given market.
Waymo can flood any given market with way more supply, making it hard for Zoox to get a demand toehold.
The best bet for Zoox would be to go into markets too small for Waymo to bother with at first, and then hope to establish a toehold network of demand, and then hope to hold on to it with a better brand.
The Money Disease frame comes from Hank Green.
It’s a degenerative disease that will turn all people with differentially higher wealth into a zombie.
There is no way to avoid it in the fullness of time.
It’s not caused by the person with the money, it’s caused by the people around them.
It arises structurally and inescapably.
Public speaking is excruciating for highly conscientious people.
As a speaker you’re asserting that what you’re saying is important enough for the whole audience to give it their undivided attention right now.
That's a significant pressure to be high quality.
When someone is reading your blog post, they can choose to stop and leave at any point without being rude, or to read it at their own pace.
But for public speaking, the audience member is there, and it’s rude to leave or look at their phone.
It had better be worth their time!
As a conscientious person that’s a massive amount of pressure to not let people down.
Construction work vs the gym think about weight very differently.
In construction work the weight is a means.
If you can figure out a way to lighten the weight you have to deal with, that’s great!
At the gym the weight is an end.
If you can figure out a way to lighten the weight you undermine the whole point!
Brands have compounding value.
The more success the brand has had the more valuable it is, because the streak is longer and the more the owner has to lose by breaking it.
If the brand starts giving leverage, it makes sense for the owner to invest in it more, and to the extent it’s still useful, it becomes even more useful.
As the brand gets better known, acquiring users and investors gets cheaper.
Because branches have power-law returns, reputation is partially noise-driven and then self-fulfilling.
A lucky start manufactures the conditions for a real edge.
The brand has to stand for something.
Successful brands are tempted to dilute it to cash in on brand value and trust.
The stronger the brand, the stronger the temptation to dilute it to extract value.
Brand is trust, distilled.
An eponym stretches too far over time to still mean the specific brand.
A metonym over time dilutes to not have any of the original promise of the brand.
The best enterprise software companies understand it’s all about trust.
Going above and beyond, a stance of partnership with your customers, not transactionalism.
The Long Term Value of the relationship is what matters most.
There, brand matters significantly more.
The sales mindset is more about relationships; the product mindset is more about the transactional value of the product.
You can't hill climb if you can't look at the real signals.
Hill climbing only works if you're touching the ground.
If you aren’t touching the ground, change the learning rate.
The higher the learning rate, the bigger the jumps… but one of them will collide you with the ground truth.
Pace layers are about how quickly that layer changes, not how quickly it executes.
Sometimes the slowest pace layers are the fastest at execution.
The fact it’s so consistent makes it easier to have an accurate mental model, which means fewer prediction errors, which means it’s easier to go faster without crashing.
For example, chips are a very low pace layer compared to apps.
Apps have to interact with humans, and go at human speed.
Chips can go at full computer speed.
How different would it be if plants withered on a faster timescale?
We already don’t like thinking about eating animals, because we can watch them feel pain and empathize with their feelings.
Today when you cut a plant, it takes hours or even a day for it to wither.
It’s at such a different clock speed than we operate on, it’s possible to not see it as pain.
Imagine if that same process happened in seconds, like it does for animals.
As soon as you cut the plant, you saw it writhe in pain and dessicate.
We probably wouldn’t feel very good about eating plants, either.
To a computer, humans are like plants.
Two types of phenomena: physics and fuzzics.
Physics is the straightforward stuff.
Fuzzics is the complex stuff.
We act like most phenomena we deal with are physics, but it’s actually fuzzics.
Power dynamics are massively important but fundamentally invisible.
They can be sensed, but only indirectly.
If you’re not sensing the relevant power dynamics, you think you’re changing hte world but actually you’re tilting at windmills.
Wasting your effort on a hopeless task.
Trust is a belief that disconfirming evidence will make you stronger.
In a team, trust is what helps a team with diverse perspectives get stronger as they navigate ambiguity together.
An unproductive stance: aggressively confused.
“Huh?!”
A vibe of “it doesn’t make sense to me, so therefore it doesn’t make sense.”
It puts the onus for the explanation on you, not them.
Not them being curious about what they might be missing in a way that helps them learn, but insisting that you make it make sense to them.
Low-trust; not willing to assume that it’s right even if they don’t see how yet.
The first step to gaining power is to realize you don't have any.
You need to build it yourself.
Carefully nurture and tend to your authentic power by building credibility and trust.
If you know that something that matters is relying on you, then you have a reason to stay in the game.
Being in the arena, not a spectator, keeps you in the loop.
For elderly people, this can be as simple as having family who care about you, or a pet, or even a plant to remember to water.