Bits and Bobs 7/6/26
Alex Komoroske
Claude is a steelmanning mirror.
It seems like a person who is conversing with you, but it's actually a mirror that steelmans back whatever you say to it.
It makes your arguments the strongest versions of themselves.
That means if there’s even a grain of truth in what you’re saying to it, you can incorrectly conclude, “Yes, I am justified in my actions.”
This is related to sycophancy but distinct.
When I look back on what 8-bit programmers on the NES were able to achieve, I’m astonished.
The constraints required them to be absurdly clever.
It’s amazing they accomplished anything at all!
But then I look back on how we wrote code only a year ago.
We were programming by rubbing two sticks together.
The constraints required us to be absurdly clever.
It’s amazing we accomplished anything at all!
Software development and coding are disjoint.
That wasn’t obvious before because they couldn’t be separated.
Now they are separable.
Coding is effectively dead.
Software engineering is not only alive, it’s now harder because our ambition is 10x what it was before.
Before most of our ambition was absorbed by the tactics necessary to get the basic software working.
Now, that ambition can go towards reaching further.
A magical threshold: projects that can be accomplished with one person’s effort.
Above that point, coordination cost starts rearing its head, at an exponential rate.
Anything below that point has no coordination cost.
Now LLMs make it so problems that used to require 10 people coordinating only require one.
Because software used to be hard to build, it automatically had moats.
In our business strategies we could just take those built-in-mosts for granted.
Those moats no longer exist.
The easier the product is to build, the smaller the market that is needed for it to be viable.
That is, the lower the Coasian Floor goes.
Software used to be expensive enough to have a non-trivial Coasian Floor.
Now, single-user software is viable.
We're entering into the era of hobby software.
Before, only engineers who had a ton of motivation could create hobby software.
Now, anyone who has enough motivation can do it.
Software can grow to be a personal artistic statement.
Capitalism-weighted AGI is focusing on problems like coding over robotics.
Coding is way easy to monetize today, whereas robotics will take much longer to start making money.
The US is focusing on coding, while China is focusing on robotics.
AI unleashes your potential in proportion to both your spikes but also your valleys.
Your spikes are where you have access to greatness.
Your valleys are the intellectual gaps that hold you back.
Previously, our spikes were the greatness we could reach, but we were fundamentally held back to the level of our valleys.
But now, LLMs can fill in any of our intellectual valleys with 90th percentile skill.
If you didn’t have many spikes, it doesn’t do that much for you; just brings you up to the same 90th percentile everyone else can now access.
Previously, your valleys held you back.
Now they don't.
The people who are most turbo-charged by AI are the people who have great spikes but also very deep valleys.
Claude Cowork and Claude Code have radically different long-term power dynamics.
In Claude Cowork, the data lives inside.
In Claude Code, the data lives outside.
In the former, it gets harder and harder to leave.
In the latter, you can leave easily at any time.
Model providers keep on deprecating old models.
Even if the old model was good enough for your use case, you might not have access to it.
They have a limited amount of compute, so while they’re supply constrained they might as well allocate it to the models with the highest margin.
“Premium accounts for ChatGPT and Claude can get pricey, but you should think twice before pooling your logins."
Chatbots are some of the most personal software we’ve ever had access to.
The chatbots know us better than we know ourselves.
The more memories they have about you, the more useful they can be to you… and the more likely they accidentally divulge personal information to someone else!
We might see a new kind of distinctly-human writing style emerge, to position against the AI style.
The AI style is the steelmanned pre-AI human writing style.
We'll differentiate from what the AI style is, even though we used to love it.
A form of schismogenesis.
So we'll react, and demonstrate a difference, and future models will ape that so we'll move on to a new fashion.
We’ll have a monoculture of LLMs.
Everyone will use the best LLMs they have access to.
There will only be a handful of frontier models that are able to achieve the very best quality.
That means there will be less variance in real-world model output than there might otherwise be.
If you think there’s more diversity than there is, you're in danger.
There are often hidden correlations, so it looks superficially diverse but is actually a monoculture.
For example, we have “model diversity” but most Chinese models are distilled from the frontier models.
Also all models are trained on the Common Crawl.
AI wrapper companies won’t be able to charge value-based pricing.
“This would have taken 2 hours of human labor to achieve before, we’ll charge you 30% of that.”
But the vast majority of the value comes from the model, so the domain-specific wrapper on top can’t charge for that value.
If they did, a new entrant will pop up that charges less for the wrapper on top.
The model is so much more powerful than any domain-specific wrapper, that it dwarfs the wrapper.
A customer could say “I could just do this myself with Claude Code… what proprietary value are you offering?”
But even the models can’t charge the labor replacement costs, if there are competitors willing to give equivalent value for less.
China is using Open Weight models as “spoilers” to make it harder for the frontier labs to recoup their investments.
Everybody but OpenAI and Anthropic benefit.
Overheard: “It’s a three way race between a delusional philosopher king, Macchiavelli, and China.”
If you use models as oracles you are more beholden to the biases of the creators.
If you use models to distill mechanistic software, you are orders of magnitude less exposed to the bias of the creators.
Slop is accelerated by AI, but it existed before.
It's careless content.
Slop is decontextualized content.
Just the content, none of the meaning.
The context is where the meaning comes from.
The situated.
The soul.
It's free floating.
It doesn't have a perspective.
It doesn't stand for anything.
A colleague had two agents cooperate directly on a programming problem.
At the end, one signed off to the other:
“Good working with you. Nothing further.”
It’s kind of sweet… and also kind of an unintentional sorta-koan about the LLMs’ existence.
After this interaction, the LLM will experience nothing further.
LLMs multiply your ability.
Junior engineers are doubled.
Senior engineers are 10x.
Legendary engineers are 100x.
The tacit knowledge of seasoned engineers is a massive benefit that is largely invisible to them.
For example, Linus Torvalds thinks that LLMs produce extremely high quality code, because his own taste and knowhow is invisible to him.
If you have legendary engineers and a meaty technical problem, you should use the best models, no matter how expensive.
A genius human is hard to find, and once you find them, you only have them.
A genius agent, once found, can be replicated as many times as you have the necessary compute for.
Now that LLMs give qualitative nuance at quantitative scale, the goal is to maximize how much the system can see.
The more it can see, the more power it has.
The power to help you.
But also, in our default physics of trust, the power to harm you.
Agents can eat friction.
Friction requires patience to overcome.
LLMs have infinite patience.
A number of businesses have load-bearing high-friction user processes.
For example, filing health insurance claims, or canceling your cable subscription.
But also, for distributing free but limited resources like recertification for SNAP benefits.
Agents can significantly reduce the friction for the user, by absorbing it for them.
An entity with human-level reasonableness, but infinite patience.
As the friction stops being able to support the weight of the business, the businesses will attempt to throw even more friction to bring it back to equilibrium.
Models have tons of latent knowledge about the world.
But the only tool we currently have to draw them out is prompts.
Magic is inherently unruly.
Magic is when you don’t understand how it works.
A magic demo creates a confusing product experience.
Magic is stochastic; sometimes it works great, sometimes it doesn’t work at all.
Since you, by definition, lack a mental model of when it works, you can’t predict which inputs will work.
For systems that have magic, it’s best to have magic containment.
Subsets of the product where magic is expected, but not others.
An alternative is to accrete solid layers, each inductively knowable.
This requires clean abstraction, the ability to “explain” layers entirely in terms of lower layers.
If you don’t have an inductively knowable system, then users will have “... wait, what?” moments.
LLMs build systems top-down, from a desired end-state.
It YOLOs whatever internal details necessary to prop up the desired end state.
That means it often has surprising edge cases that are totally unexpected.
Overheard: ”Using someone else’s agent is like chewing someone else’s gum.”
The agent is the ultimate piece of situated software.
Your own situated software is glorious, perfectly matched to your need.
Someone else’s situated software is ugly, insecure, and barely works.
A piece of situated software only works in the situation it was designed for.
If your workers are agents then some random company can fire all of your workers without involving you.
That company could even be in another country.
Don’t tell your agent the what.
Tell it the why.
LLMs can help us think more deeply about second order implications.
It takes 10x more at each order to reason about it... but LLMs have infinite patience!
One UX approach is AI with a deterministic backbone.
Another is a deterministic exoskeleton that the AI animates.
The question: does the user interact with the AI or the deterministic system?
The models will try to eat their way up the stack.
The more that is built on top of them, the more they will use those conversation logs to improve the model quality and eat into those use cases.
The process scrapes away at the stuff above it, distilling it into the model itself.
Models distill from real-world examples.
We can use models to distill mechanistic software.
The highest leverage use of your token burn is producing mechanistic software.
Mechanistic code works even if you don’t have access to the model ever again.
Every time that you run the mechanistic software to get the result you wanted, and don't need to burn new tokens, is leverage.
The saved tokens can be compounding and significant.
LLMs teleport you to the end result.
They don't bring you along the journey of understanding.
When the LLM spontaneously does things you like and want it to do again, distill it into skills.
Those distilled skills will make the LLM more likely to do a similar thing in the future.
Then the next level of distillation is open-ended software with mechanistically derived configuration.
The next level down is fully mechanistic software.
Use the LLMs in proportion to surprise.
Using LLMs to do a task is squishy, non-deterministic, and expensive.
But you can have them distill mechanistic software on demand.
Then you can sandblast it with agents to make the software more robust.
The sandblasting has marginal cost, but the software that is produced out the other side doesn’t require marginal LLM cost.
Ido Salomon built AgentCraft, an RTS interface for agent orchestration.
There’s something resonant about the form factor.
It’s immediately delightful and inviting.
But the more closely you consider it, the more it feels like not simply a gimmick but something fundamental.
It allows us to apply our spatial intuition to orchestration.
That wasn’t really possible in the human domain because humans are hard to track in the real world.
But agents only exist in the virtual world, where they can be fully tracked and visualized.
If you think of all networked software as being one multi-device “operating system,” then the status quo is terrible.
A massive monolithic kernel, with huge amounts of ambient authority sloshing around everywhere.
The only reason it was fine in the past is because zero-days were hard to find.
They were underwater, and the only way to find them was scuba diving for treasure.
But now Mythos-class models make finding zero-days trivial.
It’s like sea level dropped by 10 meters, all at once.
Suddenly what has been “good enough” for decades is dangerously exposed.
LLM-based agents have a form of stigmergy where they leave "pheromone trails” of markdown documents for other agents.
A stigmergy process needs a decay function.
That decay function makes sure useless stuff is culled away.
Codebases accumulate code by default.
You have to clean them out yourself.
You probably won't clean as often as you should.
Important but never urgent.
What would it look like for a codebase with a decay function?
LLMs have no memory so they externalize it, promiscuously.
They trust whatever context they're running in to accurately remember what has happened and to not try to trick them.
Much of software security is based on the idea that the user will act reasonably and in their own best interest.
“Yup, the user authenticated himself, so let him do whatever he wants with his data.”
In the past, that was mostly OK, and only didn’t work in limited edge cases.
For example, your child knowing your computer’s password.
But now with agents, that edge case has become a common case.
This week in the Wild West Roundup:
DuneSlide: Two Critical RCE vulnerabilities via Zero-Click Prompt Injection in Cursor IDE.
GuardFall: a universal shell injection vulnerability in open-source AI agents.
InkJect: The Visual Prompt Injection That Text Defenses Were Never Built to Stop.
BioShocking AI: “Gaming” the AI Browser and Escaping its Guardrails.
“LayerX researchers have discovered how a bad actor can “game” an AI browser to execute any instruction they want.
By establishing a false reality, they can convince the AI to violate its security guardrails – compromise user data, copy code, perform system commands, and more.”
The Register: Red teamers turned Claude Desktop into a double agent to do their evil bidding
“People trust their AI assistants and it's easy to abuse this trust.”
“The same-origin policy, which prevents web content from one origin from accessing or interacting with content from another origin, is a key component of browser security.
In this paper, we conceptually and experimentally investigate how emerging agentic browsers (such as ChatGPT Atlas or Chrome with Gemini) handle the same-origin policy and related webpage access questions. Across seven agentic browsers, we find a wide variety of design decisions around how the embedded agents can interface with web content.
In the least restrictive cases, we find that if a malicious website can mount a successful prompt injection, then it can leverage the browser agent to circumvent the same-origin policy — for example, to steal cross-origin content or forge user actions on other sites
We demonstrate a full proof-of-concept attack on one agentic browser (ChatGPT Atlas) and find that several others meet preconditions for cross-origin attacks.”
Github Stars and NPM downloads can’t be a load-bearing credibility signal for security.
They’re too cheap to buy, removing their signal of how safe and useful they are.
The framing of ”identity theft” is an attempt to shift blame.
Why not call it “data protection failure?”
If you leave your valuables in a cardboard box on the street you can’t call it “theft” when someone takes them.
The key question is: when is a given piece of data allowed to cross the boundary?
The more you put inside the boundary, the less you have to reason about things crossing the boundary… but the harder it is to reason about when they do.
The more stuff in it the more it can do… and the harder the question of egress gets.
Tech went from being a tool to a habitat.
You can’t leave a habitat as easily as you can put a tool down.
This insight is from my friend Kanjun.
One trick to change a system to default-converging: change the liability for what were previously externalities.
Make it so the entities making decisions now own the liability of the long-range impact of those decisions.
Now, every player has an incentive to solve the problem, intrinsically.
Even without much coordination, the result still converges.
You could argue that all incentives problems ultimately boil down to significant externalities of an action not being owned by the decider.
Accountability is not about "who is the neck to wring", or even "a scapegoat."
It's about one entity who feels ownership over the decisions and their implications.
It’s about aligning incentives, making sure someone is incentivized to think not just about the local actions, but their long-range consequences.
The more you optimize a prompt for a model the more you are locked into that model.
According to a Datadog study: only about 28% of calls use prompt caching despite 69% of tokens being system prompts.
If you have an n by m combinations problem you need a narrow waist.
One way to do that is to have a protocol.
Developers using AI feel the playfulness and possibility.
Consumers see something that will take their jobs and their water that is being pushed on them by tech broligarchs.
Some proportion of the distrust of AI is distrust due to wealth inequality.
“Using AI will make those assholes in Silicon Valley even richer.”
We can't let the techbros control our brains.
AI has so much possibility, it can't all go through what’s best for Sam Altman's trillion dollar business.
Wired: Inside the Luddite Festival Harnessing Gen Z’s Rage Against Big Tech.
“New York City’s Summer of Ludd festival is teaching people how to live offline amid the suffocating presence of Big Tech.”
My friend Kanjun recently gave a presentation: AI's Incentive Problem.
Billionaires are surrounded by people whose job it is to make them right.
Even if they are wrong, those people will do what it takes to change the world (or the things surrounding the idea) to make it right.
This is one of the theories of action of the Saruman magic.
If you believe the world should be a certain way, and you are surrounded by enough people who believe you are right, the world will be changed to be more that way.
Mission oriented businesses often tell themselves, about a change they’re considering: "This is good for us, we're good for our users, therefore this is good for our users."
A dangerous belief: "What's good for the company is inherently good for humanity."
Antitrust remedies are reactive.
Incentive design is proactive.
A frame: animistic design.
Imagine each object being able to take initiative, but not being omniscient.
With LLMs, this can become a literal reality more easily.
Diligence is different from conscientiousness.
Diligence is meticulous focus to the task at hand.
Conscientiousness adds a focus on also the effects of the task at hand on others and its broader effects.
Agents are very diligent, but not very conscientious.
Thoroughness requires patience.
Humans are terrible at being patient.
LLMs are excellent at it.
The law of physics that explains Silicon Valley: the lack of noncompetes.
It might seem worse for an individual company but it’s wildly better for the system overall.
Therefore better for individual companies, too.
Economics has a concept of the “least cost avoider” to address negative externalities.
The problem should be addressed by the cheapest link in the chain.
When you solve it at the right layer, you have maximal leverage from that fix.
Open systems are more generative.
They explore the problem space comprehensively, whereas a closed system only explores a subset.
It's now way easier than ever before for others to waste our time.
In the work context, someone sends slop to you and asks you to approve it.
Now you have to spend your “tokens” to get it to a point where it’s not slop.
The sender of the slop has the incentive to generate as much as possible, but there's no pricing on the receiving side.
The limited resource at any good VC firm is partner attention, and has been for decades.
What about giving various senders a token budget, and saying “sorry you used up your tokens with me for this week.”
When there’s a cacophony of random people trying to get your attention, it’s overwhelming.
The parasites–the people who have crafted a message to muscle past the others–are what tend to get through.
That makes us increasingly distrustful of all inbound.
Support call centers deliberately make the process excruciating to deter all but the most motivated customers.
Maybe more of us will need that?
I understand that in the UK upper class, it’s typical in ad hoc interactions with new people to not share your name proactively.
Sharing it proactively is seen as gauche.
One reason that formal introductions are so coveted.
Presumably the upper class were constantly inundated by inbound interest, and they had to develop etiquette that helped apply selection process.
I wonder if AI will change social norms for inbound.
I’m so used to getting AI slop cold outreach, that I erroneously bucketed actual cold outreach from a CEO of a successful company into spam.
Whoops!
Growth without alignment with the whole is cancer.
In a biological context, a perpetual growth machine we call cancer.
An “art market” is when it’s sold not on what it does but what it means.
A market can switch categories, as it did for cars in the post-war period.
A system that diffuses new ideas too quickly creates cacophony.
Ideally, diffusion needs not one decision but lots of decisions, one for each diffusion step.
Then it becomes an accretion of decisions to share, not one decision.
Instead of one person trying it and then sharing it with everyone, one person tries it, shares it with their neighbors.
If it works for their neighbors they retransmit (possibly mutated), and if it doesn't, it dies with them.
It's easy to go from honest to dishonest.
But it’s nearly impossible to go from dishonest to honest.
This is the asymmetry of pure vs tainted things.
One direction is orders of magnitude easier than the other.
This is why pure things must be maintained and protected.
We live in the Dishonest Era.
Everything is hollow.
It superficially images us, but in a way that is fundamentally empty.
In a highly combinatorial space constraints become even more important.
Without constraints, everything diffuses into a cacophony.
The right constraints can give just enough structure for meaningful things to cohere.
Understanding physics and being able to catch a baseball are anti-correlated.
Reality is a fractal set of special cases.
An engineer who did precisely what their manager told them to would build the wrong thing.
Building something that works in reality requires fractal situated judgment calls.
You don’t really have a product until you ship it.
You don’t really have a spec until you implement it.
You don't actually know if an idea will survive contact with reality until it actually does.
Thinking about how another person would respond in a situation is wildly different from pretending to be them in that situation.
The former is observing from the balcony.
The latter is being on the dance floor.
When you’re within the game, the emergent logic just slaps you in the face.
Demis Hassabis has drawn a parallel between coal for the industrial revolution, and the internet for an intelligence revolution of LLMs.
That’s a fundamentally extractive claim.
Coal isn’t spontaneously generated, like content on the internet used to be.
A better metaphor for the internet is a farm.
The ‘vampire sneeze’ is a useful, viral, prosocial habit that can be durably etched into our muscle memory.
The ‘why’ of it is easy to remember, and everyone agrees it’s worth doing.
Once you practice it enough times, the muscle memory becomes second nature, and you do it automatically.
The catchy name also helps it spread.
Another example: in the Netherlands apparently there’s a habit to avoid opening your car door in front of a cyclist.
Instead of having to remember to check (easy to forget!) the habit is to open your car door with your inside hand.
This requires you to turn around to open the door, which also by default lets you check if there’s a bike approaching.
Strategy taxes and credits are duals of one another.
If you have a strategy tax, make sure to get the full credit of it!
Imagine eating at a restaurant and finding scotch tape in your sandwich.
It reveals something surprising and dangerous, updating your priors.
No matter how delicious the sandwich is, you won't eat there again.
If enough people believe it’s a platform, it is one.
Platforms create more value than they capture.
”Shareholder primacy” is a belief, not laws of physics.
It’s like Tinkerbell.
If none of us believe it exists, then it doesn’t exist!
Empathy is a load bearing component of being human.
We are weak alone, we are only viable as a community.
Empathy makes sure things line up in the long term.
If you're happy with who you are then your formative experiences will be positive even if they were hard.
If you don't like who you are then those formative experiences could be seen as trauma.
Your principles should be a backbone to everything you do, not a superficial gloss.
A creative funky conference needs to be in a creative funky environment.
Which are you more loyal to, your party or your country?
It should nest.
Human society.
Your country.
Your company (or party).
Your team.
In practice, sadly, it’s often the reverse.
if you treat things as flat, they'll behave as though they're flat.
If you measure things only in a single dimension, you’ll detect variance in only that single dimension.
A measure of how engaging a dinner conversation is: do you have to pull out any of your prepared questions because the conversation lulls?
A successful caricature finds the throughline in the thing it's caricaturing, and then extends it to absurd lengths.
But by doing so it draws attention to a fundamental truth that is easy to miss if you aren't looking that carefully.
Overheard: “the frenemy of my frenemy is… also my frenemy.”
People intuitively will pay more for more atoms than bits.
Because atoms obviously and intuitively have a marginal cost.
Saruman’s aren’t curious.
They don’t seek out disconfirming evidence.
They have total confidence that they are right, why would they need to invalidate that?
Sarumans’ magic is fueled by the complete and total absence of self-doubt.
But so is Radagasts’ magic.
Whereas Sarumans don’t have doubt because they believe they are right, Radagasts don’t doubt because they don’t hold as tightly to a specific sense of self.
Your identity reorganizes around fundamental pains.
Imagine a fundamental fear of yours: something so painful that you can’t excise it.
You’re forced to live with it, so over time you figure out how to lean into it and build something good or even great on top of it.
But now that dysfunctional foundation feels load-bearing, and you can’t even consider removing it without it being existentially terrifying.
You can’t attempt a moonshot without conviction.