Bits and Bobs 8/4/25
Alex Komoroske
I see three seeds of massive possibility in the era of AI, but each currently with a low ceiling.
MCP shows the power of integration of data.
However, the lethal trifecta sets a low ceiling; the more you integrate with powerful tools, the more dangerous prompt injection gets.
Vibe coding shows the power of infinite software.
But vibe coding still requires you to think like a PM–still too high a barrier for most people.
Also, the same origin paradigm requires users to trust the creator of the code in an open-ended way.
An OK assumption if software is expensive and takes significant investment, because the creator has a lot to lose.
Not a good assumption if some rando can sneeze and produce software.
Infinite software in the same origin model are tiny little isolated universes.
The same origin paradigm is the wrong distribution vehicle for infinite software.
Chatbox UX shows the power of open-ended UIs.
But it’s limited to walls of text.
GUIs are lindy for a reason!
UI teaches you how to use the thing, by showing affordances for what you can do.
UI helps us keep track of structured information over time in a way that drafts off our inherent spatial reasoning.
The future of software will be a blossoming of those seeds, with the ceiling somehow transcended.
As code shrinks, do you get micro apps or do you get data that comes alive?
I think ChatGPT has over-rotated on single-player chat.
Chat as the primary UX, instead of just another feature in an interactive UI substrate.
Chat is single-player; it bounces between the user and the assistant.
Ping-pong interactions.
It’s unclear how to even allow multiple people to participate in a chat with an assistant.
Vibecoding can lay down code but not do self-directed refactoring well.
What structure you lay down you're stuck with.
So each layer you lay down, you have to make sure it is good to accrete on top of.
Very hard to evolve and fix later.
Why do hyperscale products become one-size-fits-none?
A hyper-scale product wants to get as many users as possible.
As they chase low-intent users, they dumb down the product.
They do this until it reaches an equilibrium where it’s minimally useful (better than the alternatives) to the maximum number of people.
A one-size-fits-none product.
This is building on Ivan’s Tyranny of the Marginal User.
The "there will be no more entry level jobs in an era of AI” assumes a static distribution where people behave as they do already.
But as Simon points out, "humans have agency."
The world is not a static distribution; it changes in response to how things evolve.
This is what makes complex systems so hard to predict.
But it is also why obvious dystopias often fail to materialize; a counterbalancing force emerges automatically.
For example, Shopify is now hiring more interns since they can give so much more leverage.
Val Town asserts that vibe code is legacy code.
This tracks to me: legacy code is code that works but that you didn’t write and don’t understand.
If you were to refactor or improve it it would take some time to “get in the loop” with it.
It works, but it’s not pretty.
Vibe code by default has the same basic dynamic.
However, vibe code can be created that doesn’t have this problem.
If you think carefully about guidance for it and specs, then you have influenced it, just at a different layer of abstraction, and do understand it at a high level in a way that didn’t apply to raw vibe code.
A friend on a future of slop:
"I don't think consuming slop is going to be good for us.
Most people will like it, for the same reason most people like tabloids, fast fashion and fast food.
But a diet of slop will make it even harder for humans to generate the kinds of novelty that demonstrates their differentiated value in a world of AI."
An observation from a friend:
"The AI labs have the potential to become the apex predators of capitalism."
Don't let LLMs make load-bearing security decisions.
A striking study of LLM security issues:
"We deployed 44 AI agents and offered the internet $170K to attack them.
1.8M attempts, 62K breaches, including data leakage and financial loss.
🚨 Concerningly, the same exploits transfer to live production agents… (example: exfiltrating emails through calendar event) 🧵"
A prompt injection technique that hides the injection in legal boilerplate in the terms of service.
Drafting off the fact that no one reads that anyway.
We’ll see many other social hacks.
A new class of LLM attacks: “man in the prompt.”
Trail of Bits releases some mitigations for security issues in MCP.
However, superficial layers on top of a fundamentally insecure base actually makes it more dangerous.
It lulls you into a false sense of security.
The Gemini CLI had a massive vulnerability around allowlisted commands.
The allowlist matching performed inadequate parsing.
It extracted only the 'root command' without validating subsequent commands in pipelines or command chains.
My friend Varun Godbole on “What is an AI-first software company?”
Bonus points for using ‘nebulosity’ as the core frame!
An insightful take from Grace Shao in Exponential view on the US and China in AI:
"The real split is over where each country’s tech companies believe the profit will land. China bets on applications; America bets on the model itself. Palo Alto obsesses over model‑led destiny: ever‑bigger parameters, safety benchmarks, and a near‑cult-like obsession in the pursuit of AGI. As Karen Hao recounts in Empire of AI, Sam Altman and his peers see AGI as a world‑changing force capable of solving humanity’s most significant challenges. The wager is long, deep‑pocketed and proprietary: pour venture billions into loss‑making models today; own the platform that reorganizes industries tomorrow.
China’s approach is more pragmatic. Its origins are shaped by its hyper‑competitive consumer internet, which prizes deployment‑led productivity. Neither WeChat nor Douyin had a clear monetization strategy when they first launched. It is the mentality of Chinese internet players to capture market share first. By releasing model weights early, Chinese labs attract more developers and distributors, and if consumers become hooked, switching later becomes more costly.
Entrepreneurs then have the opportunity to utilize these models as free scaffolding. Taking the EV industry as an example, over twenty Chinese automakers, including BYD, Geely, and Great Wall Motors, have integrated DeepSeek into their in-car AI systems to enhance smart assistants and autonomous driving capabilities. In healthcare, it is said that nearly 100 hospitals across the country have now integrated DeepSeek for medical imaging analysis and clinical diagnosis support. Every new integration expands the model’s footprint, tightens switching costs, and shifts margins to the services sitting on top."
Geoffrey Litt says Enough AI Copilots, we need AI HUDs.
A Heads Up Display (HUD) gives you filtered, relevant, real-time information.
The human is still in charge, they’ve just got some assistance.
Though the HUD still has to be trusted to pass on accurate information and do a good job curating the information the human needs.
If the HUD is not aligned with the human’s interests, you get something like the Hyperreality hellscape.
This week I heard a story about the hollowness of current corporate customer service.
A customer was on the phone with a customer service rep (a real human) who was following the script closely.
The customer, frustrated, said, “Please don’t give me the AI, I want to talk to a real human, please.”
Even if it’s a real human, if it’s within a strict system, it can feel just as inhuman.
Just because a system has humans in the loop doesn’t mean it isn’t inhuman.
A psychiatrist shared his thoughts on ChatGPT psychosis:
“1. This is not schizophrenia.
2. Psychiatric disorders rarely appear out of nowhere.
3. What we may be seeing is a kind of digital folie à deux.
4. What we urgently need is careful, context-aware research.
5. One of the most interesting questions is: what happens when you take the AI away?
6. Any attempt to say “this isn’t a social interaction” is disingenuous.
7. Psychiatric insight is glaringly absent from AI safety.
8. There’s no reason not to build in safeguards.
9. We should avoid moral panic. We should avoid complacency.
10. This isn’t just about psychosis: there may be implications for other mental illnesses.
11. We need better research and better funding.
12. We need to consider what AI will look like in the future, not just now”
Another example where the big alignment problem has taken most of the energy, but the little alignment problem is also important in practice!
This joke tweet shows why ARR is the wrong metric in the age of AI.
"I'm excited to announce that I'm starting my next company 🚀
We give customers $100 and they give us $10 back
We're going to be the fastest company in history to hit $100M ARR
Just need to raise $1B
Who's in?"
ARR being the most important metric assumes zero marginal cost.
That was a good assumption in the world of Saas.
But it’s a bad assumption in AI!
People get tricked by asking the AI "why did you say that?" and thinking it gives a good answer.
It doesn't!
It can’t!
It's a category error.
It can't introspect its own thinking any better than you can.
It's just very good at coming up with plausible answers on the spot.
It's an amazing retconner.
Geoffrey Huntley: LLMs are mirrors of operator skill.
I love the Radagast energy in Code is a Joy:
"Joyful artistic technology is humane, useful, interesting, accomplished and occasionally it may be profitable and world-changing.
It is not something that will ever be produced by AI, nor can it be produced by the apparent inhumane factory conditions that big tech firms desire to turn their programming operations into."
"Never forget that the "techno-optimists" have no room, no time, and no interest in that joy.
All they have is cold equations for someone else to execute for them--over and over, ever faster, at ever greater scale--for pay, for a living, for food, for healthcare, for an opportunity to create art outside of them.
The cold equations they are unwilling to execute themselves but all too willing to grind us through.
The programmer is nothing but a process, ground through a function, into so much pulp for the fires of VCs and billionaires' hopeless and heartless worlds."
In a hobby context, code can be an end in and of itself.
Well factored and precisely structured.
But in instrumentalist constexts, code is a means, not an end.
The more precise it is, the more ossified it becomes, the harder to adapt it.
Even if a time traveler told users in 1990 about the web and how important it was in the future, people still couldn't have understood it.
You had to use it to understand it.
Privacy in the same origin model requires close-endedness.
But in a different model it should be possible to have privacy and open-endedness.
The original sin of the same origin model is fusing data to apps.
And then putting the app in charge.
All of the negative consequences are downstream of that decision.
The same origin model has been amazing, but in the era of infinite software we need something to complement it.
In infinite software, instead of expecting fixed, static software like today we'll instead expect software to be malleable, adaptive, to feel almost like it's alive.
For a new world-changing substrate for software you need two things:
1) That early adopters can build software for their own needs easily enough.
The expected benefit is greater than the expected cost.
Claude Code is so good that this one can be taken mostly for granted.
2) That the work early adopters do also helps other people in an indirect way, too.
That is, that it’s concave to collaboration.
This property is very hard to create, and no current solutions today have this property.
Hyper aggregation is the only way to do horizontal use cases in the same origin paradigm.
People like horizontal use cases, where their data can show up on demand in the places they need it.
But the only way to do it in the same origin paradigm is hyper aggregation.
I'm not saying "the same origin model is bad and should go."
I'm saying "the same origin model is one particular approach that has a number of massive positives, but also a number of negatives that we don't even acknowledge because we never realized we could have complementary origin models too!"
The same origin paradigm has a cold start problem for every app within it.
Since every origin starts with no data.
But that's not a law of the universe, it's just a law of the same origin paradigm.
Another model that allows safely using data from other apps could significantly mitigate that cold start problem.
In the same origin model you have to trust code.
But code is open ended!
Instead, trust a closed set of policies on your data.
That then allows open-ended possibilities without open-ended trust.
Apps are cages for your data.
Structure, but not optional structure.
What would it take to have self-distributing software?
The silicon valley playbook is all about distribution, not tech.
That’s because distribution is the bottleneck in the same origin paradigm.
The only routinized 0-to-1 playbook in the valley is for things with no technical innovation: vertical saas.
Simply a matter of execution and selling.
The normal VC model is to do a single business need well.
If you combine a couple for a strategic play, it's a three legged race against two-legged sprinters.
Starting two unrelated cold-start things tied together is super-linearly less likely to work.
A friend said this week: "The web, as a jungle to be navigated, and which surfaces wonderfully fun and quirky things, is dead.
It’s now just an interstate between large shopping malls and arcologies which are rapidly depopulating and filling with bots."
In the same origin model you have to trust a stranger's incentives.
This is impossible to do, and if you do it you open yourself up to significant tail risk.
The right architecture calls out the best of us, by default.
The wrong architecture brings out the worst in us, by default.
The right architecture you don't need to trust.
The more you trust a system, the more you can get out of it.
But also the larger the danger if it abuses that trust!
Technology today is like a TED talk.
Someone with something hyper polished talking at you from the stage.
One way, not participatory.
We live in an era of Hyper.
In an age of cacophony, only the things that stand out from the noise can be noticed.
It’s a steady selection pressure for the most.
In a cacophonous world, all that matters is appearances because no one has time to sense the fundamentals.
This is why in the industry today often The Hype is the Product.
We live in a world of hollow tech.
I want a world of resonant tech.
Hollow things are only superficially resonant.
Over time as it becomes more efficient it becomes ever more superficial.
Ever more hollow.
Resonant things connect on multiple levels, deeply.
Resonance is adjacent to harmony.
What does it mean to be in a healthy, positive sum relationship with technology, both at the individual level and the society level?
Remember the fight against DRM?
Now we’ve landed on an equilibrium where we all just rent all of our content forever.
It’s convenient, but feels hollow.
The content can be more easily taken away because you never actually had it.
A question: can your ebooks be passed on to your heirs?
Even if they can, the account costs an ongoing fee to continue having access, so if you stop using it you can’t access it.
Compare that to physical media, where as long as you have the physical storage space to hold it, you have continued access to it with zero marginal cost.
Superficially looks the same, but one is way more fragile than the other!
Technology is so powerful, it tends to overshadow the other parts.
What would humble technology look like?
We live in a world of parasitic tech.
What would symbiotic tech feel like?
Intentional does not mean "pre-planned".
It means being reflective.
Thinking about how your actions align with your aspirations.
Intentional tech is not about figuring out some global alignment for everyone’s aspirations.
That would be as impossible as coming up with a single ranking function for meaning for all of society.
It's that people should align on their own aspirations.
Helping them become the kind of person they’d be proud to become.
This is something that is a) plausible, and b) would be obviously and significantly better than today’s world.
The modern world’s problem is partially a lack of villages.
There’s no meso scale computing or products or experiences or communities.
Reclaiming hollowed out words is hard.
Trust, safety, symbiosis–all hollowed out, MBA-ified to death.
"Queer" was used as a hateful word, so the LGBTQ community reclaiming it was an act of defiance.
But reclaiming a hollow word is harder, because it just looks like not an act of defiance but of just being cringe.
Doing things you could do already, but better: immanence.
Doing new things you couldn't do before: transcendence.
The former is obvious; the latter is novel.
Managing a product that already exists is a very different skill than creating a new product.
I’m finding David Chapman’s exploration of “nobility” fascinating.
Most recently, Seeing like a Good King.
We need philosopher-builders more than ever before.
Tech has inherent leverage, and so it makes the tech-wielding builder a king.
But we’ve been missing the “philosopher” part–the part that thinks about more than one ply of consequences.
AI gives more leverage to tech than ever before.
Which means we need philosopher-builders more than ever before.
Anthea’s newest piece compares LLMs as freeing a caged tiger.
The thinker is a caged tiger set free with intellectual collaborators willing to go wherever you want to go.
A cage is also a form of shelter, preventing you from having to challenge yourself.
Meta thinkers who are curious could get stuck in the sycosocial hall of mirrors more easily than others.
I randomly came across a Claude artifact produced by someone about multiple levels of emergence in complex systems.
At first it sounded fascinating, until I realized that it was mostly superficial gobbledygook.
I fear that I sound like this person, or will fall in the same gravity well of metaness that they do.
LLMs are amazing thinking partners for “caged tigers”, but their sycosocial approach makes them inherently dangerous, allowing hyper curious people to follow a trail away from reality.
James Cham tweeted a gem: "Buried at the end of George Dyson’s discursive and fascinating book on analog computing … is this terrific cybernetics-flavored proposal for three laws of artificial intelligence."
"There are three laws of artificial intelligence. The first, known as Ashby's law of requisite variety after the cybernetician W. Ross Ashby, author of Design for a Brain, states that any effective control system must be as complex as the system it controls.
The second law, articulated by John von Neumann, states that the defining characteristic of a complex system is that it constitutes its own simplest behavioral description. The simplest complete model of an organism is the organism itself. Trying to reduce the system's behavior to a formal description, such as an algorithm, makes things more complicated, not less.
The third law states that any system simple enough to be understandable will not be complicated enough to behave intelligently, while any system complicated enough to behave intelligently will be too complicated to understand."
Swarming sort powered by real people can discover novel things.
That’s because it's millions of real people making real decisions that align with their authentic needs and context.
LLMs are a fossilized version of real people's decisions; it can't pick something novel.
It looks superficially the same, but it’s fundamentally different.
If all traffic goes through the same mid thing, it compresses ranking so the already popular get more popular.
That’s why we see this phenomena where LLMs recommend the same popular things, entrenching their popularity even more.
This would not necessarily happen if it were a swarm of humans.
In a new ecosystem the swarm is hunting for the new moats.
Huge amounts of capital flowing in that will be "misspent" but also it's only possible to know which was ‘correctly spent’ in retrospect.
Moats are relative.
Even if what is a moat is reshuffled, some still exist.
Over time, even "weaker" moats are asymmetries as long as they’re stronger than what competitors have.
If there's a concave bias, then no matter how noisy, more data gives better results.
If the bias points in the same direction for nearly everyone then it’s naturally concave.
Emergent phenomena are structurally impossible to explain in a clear, obvious way.
Understanding emergence requires at least two ply reasoning.
“The markets can stay irrational longer than you can stay solvent.”
Bill Janeway added: “...and therefore you’re compelled to act irrationally.”
If you don’t act irrationally in an irrational market you’ll be knocked out of the game.
Similar to kayfabe in an organization.
I wonder: does the inexorable pull to short-termedness in the modern world come mostly from no long-term commitments from investors?
If shareholders had long term commitments they couldn't sell, would we see less short-termism focus?
Presumably there would be all kinds of other negatives, too.
But maybe not?
Fred Wilson’s advice: “be generous”.
The more stingy you are the more you make people not want to contribute to it.
Are you betting on growing the pie or extracting the margin?
The former is abstract / long term. The latter is concrete.
Two ply arguments typically lose to one ply arguments in practice.
But it’s the emergent logic of the two-ply argument that creates value.
This week someone framed alignment to me as "minimizing agency cost, as defined in the principal agent problem."
When you interact with a superior intelligence, the principal agent problem becomes more important to deal with, because the power differential gets larger.
Swarms, like organizations, are a form of emergent collective intelligence, and can be “smarter” than any individual member.
Stable business models require that where the cost and the value accrues are aligned.
If they're disjoint then the user feels like they're getting cheated, or the business model is nonviable.
For example vibecoding tools that charge a high margin for hosting of the final app (despite zero marginal cost), but not for the creation of the app (despite significant marginal cost).
Your "why now" argument has to rely on something new that has changed.
Not "no one tried this yet," because in a hot enough space with a big enough swarm, everyone tried it.
Being "in the loop" with a thing--trying to modify your own actions in response to it--is how you develop the intuition of how the thing works.
It shifts you from a passive to active stance.
It’s the active stance that builds your intuition at an order of magnitude better rate.
Every organization that doesn’t have to fight to survive succumbs to kayfabe.
The larger and more ground truthed the organization, the longer this process takes.
A way that large organizations get stuck: thrashing between bold ideas.
This kind of culture does a lot of first steps on bold ideas that never get to the later steps.
The person with the boldest idea that stands out from the noise gets funding.
But then the swarm moves on after the first step doesn’t show immediate huge results.
To have bold results requires a consistent vision with compounding value.
An AI native organization will probably optimize more for talent density.
That is, a small number of exceptional people over a larger number of merely competent people.
Before LLMs, many execution tasks required a human, so you needed bodies.
Now exceptional employees can have more leverage.
A measure of ecosystem health is multiple options for each service provider.
Each layer has competition to keep it healthy and strong.
For competition to work, the entities have to be independent.
Otherwise you get the appearance of competition without actual competition.
That’s worse than no competition, because the lack of competition is hidden, and thus easy to miss.
You get a hollowed out ecosystem.
For innovation, which is more important, creation or curation?
Intuitively the answer is creation.
Without the creator, the thing wouldn't exist.
But even a noisy process can create variance to select over.
Curation is at least as important; because the spark will likely die out unless it's boosted.
Someone looks at the spark and says "this one has what it takes", to give it life beyond its initial gasp of energy.
So you clearly need both, but if you could only have one, the curator is the more important.
With network effects, even if you're 3% better than alternatives, over time those compound into aggregation effects.
The compounding effect is more important than the rate.
It doesn’t matter what the rate of improvement is, compounding rates pull away from all alternatives.
This is true as long as it's a low switch cost environment.
To coordinate you need some static touch points between entities
Some frameworks and scaffolding.
Those have to be pinned down as nucleation sites for collaboration.
A financial lens I learned this week: wholesale transfer pricing.
If the wholesaler has a rare input they can extract most of the net value of the innovator on top, since they’re the only supplier.
A related observation: the old school restaurants that are left are the ones who own their own buildings.
Otherwise, the landlord would have priced them out long ago.
“I’m bored” in kids is often “I want to do a thing I'm not supposed to do and can’t think about anything else.”
In jazz there are no wrong notes, you just have to own them.
If you make a mistake... do it three more times so it looks intentional!
The rule of good design, in all contexts, is “make it look like it was intentional”.
Writing forces you to be more of an organized, multi-ply thinker.
Single-ply thinking is much easier to get away with if you're having short conversations with many different people.
It's much harder if you're having much longer conversations with a smaller number of people... and especially if the information is written, not just ephemeral.
Why does Hofstadter's law happen?
Hofstadter’s law: Even when you take Hofstadter’s law into account, it will always take you longer than you think to complete the task.
It shows up because timelines are emergent, based on beliefs, expectations, etc.
There's a recursive loop.
People update their estimates based on what they believe.
Also, because the curve of effort to fidelity is fundamentally logarithmic, but we naively assume it’s linear.
A new study implies that the reason animals all sleep has to do with mitochondria.
The mitochondria need a break from oxygen build up.
Given that animals need to rest, it also makes sense for other maintenance tasks (like memory compression) to happen in that time, too.
When you think about it, it’s wild that nearly every animal does something like sleep, since it’s inherently dangerous.
But clearly the reason for it is something that all animals share.
Markov Chains presume a stable / stationary distribution.
The recent Veritasium video on Markov Chains was great.
It made me realize that Markov Chains work great for stationary distributions.
That is, systems without an internal feedback loop.
Many systems have a compounding component.
Every system has some kind of feedback loop with its surroundings.
How well the “isolated system” assumption applies is how slow and weak the feedback loop with the environment is.
I found this new Kurtzgesagt on Quantum Immortality mindbending.
Spoilers for The Prestige!
The many worlds interpretation is what happens in The Prestige.
He is always the one that survives, because of course he is.
To tell if the experiment succeeded you need to be the one perceiving it.
The one perceiving it is the one with the continuity.
That doesn’t mean the others didn’t die!
The Stockdale paradox: Those without short-term hope are the ones who survive.
Hope mislaid sets you up for crushing failure.
If you’re too optimistic, you will be disappointed and that could crush you.
Superficial “I know it’s going to disappoint me” combined with deep “I believe in the end it will work out.”
An Indian saying: “It all works out in the end. If it hasn’t worked out it’s not the end.”
Cromwell: "I beseech you, in the bowels of Christ, think it possible you may be mistaken."
The Saruman’s power comes from a complete and total absence of self doubt.