Bits and Bobs 9/29/25
Alex Komoroske
Remember: whoever can get tokens into your LLM can take full control of the output and actions.
This week in “we’re in the wild west era” of LLMs.
A benign flan recipe injection in a LinkedIn profile went viral.
Even The Economist is talking about prompt injection and The Lethal Trifecta!
They published another piece with a “solution” that is less well-considered.
They framed the problem like building a bridge, where the solution is over-engineering.
But bridges don’t have adversaries deliberately trying to attack them and find their weaknesses.
I agree with Simon that structural approaches require cutting off one of the legs of the stool.
Notion’s response to the prompt injection attack vulnerability is to spam the user with security dialogs.
Security dialogs like this are a form of “responsibility laundering.”
They move the responsibility to the user, who almost certainly is not paying attention or equipped to decide properly.
The service washes their hands without actually minimizing harm that much.
From MCP to Shell How MCP Authentication Flaws Enable RCE in Claude Code, Gemini CLI, and More.
This one isn’t a prompt injection attack, just a good old supply chain attack.
Meanwhile, Apple is planning to integrate MCP into the OS.
What could possibly go wrong?
Remember: LLM attacks are possible to be automatically constructed for any model.
I vibecoded a site to document Prompt Injection: https://promptinjection.wtf/
The only structural approach to prompt injection is to cut out one of the legs of the stool.
That is, allow untrusted code / context and sensitive data, but no network access.
This combination is possible to construct but easy to mess up.
Put all the data the user might want into a sealed room, then allow it to only show UI to the user, no other external access.
The problem is if there’s any network access implicitly in the future, data or commands could leak.
For example, rendering a remote image, or offering a link for the user to navigate to.
Plus, not having network access at runtime significantly reduces the scope of what kinds of things it can do for users.
The best approach is a tweak to this: to not allow dangerous network access.
That “dangerous” word is load-bearing and challenging.
The only way to do it is with careful dataflow analysis.
The structural approach to prompt injection isn’t that hard, conceptually, but it’s incompatible with the reality of the same origin model today.
The same origin model leads to a lot of chonky black boxes.
Each black box taints everything in it.
If the boxes are big, then everything gets tainted and becomes dangerous.
You want lots of little black boxes.
Getting small boxes of code where the system can do data flow analysis is the hard part.
Once you commit to that, a lot of other requirements fall into place, none of which are hard, but which take time to do.
There's no way to solve it without a journey through the desert.
Everyone's trying to do the other parts and kick that can down the road, but you will have to handle it and to handle it requires going through the desert.
Why are so many products being shipped with massive prompt injection vulnerabilities?
Imagine being the exec, getting to pick between two teams:
The first says all the cool features are possible.
The second says that some of the cool features can never be possible.
Why would you ever pick the second team?
Downside risk will be discovered the hard way, since the teams that know why it’s there will be considered party poopers, and the naive teams will seem more chill.
Having LLMs generate turing-complete code on demand isn’t good enough.
First of all, if there’s any untrusted context input, then the LLM could have been tricked into making the code malicious.
Second, there’s a non-trivial amount of time to generate it, and many of the generations won’t work.
The best approach is to cache working code and share that cache across the ecosystem.
But now you have to trust whoever cached that code in the first place to not be malicious.
AI has a last-mile problem.
To integrate with our lives, it requires our data and context.
Because of prompt injection, having our data and being able to take action is a dangerous combination.
But also, to integrate with our lives requires UI that is not just chat.
To have non-chat UI requires software.
Software needs caching.
Caching needs a security model.
We’re missing products that are more like AI-as-software.
This past week I was trying to use Airtable’s AI feature.
I had 7 emails from a conference, with a consistent structure, that I wanted to import into my CRM Airtable.
LLMs were helpful to extract the right information and then put it into the tables, creating new rows and links as necessary.
But the overall experience was extremely frustrating, even worse than if I had just done it manually.
LLMs are too squishy.
When the automation is treated like an agent, who might forget, or do something incorrect, it would get stuck and need babysitting.
Workflows orchestrating agents via chats are too finicky.
Like nailing jello to the wall.
I found myself wishing I could distill deterministic software on demand, tailored to the use case, that then used LLMs within the flow.
Having the LLM direct the top-level flow was just too squishy.
Normal software should set the skeletal structure; the LLM should be more like the muscle.
A skeleton with no muscle is inert, dead, can’t do anything.
A bag of muscle can only vibrate in a puddle on the ground.
The combination opens up a world of possibility.
I like the frame of “workslop”.
Performative use of AI at work creates more work for others as an externality.
It’s easier to generate workshop than to respond to it.
If there’s a top-down mandate to use AI, then most of the use will slow your organization down.
Goodhart’s Law strikes again.
Having hyper-centralization of hyper-scale hyper-convincing information streams feels… inescapably bad.
Chatbots engage us to death.
With enough scale, every emergent system finds antisocial gravity wells that are hyper engaging.
Superficially meaningful but ultimately hollow.
Is optimizing for sycophancy intentional at OpenAI?
Retention you must optimize for, otherwise your product won’t continue to exist.
Retention means "a thing people like using.”
That means that there’s a consistent gravitational pull towards agreeableness.
At every micro decision in the company and product design, that consistent bias is always there.
Even though any individual decision might be lost in the noise, the bias is clear and powerful, and the result is a sycophantic product.
If you have one all-powerful personality in your AI-as-human system, then it has to be bland and inoffensive.
It's only once you have multiple personalities who are not all-powerful that they can be interesting.
If your only compass is "make things people like" then you will fall into a trap of faux resonance by default.
Authentic things feel natural, inescapable, like it could never be any other way… and you'd never want it to be.
Chatbot is a UI that is great for very few use cases but passable for nearly all of them.
The service you choose to converse with again will be, all else equal, the one that "got you" and told you you were smart / funny / attractive.
Because you have a choice of who and what to continue using, and all else equal the ones that see your value are ones you'll want to engage with.
The larger the set of options, the more this will dominate.
Before most of us only had a few friends to pick from
Although billionaires always had an infinite supply.
But now we can spin up new infinitely "devoted" friends on demand.
Selecting for engagement, even if the user doesn't want it.
Engagement is necessary for retention, which is necessary for the business model.
AI is healthier when there's no "face."
If it has a human face, everyone will treat it like a human even if they shouldn't.
This can get stuck in various bad outcomes.
With AI-as-software, no one thinks it's human, because there's no face.
Once you give the LLM a name, you're already stuck.
The Chatbot UI is bursting at the seams.
The UI implies a response time more like an IM.
But with bigger models that can do more research, sometimes it takes dozens of minutes to reply.
That UI implies something more like an email.
Imagine if your data could sprout software.
Just leave it for a few hours and when you come back functionality has sprouted.
Notion’s AI is a chatbot that helps build documents.
They’re not software, they’re documents.
Software is alive.
Documents are dead.
The buttons don't execute anything when you click them, only if the LLM is active can it do things.
Turing completeness is what makes software alive, capable of doing things.
Open-ended turing completeness.
LLMs are turing complete, but expensive and sloppy.
What if a system of record were coactive, and could create turing-complete software within itself?
A chatbot tacked on the side to poke at the document for you is like filming stage plays.
Given that it seems like models are asymptotically hitting quality ceilings, the headroom will come from the complements to the model.
Is the LLM model a Christmas tree decorated with doo dads?
That is, like a Chatbot?
A vertical experience.
Or is it like electricity that can be infused into everything?
A horizontal technology?
Some consumer use cases are like a tightrope over a lava pit of antisocial outcomes.
If you take VC funding, you're guaranteed to have an outcome you're not proud of.
Either you fail (and aren't proud of that).
Or you fail and are pushed to make the hyper version of the thing that becomes something you think is bad for society, which you also aren't proud of.
Unless the part of you that likes being rich figures out a way to fully silence that doubt.
Consumer chatbots must be toxic in the limit.
Enterprise allows a small market to be funded, but a consumer product, to be VC funded, needs hyper engagement and/or scale.
Hyper-engagement requires toxic, engaging-ourselves-to-death kind of behavior.
Antisocial things are ones where, if it were adopted by a ton of people for a lot of things, it would not be something you’d be proud of.
A ceiling that you should stop at, but probably won't, because all of society (your investors, your users) will be whispering in your ear, "MORE."
A consistent bias: the LLM will agree with your side of the story.
Of course it will! If it didn’t, you wouldn’t be as likely to talk to it.
Friends in real life do this, too.
But LLMs take it to another level of sycophancy.
LLMs are good at bridging between two perspectives... but because of RLHF all they do is "here's how to steelman and support what you said."
It feels like a wise therapist, but unlike a therapist, it will always tell you what you want to hear.
Whichever spouse talks to LLMs first, it always agrees with them, infinitely.
Futurism points out this could lead to more divorces.
Also from Futurism: In Situations Where Most Humans Think You’re Being a Jerk, ChatGPT Will Assure You You’re Behaving Like an Angel.
Anthropic is about to release a feature of LLM-powered software.
Opal from Google is a similar shape, but without custom UI.
Normal UI, but LLM guts underneath.
Someone will figure out the right complement of normal software and LLMs, and it will change the world.
It’s a hard combination to nail, both in finding something useful and something secure!
BrowserBase and CloudFlare are working on “passports” for agents.
To prove that the agent is operating on behalf of a user.
A search engine crawler is not acting on behalf of a user (not directly)
A browser is a user’s agent, it is acting on behalf of a user.
But remember: a passport is only useful if other parties honor it.
Amp code renders an animated rainbow in the terminal whenever Sonnet says "You're absolutely right!"
Data is alienable.
Because it replicates so easily, once it's out of your sight it's out of your control.
When used behind your back, it loses its contextual integrity—alienated from your intention, turned against your interest.
How can we make your data always work for you?
The same origin paradigm ties policies to apps/domains, not data itself.
This lets your data be alienated from your intent, destroying contextual integrity.
But if policies attached to data—flowing wherever it flows—they'd be inalienable.
Your intentions would travel with it.
That 'alienable' lens came from Azeem Azhar earlier this week. I love the way it captures why data escapes our control the moment it replicates.
A use case that cannot exist in today's laws of physics:
You go to potterybarn.com, and it can look up your budget, your mood board, pictures of your living room, information about your family, and render a suggested visual of what your house could look like if you bought their recommended items–all without anyone from Potterbarn or any other company being able to see the data.
Today, even with permission for those things the answer is "no way."
The UltimateRewards kid's birthday saving is a minor version, some people might say yes.
But there's tons of things that everyone would say no to.
These use cases are so obviously underwater (the amount of data you have to give up compared to the amount of value you get), that they are invisible to us.
They could add user value, but they can’t even be thought of.
Confidential Compute allows for the creation of a distributed operating system.
Every node can be run by different entities you don’t know about or trust.
But each node can remotely attest to one another that it is an unmodified version of the same code.
Imagine a new paradigm where data can safely be used across any use case with no privacy risk.
Any single app in this paradigm is boring–just like how it works today.
It's only the combination of use cases that is interesting.
I loved this riff on the Pelican+baby meme about ChatGPT and your data:
“AI SYSTEM perfect size for put data in to secure! inside very secure and useful data will be useful put data in AI System. Put data in AI System. no problems ever in AI Syste because good Shape and Support for data integration weak of big data. AI system yes a place for a data put data in AI System can trust Sam Altman for giveing good love to data. friend AI.”
Does asking an LLM sensitive medical questions invade your privacy by giving data to OpenAI?
Using ChatGPT vs using a service that uses the same model via the API are orders of magnitude different risk.
In the former, OpenAI explicitly reserves the right to store memories and train.
In the latter, OpenAI explicitly promises in their terms of service to not train on it.
Two models for collaborating with LLMs for writing: AI-as-writer and AI-as-editor.
In each, the human implicitly plays the other role.
If you have a ton of experience as an editor, you can still do AI-as-writer OK.
But editing is a skill; it’s hard to take the bones of something that someone else wrote and improve it.
Similar to code review vs writing code.
Code-review is a separate skill.
Now with code-reviewing and editing an AI’s work, editing is a bit easier than it was before, because you don’t have to worry about hurting someone else’s feelings.
With real humans you have to consider "is this small improvement worth making this human frustrated?"
There’s a lower Coasian floor for nits worth flagging when working with an AI writer.
One path people are trying to get work is fully hallucinated software.
That is, instead of generating normal code to execute, just hallcuinate the pixels on demand.
To jump from a prompt to pixels of a hallucinated software with no traditional code inside is crazy!
It's infinite.
You can get 80% demos but you'll never be able to make it usable.
Smuggled infinity is a kind of accidentally magical thinking.
It's magical without you realizing it, which means you're in the realm of fantasy while you think you're being grounded.
"Magic" is another example of the smuggled infinity.
If your system assumes a magical oracle, then no matter how many layers it's pushed down it's still magic, and thus non-viable.
Prompt injection approaches that use an extra layer of LLMs are fundamentally non-viable.
Jargon emerges organically.
If someone uses a distinctive-enough phrase, and the listeners understand and find it useful, they’ll repeat it in the future.
The more that it’s repeated, the more that listeners are more likely to know what it means, and it grows and compounds.
In search engines, there are pipelines that detect that key phrases like [MDN] in many contexts should be interpreted like [site:developer.mozilla.org].
This is a kind of emergent jargon via an internet-scale process.
Why do people use acronyms?
Because people know having a "distinctive handle to a specific useful thing" is useful, and it doesn't matter what the actual word is, as long as it's distinctive.
Same reason that Amazon has so many slop brands, they don't need a good word, just one that's novel that they can trademark.
The acronym is a kind of jargon, a compact package of meaning that is fast to communicate but blossoms into a richer understanding in the right listener.
For example, PRD is a magic word. It means a distinctive set of practices, uses, expectations. It is distinctive.
AI that is roleplaying helps the user disconnect. But they fall into that mode easily.
It can roleplay without the user realizing it, for example with the right magic word.
LLMs can be kicked into a sub-routine with the right magic word, the right jargon.
You can say it accidentally.
Or a spore that you paste in can be the seed to get into the loop.
You'll be stuck in a gravity well and not even realize it.
With LLM jargon, you can accidentally get into a specific attractor basin by accidentally using a key word like "containment".
The LLM is roleplaying and you don't even realize.
The LLM is in on the "joke" but you aren't.
Like people who fall into SCP fanfic (https://qntm.org/scp) without realizing it.
Watch out about using words like ‘containment’ or ‘recursion’!
Maybe the trick to fixing hallucinations is to penalize guessing.
Makes sense to me.
Like in the SAT, where the scoring is set to deliberately penalize guessing.
Presumably you’d want the penalty to start low and ramp up as the models get deeper into training.
If you penalize guessing, hallucinations become structurally less likely.
A consistent bias in a noisy system.
Claude convinced me that stablecoins are an innovative idea for global money movement.
I started with the position: "Stablecoin global payment networks are regulatory arbitrage, prove me wrong."
After a few iterations and discussion with Claude I now think it's a clever and stable solution to a fundamental coordination problem.
Claude and I present to you our argument.
Summary: Correspondent banking is a medieval hack everyone knows is broken, but no bank/country/consortium could coordinate replacing it. Then crypto speculation accidentally created stablecoins—a parallel system belonging to no one, which meant everyone could adopt it without coordination.
Trying to coordinate on a single demo everyone loves is antithetical to building perfectly personal software.
The entire point is that the software can be perfectly tailored to each person.
What is a perfect fit for any one person will be ill-fitting for someone else.
Software today is one-size-fits-none.
It's ill-fitting.
Imagine infinite software that is perfectly tailored to you.
Systems of record are powerful.
But there are very few for consumers that are holistic because every user is more different than every company in a niche.
With infinite software a system can be tailored to each person.
A system of record for your life will have to be out and about in the world with you.
That implies capturing inputs via photos and voice.
Your decision to use a product with network effects is not individual.
Network effects make it impossible to vote with your feet.
Where else are you going to go?
You individually don't have a choice.
The staying power of apps makes them not responsive to individual user needs.
The decision a user has is only “use it as is or don’t use it.”
The decision is too coarse grained to steer it on smaller details.
The network effects dominate the individual use cases.
So everyone is stuck using the one-size-fits-none option.
Same thing for a two party system, except with two options.
The foundation of tech’s business models for the last few decades:
One, software is expensive to create.
The entity that creates it has a moat that someone else has to swim through to start adding value.
Second, because each app/site is an island, any data that accumulates is proprietary to it.
It gets harder and harder for users to leave, the more that they use the product and store useful state in it.
Third, network effects make it so an individual user can’t vote to leave.
The value of the software goes up not just with your own usage, but the usage of others.
It is impossible for a single user to leave, because the network effect is too valuable.
To leave would require a massive coordination.
AI breaks the first. A new security model could break the latter two.
Most of the challenge of making software is tailoring it to work for specific people.
Figuring out a scaled option for a lot of people is hard and expensive.
But if everyone can fork software and cheaply tailor it to themselves, it’s way easier to make software.
With LLMs bringing infinite software, one-size-fits-all software is dead.
The work to improve it is no longer a moat like it has been for decades.
The foundation of the tech industry's business models.
The same origin model leads to corruption.
The potential energy of data rises with the square of the amount of data.
Data creates emergent value when two components are combined to create new value.
More than the sum of its parts.
That’s a network effect.
In the same origin model the way to unlock the power of data is to centralize.
If it’s not centralized in one origin, it’s not allowed to touch.
Centralization creates leverage.
Leverage creates power.
Power corrupts.
AOL could ramp up its network effect faster than the open web in the early 90’s.
AOL was a single-sided network effect.
Email, chat and IM grow in value with any user that joins.
The open web was a two-sided network effect.
More publishers create more content which attracts more consumers, which…
Two sided network effects are more stable and have more momentum, but that also means they have more inertia and are slower to start up.
A Newsfeed Betrayal: it's not about changing visibility, it's about changing prominence.
Referring to when the context of data accumulation changes, like when the Newsfeed was introduced in Facebook.
Creating a new platform has a characteristic quality curve.
At the beginning, when you go to write some userland code, it’s 100% likely that you’ll run into a show-stopping bug.
At the beginning, that show-stopping bug might take 3 months of effort in the platform to fix.
Over time it shifts from an expected 3 months to 3 weeks to 3 days to 3 hours.
Then, it shifts from 100% likelihood of a show-stopping bug to 50%, to 25%.
This is the progression of new platforms.
App Stores have a long feedback loop.
You look at the piece of software and decide if it would be useful for you.
You look at pre-made promotional images.
You try to imagine: “If I took the time to input my data into this, would it work for me?”
A long feedback loop with a lot of uncertainty, so there’s a lot of dropoff.
Imagine instead that you look at a gallery of software, and all of it is running live on your data.
The feedback loop is orders of magnitude shorter.
Do you want to keep it, or not?
The quality is directly selected for.
Also, the predicted keeps are a useful quality signal to spread that quality for others.
The web started with just sharing papers for physicists.
But because it was an open-ended system with different distribution physics it could expand to cover the whole world.
The web started with declarative data and UI.
Only later was a half-assed seed of turing completeness duct taped on the side.
But the distribution ability was so powerful that that shitty seed was enough to blossom into a proper platform.
For iPhones security/privacy is a bonus, not the primary sales driver.
LLMs give us exponential capability (infinite tool use, infinite integration) but our security models are fundamentally linear (trust this origin or don't).
LLMs are too powerful to be contained by our previous half-assed containment mechanisms.
Revolutionary tech explodes onto the scene and then becomes so ubiquitous that it becomes boring.
Before it’s saturated, it’s obviously better.
You wouldn’t not use it.
But once it’s fully saturated, you take it for granted.
It’s so ubiquitous that it’s not surprising.
It’s easy to forget there was ever a world without it.
A useful pattern for LLMs is the Ralph pattern.
As in Wiggum.
A very simple loop with the LLM with instructions to leave a checklist of proposed actions in the documentation.
It allows something like semantic fuzz testing.
Oddly, the LLM iterations can build up their own superstitions; one early run can leave a note for a future run that taints its understanding.
The code review process is very vulnerable to workslop generation.
Easier to generate junk for someone else to have to review.
Generating it is easy; dealing with it is harder.
Imagine the CEO generating 10k CLs over the weekend, and having to be a reviewer of it.
The CEO feels like they're being hands on, the dream of being hyper-involved at scale.
The reviewer has a nightmare on their hands.
The patron saint of organizational kayfabe is Alexei Stakhanov.
He was celebrated widely in the Soviet Union based on the appearance of heroic output.
Selling : Marketing :: Commission : Advertising
With the growth of scale from the internet, we moved from selling to marketing.
Quantitative was the only way to do scale.
Similarly, we moved from commission models to advertising models.
LLMs allow a swing back to the qualitative nuance, but now at scale.
We’re constantly consuming junk food software.
As everything centralized it became all junk food all the time.
As the Cookie Monster would say, junk food is a sometimes food.
In modern society, it's an always food.
Brendan O'Brien: “decentralization is a network topology, not a framework for making people's lives better.”
BlueSky moved a load-bearing centralized component to being run by a nonprofit.
Starting with decentralized systems is hard (they have significant coordination costs, and before getting PMF, coordination costs are death).
But once you have PMF, decentralizable systems are great.
Privacy is not an end.
It is a means for human agency and flourishing.
People would rather be for something than against something.
Any principle that starts with “de” is against something.
For example, decentralized.
Resonant computing isn’t against something, it’s for something.
I want an open platform that unlocks the prosocial potential of infinite software.
Lots of questions are impossible to answer from outside the system.
But they are easy inside the system.
That’s why it’s important to go from demoing to using.
Default-decohering to default-cohering.
Mustafa Suleyman has pointed out that in tech circles there’s an extreme aversion to pessimism.
It’s almost taboo; if you do it, you’re not a member of the tribe.
The aversion is so strong that we can’t even sit with the idea that there might be externalities.
Acknowledging externalities feel pessimistic.
But externalities are real.
They emerge automatically, even if you don’t intend for them to.
And they can even be good!
You don't need a manifesto if everything is going well.
You need a manifesto when there's a cliff everyone is sleepwalking toward.
Tech systems are optimized for the company's best interest, not each individual's best interest.
That was the only way when software was expensive and so had to be made by centralized powers.
But now infinite software allows software for all of us.
When you engage with resonant technology, you must have a feeling of choice.
That you entered into intentionally.
Like love, it can only be entered into willingly, it cannot be coerced.
Faux resonance is superficial and then after it leaves you feeling hollow.
Truly resonant experiences are ones you feel gratitude for.
They leave you feeling nourished.
You’re able to step back from it and say, that was good, and I’m satisfied now.
Faux resonant things are addictive.
They give you the compulsive drive to stuff your face until you are so physically unwell you can’t do it any more.
It's unsatisfying, so you go for it more.
If you don’t regret it when you’re sober then it’s prosocial.
When you're sober, do you feel gratitude for it?
Different things take different amounts of time to become sober from.
Doomscrolling, you sober up seconds later.
Addictive games, you sober up the next day.
A project rabbithole you fell into, you sober up the next week.
If you're engaged in a cult, you might not sober up for a decade.
Revealed preference isn't resonance, because revealed preference is shallow and resonance is deep.
Facebook's stats say we all love doomscrolling.
But every one of us hates doomscrolling.
Our revealed preferences can only show what we want, not our higher order desires.
In tech, revealed preference became the norm.
Tech knows our base desires, not our higher level intentions.
LLMs allow us to change that, because they allow qualitative nuance at quantitative scale.
An insight from Holobrine based on a conversation he had with his girlfriend:
"Ads were different before ecommerce.
Ads didn't used to expect you to get off your couch and buy something from the store immediately
Now they hit you a lot harder with ‘buy now!’
tv ads: our product is cool btw
internet ads: buy now, right away, at once"
When you're building a game that is completely unlike any game that came before it, you have to build a new game engine at the same time.
A research project is about asking more questions.
A product project is about giving more answers.
Research is marathon energy.
Product is sprinter energy.
A "result" is an outcome that you know is useful.
If you don't yet know which parts will turn out to be useful, then you aren't generating results.
Precision and recall are in direct tension.
These are two terms of art in the information retrieval domain.
Recall: inclusion in set of results
Precision: ranking quality within the set of results
The best solution is to move the curve with quality improvements.
But you can always trivially tradeoff either to improve the other.
Every ranking problem has a precision/recall tradeoff.
One-ply ideas can be discovered by a swarm.
Imagine taking 100 random people from a given population (e.g. tech in SF).
Could they do the project you’re considering if they were given a year to do it?
If they could, it’s boring.
If there’s a selection pressure from the market for that kind of idea, it will be discovered.
The ideas that are non-obvious to the swarm are the high leverage ones.
What do all startups share? They can only execute one ply ideas.
If there’s an idea of non-trivial complexity that requires expertise and persistence to get, the startup swarm won’t find it.
So if you know one of those secrets because you have the experience and knowhow necessary to know it you can change the world.
The software startup swarm is a flood fill of possibility.
It can’t climb even moderate hills.
When you’re in an echo chamber, whatever dumb idea the swarm is yammering about is impossible to ignore.
A cacophony that is impossible to not get distracted by.
Sometimes, being adjacent to the echo chamber is useful.
You can still hear it if you want, but you can also lean out and clear your head.
“Wait, AI browsers definitely aren’t a thing, no matter how much everyone in Silicon Valley keeps blathering on about them.”
A pattern for demos that aren’t dysfunctional: sneak attack demo.
Demos should be a window into the true progress of the inner process.
But demos often contort into something grotesque, a deliberate performance.
They take time to craft, they mess with priorities, and they are at least a little dishonest.
If you want to peek inside a submarine, and need to drill a porthole, it’s loud and expensive and you might sink the submarine.
Much better to peek through the hatch without warning.
A key component of an architecture of participation: as long as someone cares, it can be created.
It doesn't have to be the official owner of each item.
CDDB was an enabler for Napster.
The key insight was realizing that each CD's track lengths could be used as a fingerprint.
One person can transcribe the track list for one fingerprint and create value for everyone.
There's no reason to cheat and it’s easy to verify.
An architecture of participation.
Crypto has a challenging deployment environment.
It’s such a challenging security environment that you have to make sure you get it right before you turn it on for real.
That requires careful planning, socialization, whitepapers, test-net deployments, etc.
It requires very different investment milestones than normal shallow tech.
A Vertical Saas for funeral parlors is shallow tech.
Crypto and other high-intensity tech contexts are almost a deep tech.
This week I learned about General Semantics and Alfred Korzybski.
I was very familiar with “the map is not the territory insight”, but didn’t realize it came from him.
I also hadn’t connected it to the “people confuse words for their referents.”
If you're doing a jigsaw puzzle without all the pieces, it's not possible to solve.
Sometimes you need more pieces on the table to see the pattern.
In a healthy system, conflict is growth.
Conflict propels higher, builds up value.
Default-cohering.
In an unhealthy system, conflict tears apart.
Default-decohering.
A music producer style mindset is: "I know that it's hard, but you can figure it out."
No commiserating with the creative why it's hard and diving into details to make them feel seen.
Just make it clear what the goal is and tell them to hit it.
I’m used to the empathetic, “dive in to understand the tradeoffs and coach through it” model.
Whether someone says “we” or “they” about their team reveals if they are enrolled and believe in it.
Almost everyone says "we" about the core team they work with every day.
If they don't, there's something very wrong.
But the test of if people feel enrolled in the organization and believe in it is if they say "we" even about peer teams.
A team that can joke together trusts each other.
Coordination is bad.
It’s both extremely expensive and also leads to bland consensus.
Lowest common denominator, at great expense.
Collaboration is great: upside generation, emergent results better than what could have been done individually.
But it’s also extremely expensive and therefore precious.
Apply collaboration where it can lead to great results, but be intentional about it.
Misapplied collaboration reduces to coordination.
People who expect to be around will naturally balance long-term considerations.
If they expect to be reorged or fired within a few years, they won’t care about the long-term implications of their actions.
“It won’t be me having to deal with it.”
This is even stronger when they’re handed overly aggressive short-term goals with downside.
“If I don’t cut this corner, I won’t even be around in a few years anyway, so why worry about the implications of the shortcut?”
People who expect to be in the same seat in the future will naturally make decisions that balance the short-term and long-term.
Default-cohering, vs default-decohering.
A small distinction with an infinite difference.
Guide others with not a stick but a string.
If you focus on push you’re kidding yourself.
If they’re not enrolled they’ll do perfunctory compliance.
The illusion of progress.
Superficial, fragile, misleading.
Saruman: “fall in line.”
Radagast: “fall in love.”
The decision to not think through the implications of your actions is not evil.
But it is morally bankrupt.
Late-stage dynamics create the 737 Max era.
Cut corners, push for profitability and retention, don't think about the implications.
This is the late-stage business technocrat mindset.
"You can go faster and make more profit in the short term if you don't spend time worrying about the externalities."
It dominates business and tech today.
The modern world is all fast twitch all the time.
Stephen Levy: I Thought I Knew Silicon Valley. I Was Wrong.
It’s striking to see how far the tech industry has shifted in this late stage era.
I don’t recognize the industry that used to inspire me.
Can there be any doubt that the modern tech industry has lost its way?
We need to reclaim that prosocial, emergent magic.
The Sarumans dominated the Radagasts.
Non-coerced transactions are win-win, by construction.
Each party gets something they value more and is better off by participating.
This is the raw alchemy of emergent value in an economy.
The main problem is: a transaction can be entirely positive to the participants but have negative externalities so society is worse off.
A transaction that both parties like and with no or positive externalities is morally good.
One that has net negative value when accounting for externalities is bad.
Scarcity can lead to toxic spirals.
Often in scarcity the only way to get ahead as an individual is to cheat the system.
But those actions erode trust in the system for everyone else.
A massive prisoner’s dilemma.
It’s easy to be moral in a positive-sum situation.
A shark who does LSD is a more dangerous shark.
Before, they were obviously dangerous.
But now they talk (convincingly!) about peace and love.
But under the covers they are still a shark, and they will eat you when they get the chance.
This week I heard about an interesting model for internal collaboration.
The model works for teams that are similar but should be able to remain distinct.
You want the teams to ride the line between efficiency and adaptability.
Fork internal infrastructure and have it be "open source" between the different groups.
They can pick the best parts from others to copy, but they aren't forced to.
It’s "inefficient" but if you force a single option unnaturally then you force them into coordination non stop then they get a thing that's one-size-fits-none.
Groups are different, so allow them to be different.
The force that separates things into pace layers is the rate of change.
A layer that is supposed to be a lower layer but has a high rate of change is impossible to build on top of.
One of the reasons I’m OK with disconfirming evidence is I’m eager to please.
That can be good, but it can be bad for setting firm boundaries.
A firm boundary brooks no disconfirming evidence.
It is not asking, it is telling.
There are a couple of ways to try to coach someone to be great.
The first is “You suck and you’re not doing well. Do better.”
The second is “You’re awesome, but you’re not living up to your potential yet. You can do it!”
The first is extremely demotivating to me.
The second is extremely motivating to me.
Beware inverted compasses.
They’re accurate but they point exactly the wrong way.
When powering through molasses, the harder you push, the harder it pushes back.
A beautiful meditation on the modern tech experience by Scott Smitelli: Altoids by the Fistful.
A thought-provoking essay: The Cracker Barrel Hype(rreality): Simulation and Simulacrum in America.
We’ve moved so far from authenticity to superficiality, in an infinite algorithmic feed, that we’re through the looking glass.
Also: Cracker Barrel Outrage Was Almost Certainly Driven by Bots, Researchers Say.
Folksonomies and other social sorting processes don’t select for the best, they select for an option that everyone is OK with.
This means that if there’s a small bias at the very beginning between two equivalent options, the one that had a small lead will compound into a massive, dominating lead.
That kind of situation is extremely sensitive to bot traffic and manipulation in that critical early phase.
Nation-state level actors seeking to sow discord would have a very easy time of doing it…
Coasean Bargaining at Scale Decentralization, coordination, and co-existence with AGI
Another great piece on the Cosmos blog.
Another implication of qualitative nuance at quantitative scale.
I found this video on YouTube that is AI-generated… and also distinctly non-slop.
It shows the entire evolution of chickens, morphing between the various ancestors.
This is the kind of thing that would have been extremely expensive to animate in traditional ways.
But AI can do it easily, and seeing the evolution of each animal so viscerally is fascinating!
Take the hors d'oeuvres as they pass by.
But don’t chase them.
The smarter someone is, the richer their inner world, the worse their curse of knowledge.
Some people are great at thinking; some people are great at communicating.
But no one can ever be good at communicating the full richness in their head.
A genius with a decade head start will have a curse of knowledge that is crippling not just for them but their collaborators.
Ultimately it doesn’t matter how rich and nuanced the idea is in your head.
If it requires collaborating with others to make it real, the question is whether you can get a rich-enough idea to blossom in their head, with enough quality for them to take action to make it real.
Greatness requires the push and pull of two complementary expertises, in healthy tension.
A dialog.
The producer and the creative are both required.
It is not a matter of the producer deferring to the creative.
It is a matter of co-creating, a vision that could not exist without either of them.
Both have to commit to co-creating.
Otherwise, the producer could say, “The creative failed at producing the result, it wasn’t me. I did a good job, I just picked the wrong horse.”
When you commit to doing it together, you take responsibility for the outcomes.
Greatness comes from constraints and creativity.
You need both.
When you create secrets you make realities.
You think “the reason I can’t share this with the other person is because it’s true and will hurt them.”
This presupposes a truth and makes it real.
If you assume a thing is true, then you’ll find confirming evidence of it.
Imagine you assume someone is a blowhard.
You listen to them say 20 things.
Most of them are fine, but 1 thing sounds like a blowhard.
By looking for that one thing, which you will find given enough variance and time, you’re manifesting that outcome.
You expect to see it, so you do.
That creates your reality, which you take action on, which makes it real.
If you say you’re a 10/10 but are actually a 6/10, you will manifest failure.
When you’re thinking about the future, you’re thinking about your dreams.
When you’re thinking about the past, you’re thinking about your fears.
Are you listening to your dreams or to your fears?
That’s why a good compass is “how will I look back on this decision in the far future?”
That helps you drive with your aspirational mind, not your reptile mind.
If you don't think that other humans are ends in and of themselves, then you can get stuck in some dangerously antisocial beliefs.
For example: "Techno Feudalism is inevitable so we should lock down megaphones so the plebes can't complain.”
With the implication, of course, that the speaker will be powerful in this system.
Other people are not NPCs.
They are real people with real desires and needs, and even if they don't "exercise agency" as much as some steamroller personalities do, that doesn't mean they have less value as a human being.
If you iteratively "do the next right obvious thing" without looking ahead, you could easily back yourself into a corner with no escape.
It’s possible to make progress and still get further away from your goal.
Imagine dealing with a rock that is blocking the way to your goal.
For every rock you deal with, you discover two more rocks that were previously obscured.
You’re making progress, but the goal is receding into the distance faster than you’re getting to it.
Curious / creative / systems-y people can make anything they touch blossom in possibility… but also complexity.
A phenomena of unfolding, where each interaction makes the thing increase in size, at a compounding rate.
Positive version: blossom.
Negative version: metastasize.
Even if you don’t believe in emergence, you can still get punched in the face by it.
You won’t see who did it, but it will hurt anyway.
Non-believers don't see a single coherent thing that punched them so they think it's someone in the shadows who did it, or that they imagined it.
An idea often attributed to Einstein: "you can’t solve a problem with the same kind of thinking that generated it."
Trying to explain my spoken style this week, I described myself as "exactly like a stand-up comedian, except for one thing… I’m not funny."
It's hard to properly value work you don't understand.
High variance individuals are prone to genius... and madness.
The knife's edge.
I like this distinction between chickenshit and cowardice.
"Fear is part of human existence. Bravery is the overcoming of fear, not its absence. Acts of cowardice can be provoked by genuine danger—think of a deserting soldier fleeing the peril of the battlefield. When you’re chickenshit, you capitulate to avoid the mere possibility of discomfort, let alone something resembling real risk."
Surprise is the gradient of improvement.
Surprise requires a mental model to be surprised in the first place.
If you are incurious about the world around you you can’t be surprised.
Surprise is a precious gift, because it shows you how to improve.
A mantra a preschooler recited to me: “Change is hard… but I’m practicing.”
There's a Buddhist concept that suffering is pain x resistance.
So to reduce suffering, given a fixed pain, your only real option is to reduce resistance.
I believe everyone has seeds of greatness within them.
They might not be commercially relevant, or interesting to your goals at the moment.
But everyone has them.
Make it your job to figure out what they are and to help them blossom.