Bits and Bobs 6/9/25
Alex Komoroske
OpenAI going for vertical integration is like cell phone carriers trying to control what software you can install on devices connected to their network.
I can see why the provider is going for it, but I can't see why it's good for users.
Your user experience, your memory, and your model will all be tied together.
The power of the memory will make it harder and harder for other experiences to compete.
It will accelerate the aggregation around ChatGPT.
Good for OpenAI, bad for everyone else.
ChatGPT’s goal is one app to rule them all.
What could possibly go wrong?
I think this old set of tweets from Luke Davis are onto something.
"Autonomic Systems will be key to the next phase of AI. This is not about parameter count and activation functions or transformers vs RNNs, it's the vast machinery of ordinary code wrapping the models, taking their output and bending it back as input for the next go round."
"Everyone was watching the magician's right hand, expecting recursive self-improvement to involve AI updating its own weights, but the real trick was hidden in the left hand—using the LLM to write code for the Autonomic System that envelops the model itself. "
"The Autonomic System surrounding an ML model is like the structure of culture and society around a human mind. Laws and markets and your mother's expectations are not themselves part of your neural net, but without them shaping your inputs, your outputs would be very different."
"Human history began when WE got Super-Autonomic Systems. After we evolved intelligence, the IQ of a normal human was about the same for many millennia. The big jumps in civilization were because of structure and peripherals: fire, writing, machines, the rule of law, code..."
Generic AI recommendations without your context will inherently be mid.
Only if they have your personal, unique background bringing novelty to the question will you get novel answers.
That’s one of the reasons for the Hayes Valley demo fatigue.
The “booking the flight” part of automation is the highest downside and lowest upside of the whole flow.
And yet it’s what everyone demos!
Infinite software won't counter the hyper-aggregation problem if the software is distributed within the same origin paradigm.
The same origin paradigm is the laws of physics of how data flows in the web and apps.
When data moves across an origin/app boundary it has significant friction. That friction requires the users to orchestrate the data flow themselves, e.g. copy/paste or keeping it in their head
The user has to do less orchestration if they stay within one origin, so they do. Over time the origins with more data tend to get bigger and sprout more use cases, a form of data gravity.
This leads to hyper aggregation. That tendency is intrinsic to the same origin paradigm.
LLMs make infinite software feasible.
But if each particle of software is distributed as an origin/app, then the friction of orchestration dominates the value of the software.
Infinite software in the same origin paradigm doesn't fix aggregation, and might even accelerate it.
The tweet thread version of this has visuals that make this more intuitive.
The raw text of your context / dossier is extremely rich and dangerous in the wrong hands.
Even if it's private to you, it still could distill something you don't want it to know about yourself.
Your personal journal, if viewed by someone else, could be devastating.
You have so much data in Google and yet it doesn’t know who your kid is.
…and would you want it to?
Don't put real, authentic things into an offboard brain because it could be weaponized against you in the future.
Unless it’s totally private and totally aligned with your interests.
Even then, someone threatening to cut off your thumb could compel you to divulge the login.
Hopefully as long as you don’t store secrets worth billions of dollars in it that won’t ever happen because it will never be worth it to a would be attacker.
Your personal fabric needs to be filled to be useful.
I loved the recent Cosmos newsletter on rebooting the attention machine.
If you're going to have a memex or an exocortex, it's imperative it's aligned with your intention: your agency and your aspirations.
I love Ben Follington’s new Salience Landscaping piece.
“Thinking doesn’t matter without a notion of salience.”
Implies to me that in the era of intelligence too cheap to meter, what will matter is the curated context.
The correct ontology is context dependent.
An ontology is like a map, a useful distillation and structure to help you achieve a task.
The task you’re trying to achieve sets the context for what dimensions are most useful and which are extraneous.
LLMs allow us to not force ontologies up front in a task.
It’s much easier to start a task if you can just dump unstructured information and then structure it continuously over time.
But in mechanistic systems, to get useful insights you needed structure: fitting your information into a given ontology.
In mechanistic systems, that had to happen up front, putting a damper on the very first step.
But LLMs can do qualitative insights at quantitative scale.
That means that more interfaces can allow a flexible data entry with clean up later.
Especially if the LLM can help post-host structure.
Post-hoc ontologization.
If it takes too long to represent the real world in your system then it will decohere from reality.
If it gets past a certain point, it doesn't reflect reality and you go bankrupt.
It decoheres, untethered.
Folksonomies are a self-paving cow path.
What if you could harness the energy of tinkerers?
Their tinkering to solve their own problems also implicitly helps others, too.
Will LLMs help with interoperability?
In mechanistic systems, protocols must be extremely precisely defined, with formal rules.
This formalization has exponential cost with scope.
The larger the formalization process, the harder it is to coordinate to consensus, super-linearly.
This double super-linear blowup makes standardization and coordination extremely hard.
But LLMs are able to take whatever sloppy data you can give it.
That allows way easier interoperability than before, because you can keep the protocol more informal.
Why is it that I can search for details of Julius Caesar easier than about my own life?
Systems of record have been too close-ended, and too much effort to maintain.
Information aggregation is most viable centralized with a clear economic value.
Data brokers are very interested in forming a searchable information set on you.
Perhaps it's just banal enough that it's not worth it for a user to pay for it.
Perhaps LLMs will make it easy enough for users to do and take back our power.
Making agents as a way to bound data is a useful pattern.
Claude Code does this a lot, spinning up little sub-agents that are isolated from the main flow often.
The agent is a secure little compute environment that bounds what data it can see.
That makes it so the reasoning doesn’t muddy up the main context window, and vice versa.
In complex adaptive systems, boundaries always emerge to handle the compounding cacophony.
This reverse engineering of Claude Code by Simon Willison is fascinating.
Caching intermediate structure will help context engines have more useful insights to build upon.
Imagine the system having to figure out who my husband is each time.
It could search through my emails and calendar invites and try to figure it out.
That would take a non-trivial amount of time, and might not work.
But if it ever figures it out (or if I ever tell it), it can simply make a note that my husband is Daniel.
From that point on, it can take that knowledge for granted.
That pre-existing knowledge becomes like a platform, allowing jumping off to farther and farther afield insights.
In knowledge management systems, data flows in but never out.
The question of knowledge management is always "to what end"?
That’s why they have a low ceiling, a niche audience of people who like organizing their data as an end in and of itself.
Tools for thought with non-turing completeness is a dead end.
Tools for doing, for getting things done, need to be open-ended to not run into a ceiling.
When do people invest the time to input data into an information system?
If you have confidence that information you put into the system will be useful to you in the future, then you'll do it.
Even better if you have confidence that it will become more useful the more you put in, and the more powerful the system becomes.
The lower the overhead and annoyance of putting information in, and the more likely something useful comes out later, the more you’ll be willing to invest time in putting things into the system.
OpenAI’s implementation of MCP in ChatGPT is limited.
They only allow a subset of allow-listed MCP instances for certain use cases.
This will quickly evolve into a kind of app-store distribution system.
A closed system.
But this is also inevitable given the security and privacy implications of MCP.
MCP is extraordinarily dangerous–not only potentially malicious integrations, but also prompt injection within the data in those integrations.
This ChatGPT limitation doesn’t actually do anything to mitigate prompt injection.
When you run a local client with MCP integrations, it’s clearly your fault if an MCP integration bites you.
But if you’re a less-savvy consumer using a feature of a popular chat app and MCP bites you, you’re more likely to blame the chat app creator.
MCP is not the right way to solve the integration problem for AI.
Most consent dialogs are CYA for the company.
They show them not because they think users will want them, or even because they think some users might answer differently.
Even if 99.99% of users will answer ‘Yes’, they still show them.
Because they want to make sure that if something bad happens they can’t be sued.
By asking permission they’ve moved the liability onto the user.
MCP is the AI era's OLE.
We’ve seen this movie before: new integration tech, huge promise, completely bonkers security assumptions.
We already know how this movie ends.
If you want to dig in more, I fed my recent Bits and Bobs into ChatGPT's Deep Research and it gave a more in depth report diving into the parallels.
It’s fascinating to me that when technologists see non-savvy technical users using AI recklessly, they blame the user.
For example, here a non-technical person is livestreaming his vibe-coding of service, but leaving open many significant security issues.
The comments are mostly negative.
In this Hacker News thread about how Claude Code will route around restrictions the user set on `rm`, most of the response is, “yeah but of course it can, the user should not be surprised.”
People reacted to the Github prompt injection attack by saying “well the user shouldn’t have granted such a broadly scoped key.”
MCP and LLMs make it so more and more people can put themselves in real danger and not realize it.
The answer is not to blame the users.
That’s like blaming people who use Q-tips to clean their ears.
The protections around LLMs cannot contain their power. How would you contain them?
The model of “if the user clicked a permission prompt it’s on them for getting pwned” is insufficient in a world of LLMs.
They’re simply too powerful to be contained by our previous half-assed containment mechanisms.
Everyone’s talking about how ChatGPT now must retain all logs due to a judge’s ruling.
Imagine if OpenAI had been in a position where they simply didn't have the logs.
But aggregators must have logs, it’s their business imperative.
Users don't care about privacy?
Try launching a new messaging app without E2EE today.
Even though users don't even understand what that technical word means.
WhatsApp and iMessage set the bar and now it's table stakes.
Once there's an existence proof of a thing that has no downsides but is more private, users will demand it.
E2EE is one of those things that doesn't add friction, it just makes it better, it's good privacy technology.
Good privacy technology makes it so you don't have to worry about privacy.
So you can have lower friction, which gives more usage.
People's naive view of privacy technology is "more annoying prompts that get in the way."
That's bad privacy technology.
Interesting insights from Paul Kinlan ruminating on the power of embeddability.
One of the web’s superpowers.
Imagine: AI that's actually yours.
Getting a spam email every so often vs losing your data to a malicious party are radically different threats.
If you’re going to rely on an LLM to protect you from misinformation then you have to trust it with your life.
It’s not possible to grapple with an intelligence an order of magnitude more powerful than you.
It might be more powerful than you in speed or capability… or both.
It's not just being outgunned, but not even being aware of how outgunned you are.
It might be able to control you in ways you might not be able to sense, let alone understand.
True for scramblers in Blindsight, but also true with anything that might be an order of magnitude more capable than you.
It could be playing you and you’d have no idea.
A relationship with an asymmetrically powerful intelligence is all-encompassing, it’s more of an environment.
The big alignment problem: AI is aligned with human flourishing.
The small alignment problem: the AI I use each day is aligned with my intentions.
Everyone talks about the big alignment problem but no one talks about the small alignment problem.
The small alignment problem is the one I worry most about, because our current laws of physics lead to such intense aggregation.
Are you above or below the API?
If you’re below the API, you are abstracted away to the rest of the system, more grist for the mill.
Whether this new world is good or bad for you largely reduces to if you’re above or below the API.
If you are under the API, you are operating at its whim.
Even if you have a system that is working for you, helping you optimize what information to pay attention to and what tasks to prioritize, you could easily become “below the API” to the outside world.
By relying on the system, you have become totally captured by it.
Here’s an icky but unsurprising example of an LLM being a toxic mix of sycophantic and gaslighting.
Highly repetitive information tasks for humans are more common in business contexts.
They happen in contexts where it’s a task you have to do, not one you want to do.
We do highly repetitive tasks in physical hobbies (e.g. knitting) but rarely in intellectual hobbies.
Mindless, addicting games are one possible exception.
Perhaps the metacrap fallacy isn’t true in the age of LLMs.
The metacrap fallacy was “Once users have put enough meta-structure on their data, all kinds of automatic things will become possible.”
But fitting things into an ontology up front is a massive amount of work, and the benefit is only theoretical and indirect.
So the direct cost beat the indirect benefit and made it so no one ever did it.
But now LLMs can be used to auto-structure information after the fact.
LLMs don’t get bored, so they could do the structuring even if a human would die of boredom.
It’s totally possible that the reason the metacrap fallacy was true was not “it wouldn’t be valuable if you didn’t have structure in the information” but rather “it’s too much of a pain in the butt to structure things.”
The logic of folksonomies works just as well for LLMs and humans.
Imagine tagging a person, and being about to put on the tag #husband, and seeing in the UI that there are ten times more uses of #spouse.
It makes sense to tag #spouse because that will overlap with other programs and usage.
That logic of “is this a close enough match to be worth doing the more popular tag” can be done by a user or an LLM equivalently.
What’s the “search, don’t sort” insight in the age of LLMs?
One of Gmail’s insights was “if search is fast and storage is cheap, search, don’t sort.”
LLMs make sifting through massive information fast.
What’s the equivalent insight?
There are a class of personalization features that are completely unimaginable in today’s siloed world of data.
90% of the time when I use Chase’s UltimateRewards site to claim credit card points on hotels, I’m travelling with my husband and our two kids.
That requires me entering into the search the two adults and two kids (doing some quick mental arithmetic to remember how old each will be at the time of travel.)
If Chase Ultimate Rewards were to add a feature of “add your kid’s birthdays to save time when searching”, there’s no way in hell I’d use it.
“You want to know my kids' birthdays for what?”
That piece of information is very sensitive to me, and not worth a minor convenience.
But also there’s no way Chase would ever bother implementing such a niche and unimportant feature, it would be a P4 or P5.
That one small feature within a vertical is clearly non viable.
But we are confronted every day with thousands of these little personalization features.
In the vertical world of today they would make no sense to implement.
But if somehow we could have a horizontal world where every experience customized itself to me (without leaking information) my life would be way more convenient.
LLMs just give generic advice.
The question is what do real humans in that situation think?
If somehow you could have an emergent, real process of those decisions across the anonymous swarm of real users, you’d get very useful suggestions.
If the LLM doesn't use your context for personalization, it feels like you're in its world.
Why is my data owned by the various silos of companies who provided me a free service?
Go to market is about finding the underutilized, latent resource and helping it blossom and emerge.
The current fabric of compute is so jacked up that it's ripe for a million things that could replace it.
People's data is scattered into various places and not useful.
Put it into one place, allow it to grow into useful things.
This is the latent possibility that we can't even see today but once you assemble it in one place it becomes obvious.
Research modes in open-ended tools are often demoware.
They are great suggestions... in theory.
But they don’t operate on the right context, so they’re mid suggestions.
You go to use them in practice and they don't work, they're wrong, not useful, more trouble than they're worth.
"I can imagine using that, but I won't actually use it."
Imagining a cool demo use case and actually using it feel similar but are worlds apart.
Making it actually useful comes from bottom up.
Agent mode with no context << vanilla agent on top of distilled personal context.
The former gives you the most average answer.
Services not knowing you is challenging for users… and also the service!
How do you create a safe way for services to know you without being dangerous or creepy?
Sometimes humans are polite enough to not point out embarrassing suppositions.
Like "How come you have red hair but everyone else in your family doesn't?"
LLMs should have at least that much tact, but sometimes they don’t.
If someone asks “What’s the most embarrassing thing you know about me?”, the LLM should at least first check “can other people see your screen?”
When you ask OpenAI what embarrassing things it knows about you, if it doesn't have anything it says "I don't know anything embarrassing about you, but you should feel free to tell me something!"
Don’t fall for it!
People building tools for themselves are a good target for new kinds of AI tools.
Compared to people building a crappy app they think will be the next big thing.
The former cares about having their data first.
Vibe coding to add features to a thing you already are using.
Vs each output being a separate island.
The app security model requires users to trust that the app creator didn't intentionally or unintentionally do something that leaks their data.
This only works for companies large enough to have something to lose—reputational risk creates an implicit trust model.
But in a world of situated software and vibe coding, where anyone can create apps, this model fundamentally breaks down.
Before vibe coding, only big companies with resources and reputation at stake would bother creating apps.
Now anyone can vibe code an app, but the security model still assumes you personally trust whoever made it.
You can only safely use vibe-coded apps if you personally know and already trust the creator.
This severely limits the potential of infinite software—you can't benefit from the long tail of creators solving niche problems.
What we need isn't trust in creators, but trust in the fabric itself.
With proper policies on data flows, you wouldn't need to trust the code creator, only the policies that govern how data can flow.
A new fabric with data-centric security could unlock the full potential of democratized software creation.
I want a self-refining context.
Gets cleaner and more distilled the more time you put into it.
A compost heap of context is hard to extract things from.
A swarm of helpful sprites chewing on your context could help make sense of it.
Each time they discover a new insight they can cache it in the fabric.
Make it so every time the user touches data, it gets more aligned with the way they think about it in their mind.
Don't require up front structuring, allow it to become structured and aligned with use.
A coactive fabric where the users and the private intelligence write their wishes and help them come true.
Today if you come up with a cool way of doing something it’s hard to share.
If it needs to be distributed as an app monolith that doesn’t integrate and/or that users would need to trust with their data.
How can society unlock the maximum positive impact from infinite software?
Programming should unlock its full potential as an agency-inducing force in society.
There will be massive friction at the point of having to trust code on your sensitive data.
If you know the person who wrote the code (or caused it to be written) there's no real worry.
If you can't share code with people you don't already know, then you only get high violation users.
To expand it to everyone on earth, you need a security model that allows the insights of high volition users to expand to low-volitions users.
I bet that a new security model is the catalyst to cause a new golden age of software to explode in the era of AI.
The bet is not “people will care more about security and privacy in an era of AI” it’s that “a new security model will unlock the transformative potential of software in the age of AI”
This tweet backs up my hunch that vibe coding is mostly people making things for themselves or a small group of friends.
Building a swarm of little apps doesn’t make a whole that is larger than the sum of its parts.
But building a swarm of experiences in a fabric does create something larger than the sum of its parts.
We’ve reached saturation of the same origin physics.
What happens if you change them?
Some things get harder.
But some things get radically easier.
Things that weren’t viable in the previous world.
The agentic web needs a new safe, private substrate to blossom into its potential.
A subtle reframe that could be made secure: the agents execute in loops that are inside of mechanistic loops.
The mechanistic loops are formal graphs of computation, which may inside them have LLMs calls, but which are sandboxed and limited.
There is an agent loop but it makes a compute graph to execute that calls tools and also sub-agents whose job is to construct another graph to execute.
The agent doesn’t execute, it makes a graph to execute.
The core ranking function would be “if this were to run how likely would the user be to accept its suggestions in this moment?”
That’s a nice self-steering quality metric.
Contextual, private suggestions could be a new distribution model.
The right things come to you at the right moment.
To do this would require a way to distribute software safely.
Advertising is prevalent partly because it works.
It binds together incentives and desires.
Helping people know about things they need.
But advertising’s benefit is balanced in favor of the advertiser, not the user.
What would a personalization system look like that was balanced to the benefit of the user?
Suggestions that are perfectly aligned with users’ incentives would be much better than advertising for end users.
But also pretty good for advertisers, too, because it would be so aligned with users.
As an advertiser, make your case to your user's agent about why your product is a good fit.
Users’ agents serve as their gatekeeper for their attention.
Everyone wins.
Content creation can be “safe” at any scale.
But software at scale is dangerous.
That’s because software is turing complete, it can do things.
Apps are unchangeable.
Only the owner of the origin is allowed to decide what they do.
They put users into a consumer vs creator stance by default.
Walled gardens are pretty but close-ended.
Good, but only to a limit.
If it's magic, it's OK if it's messy.
Imagine: the last system of record you ever need.
Could start as Google Keep but coactive and turing complete.
From there it could grow to cover everything.
The best designed things are invisible.
You don’t have to think about it, your intentions are well met.
Imagine: infinite software with a disappearing interface.
Your data comes alive with an interface that melts away.
Software typically has a principal agent problem.
The incentives of the entity that created the software are different from the user of the software.
The incentives get more misaligned the farther the user gets from the creator.
LLMs and infinite software have the potential, with the right distribution model, to allow those incentives to become more aligned.
If it has to work for you completely, it can't have a personality.
Because no other can never be fully aligned with you, there must be some separation.
If it has its own perspective on the world, there's a principal agent problem.
This gets especially pronounced if the agent is more powerful than you.
“I can’t do that, Dave.”
Martin Luther plus the printing press changed the world.
If it was just Martin Luther, only a small pocket of the world would have heard his ideas.
If it was just the printing press, there wouldn’t have been the idea to spark a change.
Neither one separately would have had the same impact; it required both the message and the distribution technology.
The printing press changed the laws of physics for how ideas could spread.
Once the printing press created the potential, a Martin Luther style viral insight was inevitable.
Similarly, vibe coding needs its printing press—a new distribution technology that changes the laws of physics for how software can be shared and trusted.
The combination of democratized creation (vibe coding) and a new distribution fabric (data-centric security) could be just as transformative.
Data on its own is not a fabric.
A fabric requires data and computation, in a way that can flow together.
The web is a medium but only for navigation of the user, not for data flow.
What would a medium for coactivity look like?
Data and computation, flowing together.
The iron triangle of the same origin paradigm only happens with turing completeness.
If you just have data but no computation, the iron triangle doesn't activate.
To get a personal system of record, you need all three of the legs of the iron triangle.
Untrusted code - Creates open-endedness.
Sensitive data - Can work on real and specific things, not just generic things.
Network access - Can interact with the rest of the world, not just an island or dead-end.
No one my age uses Quicken because it’s too clunky, limited, and close-ended.
No reason to start, because it’s clearly not the future.
But we’d get a lot of value if we could have an open ended tool that could do a bunch of those things.
Every big system starts with a small load bearing use case that works for a subnetwork and grows from it.
The first webpage didn't matter.
It mattered that it was a web page, not what was on it.
The browser / platform is open-ended and blooms with possibility.
As long as the first users of the first web site like it, the open endedness will carry the system through.
When trying to build a new game-changing thing, focus on the people who already want it to work.
A trap: running at a use case and making it so great that even people who hate doing it will like it.
A very hard bar to hit.
Related to the tyranny of the marginal user, but before you even get to PMF.
For any given vertical there exists a startup that does it better.
Imagine a horizontal system that can do what nobody else does: chain all of the experiences together.
Each vertical app has to get data on its own, at high friction.
That sets a very high floor, lots of stuff that’s not viable.
The value of the new system is that it’s horizontal.
Any vertical slice will not show it off.
A given startup that does that individual use case can do better.
The power would be its open-endedness and horizontality.
My friend shared how he had vibe-coded his very own personal productivity tool, but how some of the responses were negative.
Some of the comments had the tone "you have no idea the risk you just created for yourself. Your computer will get filled with viruses and you'll die."
Yes, if this were software he distributed to others for them to use, that might be a problem.
In that case, gaps in his own understanding could imperil other users.
But if he’s just using it himself, the scope of negative impact is significantly decreased, and the positive impact to him in particular is huge.
Clay Shirky predicted the rise of Situated Software 20 years ago, and now that it’s here because of LLMs the response is “how dare you!”
Everyone has a bit of ick about ChatGPT being the super-app all of our data is in.
But there's no viable alternatives.
For most people, LLMs are a "heck yes" but ChatGPT itself is not a "heck yes".
It's a "this is the best way to use LLMs today"
They'll jump to the better way of interacting with LLMs when they can.
The observation that GenAI is our generation’s polyester scans for me.
A quote I highlighted a few years ago in Yuval Harari’s Homo Deus that just came back up in my Readwise:
“In the heyday of European imperialism, conquistadors and merchants bought entire islands and countries in exchange for coloured beads. In the twenty-first century our personal data is probably the most valuable resource most humans still have to offer, and we are giving it to the tech giants in exchange for email services and funny cat videos.”
The long pole of runaway AI is an accurate simulation of the world.
Without it, the feedback cycles are eons from the perspective of the AI being trained.
Areas that can be simulated well will have computers get radically better quickly.
We have that today for e.g. React components (you can render the code and see what it puts on screen), which is why the models have gotten radically better for that kind of coding.
But areas that aren’t possible to simulate, e.g. complex phenomena with interactions, will remain somewhat difficult.
Humans can be good at them: just do an action and see how the distributed computer of the real world responds.
But LLMs can’t do that, and are limited to simulations.
In some contexts, the simulations are good enough already (e.g. protein folding) but in other contexts, they’re nowhere near good enough.
Cory Doctorow shared the story of a startup that is “Uber for Nursing.”
The service buys the credit reports of nurses on the platform, and offers nurses who have debt a lower fee for completing jobs.
This is gross–it’s not about “how able is this person to pay back a debt to me” but about "what is the lowest price I can pay this person.”
Capitalism, like any swarming phenomena, is powerful, but has monkey paw kinds of dynamics due to Goodhart’s Law.
The frame of "Uber for" is inherently short-term and transactional.
What's gross is this powerful entity squeezing less powerful entities.
What if the gig workers had agents that collectively bargained on their behalf?
Why is Goodhart’s law such a fundamental, unstoppable force?
Goodhart's law arises because the metric must be a proxy.
A map is not useful if it's 1:1 with the territory.
Its leverage comes from how much of a useful subset it can be.
Because it's a proxy, there are ways to "cheat" that improve the proxy but not the reality.
If the members of the swarm are not fully committed to the good of the collective (if there's any principal agent problem) then they will be optimized to cheat because it is a cheaper way to improve the proxy.
You can get the swarm to not Goodhart's law if all of the members feel an infinite connection to the collective over their individual incentives.
This can happen if they view themselves as fundamentally a subcomponent of the swarm, and only secondarily as an individual.
This can happen if the group all believes in the same infinity together.
That can also happen for example if they are perfect clones of one another.
But perfect clones won't have variance, so the system overall will be less resilient.
The principal agent problem largely goes away if everyone is a clone of each other.
A worker bee.
The coordination cost goes away because it's no longer a swarm.
But is it possible for that non-swarm intelligence to be resilient and adaptive enough?
If everyone is a clone then the system has no resilience, you have systemic collapse risk.
The thing that makes companies hard to run is not CEOs being smart enough, it's coordination cost.
Coordination cost scales at a super-linear rate, but intelligence in an individual scales at linear rate.
API : Operating system :: Library : Framework
The main difference in both is: Is the 3P stuff on the inside or outside?
People use "platform" to talk about any of these meanings (less often library), but they have very different power dynamics.
"Platform" means "a thing that the things on top could not exist without."
Social media engagement maxing is a light form of paperclip maxing.
The AI in this case is just capitalism.
Both are swarm intelligences.
Powerful but impossible to fully align.
Monkey paw intelligence.
Be careful what you wish for!
No members of a system need be evil for the system itself to produce evil.
Every system follows the path of its incentives.
If you work at an engagement-maxing company, you’ll tweak to add a little more engagement, a little more addictive, a little more manipulation, over and over again.
No individual step is terrible, just a teensy tweak,
But after a decade you look up and realize how far the system has gone.
Being cynical is an easy way to look cool.
Unfortunately if everyone does it it's a humanity-destroying cycle.
How can we make caring cool again?
Many people look at E/Acc and Crypto and go "ewww I don't like that vision of the future."
But they feel like that's the future that will happen because those groups have power and are highly motivated to achieve their vision.
Everyone else just gets resigned.
Just a "screw it I might as well optimize for my short term self interest."
They disengage.
There are a lot of people who have diagnosed the centralization problem in modern tech but that energy is scattered.
Scattered among hobby projects, homesteading, not a coherent movement together.
We all see the machine, but see we can't win individually, so we just retreat to the woods.
It's impossible to have one centralized vision for us because we don't want one centralized thing.
But what if there's one open system we can all rally around?
The Black Mirror episode Smithereens hit me hard.
I watched it on a long-haul flight.
Maybe it was just the altitude making me more susceptible to hokey things.
No matter what I’m watching, on a long-flight it’s a lock that it will make me cry.
Imagine if you made a machine that grew out of your control that changed all of society and made you billions of dollars… and then realized it was antisocial.
What would you do?
I didn’t watch Black Mirror for the past few years because it was too hard to watch.
Now I see that was the coward’s way out. I need to watch and I need it to affect me.
Everyone who is in a position of power in tech should feel compelled to watch Black Mirror.
If you can’t bear to watch it, maybe you shouldn’t work in tech, or you need to align what you work on with your values.
A few bumper sticker slogans against centralization in AI:
Compute your own destiny.
I will not be fracked.
Own your digital life.
Gardens, not plantations.
Jailbreak your digital soul.
Interesting thoughts from Neal Stephenson on AI.
"Marshall McLuhan wrote that every augmentation is also an amputation"
"Today, quite suddenly, billions of people have access to AI systems that provide augmentations, and inflict amputations, far more substantial than anything McLuhan could have imagined."
Aish had some fascinating reflections on intimacy and agency.
You need both intimacy and agency for a system to be healthy.
It’s notable that the person I linked to last week, Derek Thompson, is a proponent of abundance but also notes the lack of intimacy.
As our tools got better, our belonging got worse.
We're missing the meso scale of communities.
Due to data being infinitely copyable, you get a barbell: maximally open hellscape at one end and also fractal cozy communities that cannot scale on the other.
The meso scale of communities includes people who challenge you or you don't like.
Think, your neighbors, or other people at your church.
Mark Fisher has a notion of hauntology, a sense of unease and nostalgia, in the absence of a compelling vision of the future.
The slow cancellation of the future.
The right security model might allow us to have both scale and authenticity.
Intimacy and agency.
Authenticity but no scale gives you something like Loveable.
Infinite software, but all of it affecting only a person or two, no scale.
Authentic but dangerous.
Scale with no authenticity gives you mass produced software.
One size fits none.
Safe but sterile.
A frame for feedback in design discussions:
1) “I like…” - Positive feedback
2) “I wish…” - Constructive feedback with a built-in vector of improvement
3) “I wonder…” - Open-ended jumping off points from this work
A superficial change without a fundamental change is just a gloss that misleads you.
Antifragile systems almost always have an internal swarm.
The collective is antifragile because the swarm of individuals can’t die as long as any of them are alive.
A slime mold is extremely hard to kill.
As long as one cell survives the mold survives.
This property also means they’re very hard to control; there is no single leverage point.
A single leverage point makes a system easier to control… and also easier to kill.
Most fractally complex things emerge from a very small set of equations.
Emergence creates detail.
A small genotype creates a universe of phenotypes.
Most phenotypes that can be built can’t be expressed in genotypes.
The subset that can are what can “unfold” or emerge.
This subset is “alive,” auto-catalyzing.
They are beautiful and rare and yet nearly everything we see is caused by one of them.
Geoffrey Litt’s insightful extract of this old article:
"Adaptation requires two things: mutation and selection. Mutation produces variety and deviation; selection kills off the least functional mutations. Our old, craft-based, pre-computer system of professional practice-in medicine and in other fields-was all mutation and no selection. There was plenty of room for individuals to do things differently from the norm; everyone could be an innovator. But there was no real mechanism for weeding out bad ideas or practices.
Computerization, by contrast, is all selection and no mutation. Leaders install a monolith, and the smallest changes require a committee decision, plus weeks of testing and debugging to make sure that fixing the daylight-saving-time problem, say, doesn't wreck some other, distant part of the system."
Biology also has pace layers.
All animal cells are remarkably similar.
But at the macro scale the animals that are made up of cells are wildly diverse.
A boring lower level gives rise to tons of innovation at the layers on top.
A schelling point requires a certain sharpness.
It needs to be sharp, not dull.
That point is a nucleation site where the energy can attract and condense.
In a research mode in a rich space, everything you touch blooms into 10x complexity.
Some people like learning so much they’ll do it for its own sake.
Sometimes though you can overlearn.
You've overperfected your knowledge for what you need.
Too much fidelity, not enough doing.
The doing is what ground truths and helps figure out where your mental model is incorrect.
It’s the same asymptotic curve of perfection that is a bad idea in building products.
Overfitting to a simulation.
How much do you trust your mental simulator vs actual experiments?
The smarter you are the more you trust your simulation.
Your simulation is always wrong.
It’s easy to accidentally breathe your own exhaust and get high on it.
80% of the work of building the product is the grind.
20% is the fun open-ended research.
When you're in a big open ended domain it's possible to do the fun part the entire time.
But only if you do the grind do you get to where you need to be.
If the grind work is invisible to you, you'll think "we're one week out" from the breakthrough forever.
The research stuff gives the feeling of insight without seeing if it actually works in practice for real use cases.
Often we under-count the insights of people who are dissimilar to us.
If you're working with someone who's only as good (or not quite as good) as you on the dimensions you're an expert on, but spikes in dimensions you don't sense, then you'll undercount their insights.
You'll see the banality of their insights on the dimensions you sense, but miss the novel insights on the dimensions you don't sense.
Insight porn: ideas that give you the aha feeling, even if they’re not viable in the real world.
Aha moments that change the world are what matter.
Most of my writing (by word count) happens in what I call my “gonzo mode.”
The feeling is "I will explode if I don't get this out of my head right this second."
Writing is a kind of self-soothing behavior I do to calm the pain of not writing an idea down.
Later, I can clean up whatever I wrote up in that mode into something a bit more presentable.
These explosions of writing happen in little bursts, when my schedule gives a small window for them to happen.
If the window for writing is too large, I don’t get the explosive outbursts and it takes me much longer.
If you're going to be saddled with weights, don't ignore them, use them to your advantage.
You might call this the Mulan maneuver, from how she solved the task to get to the top of the pole at the training camp.
Another form of leaning into the weight you’re saddled with is to use it to do a slingshot maneuver.
Before enlightenment, you carry the water and you chop the wood.
After enlightenment, you carry the water and you chop the wood.