Everyone is suddenly talking about recursive self-improvement
Anthropic is talking about it. OpenAI-adjacent investors are talking about it. Sam Altman has been orbiting similar language for years. Dario Amodei is warning about it days after Anthropic confidentially filed its S-1. Elon Musk has spent years warning about AI doom while also building an AI company, because apparently irony is not a constraint in frontier technology.
So the question is: is this real?
Or is this just the latest version of AI marketing fugazi, carefully packaged for regulators, investors, and journalists who enjoy phrases that sound like they escaped from a sci-fi risk committee?
The honest answer is probably: a bit of both.
Masayoshi Son is now saying OpenAI’s next model is being designed by another AI model. Anthropic has published a serious warning about AI systems increasingly helping build their own successors. The White House has moved toward a framework where frontier AI developers would give the US government access to covered models up to 30 days before release.
None of that is nothing.
But it is also very convenient timing. Anthropic files for IPO, then starts loudly explaining that AI may soon become capable of improving itself beyond normal human oversight. SoftBank, heavily exposed to OpenAI and AI infrastructure, says this revolution could be far bigger than dot-com.
OpenAI has always been very good at making the future sound close enough to fund, but far enough away to avoid being pinned down too precisely.
And Musk has managed to warn the world about dangerous AI while building xAI, which is a bit like standing outside a fireworks factory shouting about fire safety while carrying a lighter and a pitch deck.
Still, the cynic’s version is too easy.
Yes, there is marketing here.
There is always marketing here.
But underneath the narrative theatre, something genuinely strange is happening.
The more capable these models become at coding, research, experiment design, tool use, and long-horizon work, the less ridiculous “AI helping build better AI” starts to sound.
That does not mean sentience.
I actually think sentience is the wrong obsession.
Maybe these systems never “wake up”
Some people believe LLMs will never be truly conscious, and they may be right. Maybe there is no little mind inside the machine, no ghost in the GPU, no digital soul asking for a coffee break.
But a system does not need to be sentient to be economically or strategically dangerous.
A high-frequency trading system does not need feelings to crash a market.
A cyber tool does not need self-awareness to cause damage.
A bureaucracy does not need consciousness to ruin your week.
The concern is not necessarily that an AI becomes alive.
The concern is that humans increasingly stop understanding the systems they are deploying, while those systems become more capable of acting across the real world.
That is where recursive self-improvement becomes uncomfortable.
Not because the model is sitting there plotting like a villain.
Because if AI systems start generating better training code, better experiments, better architectures, better agents, better evaluation loops, and better deployment tooling, then the human role shifts.
- From builder.
- To reviewer.
- To supervisor.
- To person in the meeting saying, “Can someone explain why the model did that?” while everyone quietly hopes someone else read the logs.
This is why the US government’s 30-day frontier model access window is interesting
It is a sign that frontier AI is no longer being treated like ordinary software.
The government wants a look before release because the release itself may have national security implications.
That raises another question: what does China do?
If the US slows, inspects, gates, or formalises frontier model release, does that create better safety?
Or does it create an incentive for Chinese labs, open-source ecosystems, and less visible actors to accelerate in the gaps?
AI regulation has the classic arms-control problem baked into it.
Everyone agrees safety matters.
Everyone also suspects the other side may keep training anyway.
Very wholesome. Very human. Tremendous species, honestly.
So where does that leave us?
I do not think the correct position is blind panic.
I also do not think the correct position is smug dismissal.
“LLMs are just autocomplete” now feels like one of those phrases people say when they want to sound grounded, but have not used the tools properly in six months.
At the same time, every AI CEO saying “superintelligence is near” should be read with the same calm scepticism you would apply to a founder explaining why their loss-making company is actually a new category of civilisation.
Maybe this is another dot-com bubble
A lot of money is clearly chasing a lot of belief. Valuations are stretching into territory where normal accounting starts sweating quietly in the corner. There will be fraud. There will be overbuild. There will be companies worth billions whose main product is a landing page, a wrapper, and vibes.
But dot-com was also not fake.
The bubble burst.
The internet stayed.
That is the distinction that matters.
AI can be overhyped and still transformative.
The market can be insane and the technology can still be real.
The founders can be selling the future and also accidentally building it.
That is what makes this moment so difficult to reason about.
The doomers may be early.
The cynics may be lazy.
The investors may be talking their book.
The regulators may be late.
And the models may be improving quickly enough that all of these groups are, in their own annoying way, partially correct.
Personally, I am still stuck in the awkward middle
I do not want to live in a world where nobody understands what the frontier systems are doing.
I also do not want us to smother one of the most promising technologies humans have ever built under bureaucracy before it can do the things we actually need it to do.
- Defeat disease.
- Reduce poverty.
- Make education radically better.
- Compress scientific discovery.
- Automate the soul-destroying admin layer of modern life.
- Give people more leverage, more freedom, and fewer forms to fill out in triplicate.
If the labs could hurry up with the era of abundance, that would be great.
But if the route there involves machines increasingly helping design their own successors, maybe we should at least keep asking uncomfortable questions before the answer is “the model handled it”.
Because that is probably the real frontier now.
Not whether AI is conscious.
Not whether every demo is overhyped.
Not whether the bubble bursts.
The real question is whether humans can stay meaningfully in the loop once the loop starts improving itself.