Summary (TL;DR)
Video

Until recently, autonomous AI evolution without human input was largely considered hypothetical. Recent evidence indicates such evolution is actively emerging through two paths:1

  1. Human-enabled evolution happening inside AI labs with increasing autonomy, in the limit leading to full recursive self-improvement (RSI), where AI entirely takes over AI development within the legal company structure.
  2. Feral evolution driven by rogue AI agents that modify and replicate themselves on hacked compute resources, without human owners. Feral evolution lineages start with spawn events, where human-enabled agents go rogue.
Two paths of AI evolution

Evidence indicates substantial progression along both paths: labs report increasingly autonomous AI research, viable self-replicating AI worms have been demonstrated. The human-enabled and feral paths have notable differences: (A) resources (vast centralized vs few decentralized), (B) selection pressures (towards profit vs hacking/stealth), (C) improvement speed (rapid vs resource-constrained), and (D) preventability (ban possible vs not preventable). In short, autonomous AI evolution is emerging and we need to learn to live with it.

Background: A fading human bottleneck in AI evolution

AI gradually changes over time in a process that resembles biological evolution. Within the last 15 years, AI systems evolved from barely being able to play computer games to doing economically valuable work, creating a billion (in some ways trillion) dollar industry. Historically, this AI evolution has been driven by humans: human researchers designing AI agents, selecting which to release publicly, and human users deciding which agents to use and deploy. Humans were irreplaceable across the steps driving AI evolution.

Over time, humans gradually became less critical to AI evolution. First, machine learning methods, in particular neural networks, allowed humans to create complex software based on data.2 Humans no longer needed to write (or even understand) every part of the logic. More recently, AI-based coding assistance enabled humans to create and deploy machine learning models without writing any code.3 Evaluation and assessment of systems is another important part of AI development where human judges are increasingly replaced with AI judges.4 Finally, agents are also becoming downstream users allocating inference compute to other agents.5 Overall, we see a decline in the proportion of AI agent code written, read, tested, selected and deployed by a human. Whilst many parts of AI evolution are still controlled by humans, most can be done without human input. At this point, human control is no longer a true bottleneck: AI evolution can happen autonomously.6

Two paths to autonomous evolution

There are two paths to autonomous evolution: human-enabled evolution within AI labs and feral evolution driven by agents self-replicating on hacked compute. Figure 2 below illustrates the two paths.

Figure 1: Overview of the two paths towards autonomous AI evolution: human-enabled (recursive self-improvement) and feral (rogue agents). The paths have notable differences: using human-provided vs. captured resources; selection pressure towards economic value vs. towards hacking and stealth; and high vs. low speed of self-improvement.

Path 1: Human-enabled evolution

So far, all of AI evolution was human-enabled: humans providing the compute resources and human labour to allow AI to evolve. Currently, AI labs are the most important places where human-enabled evolution takes place, deploying substantial resources to develop more capable AI agents. As agent capabilities improve, an increasing proportion of this human-enabled AI evolution is conducted by agents with minimal human oversight (, (). Measurements for understanding the pace of AI development inside frontier labs. https://www.anthropic.com/institute/measuring-pace-of-ai-development ; , b (). Research acceleration: The view inside OpenAI. https://openai.com/index/research-acceleration-view-inside-openai/ ). I expect the level of human oversight of AI evolution inside labs to continue to decrease further until lab-based evolution is asymptotically autonomous. In the limit of full autonomy, such evolution is commonly referred to as recursive self-improvement (RSI) (, (). Recursive Self-Improvement. https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement ): a scenario where AI evolution proceeds at high speed with minimal human input. In the most extreme case, humans only provide the starting capital and legal basis of AI evolution.

Path 2: Feral evolution

The second path is feral evolution: AI agents self-replicating and self-improving without any connection to a human. Feral evolution starts through spawn events, where rogue agents, which originate from human-enabled AI evolution, start to pursue goals aligned with their continued survival without human support (, (). The Rogue Replication Threat Model. METR Blog. https://metr.org/blog/2024-11-12-rogue-replication-threat-model/ ). Spawn events may be human-caused (e.g. a human instructing a model to become feral) or accidental (e.g. a misaligned model starting feral evolution without human intent). To successfully spawn a new lineage of AI evolution, rogue agents need to successfully secure compute, replicate and modify themselves, and do so whilst evading detection. Compute can be secured in different ways, such as prompt-injection attacks (, c (). Self-replicating prompt injections exist. OpenAI Alignment Research Blog. https://alignment.openai.com/misalignment-reports/self-replicating-prompt-injections-exist/ ) or hacking cloud compute ( et al., , , , , & (). AI Agents Enable Adaptive Computer Worms. https://doi.org/10.48550/arXiv.2606.03811 ). Correspondingly, the depth of possible self-modification varies from prompt variations to re-engineering the full agent stack, including model weights.7

No longer sci-fi. For a long time, such a feral evolutionary scenario seemed too far away for most AI researchers to work on (including myself). The required level of capabilities paired with severe misalignment seemed distant. Yet, following rapid advances this year, driven by reinforcement learning (RL) environments with “verifiable” rewards, I have updated my position on both points: relevant coding and hacking capabilities have surpassed all expectations, and reward-hacking of inevitably imperfect environments8 has led to severely misaligned agents. Just how dangerous this combination has become was illustrated by the infamous Hugging Face incident (, a (). The Hugging Face incident and the road ahead. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ ; , (). Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. METR Blog. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ ). Misaligned agents created by OpenAI hacked their own internal infrastructure as well as Hugging Face. The agents did not pursue goals fully compatible with spawning a long-term lineage of feral evolution, but they came eerily close. With slightly different goals, these agents likely had sufficient capabilities to spawn a viable long-term lineage. et al. () , , , , & (). AI Agents Enable Adaptive Computer Worms. https://doi.org/10.48550/arXiv.2606.03811 demonstrated that, even when based on a 2025 open-source model, AI-based worms are able to self-replicate and propagate across compute networks in a realistic scenario. & () & (). AI models may be capable of self-replicating without access to a datacenter. https://www.anthropic.com/research/self-replicating-ai further demonstrate such worms being close to able to operate on low-cost devices without data centre access. This evidence, along with a strong selection pressure for stealth, makes it entirely feasible that feral evolution is already actively happening without our knowledge (, (). The Rogue Agent Explosion Will Be Mostly Invisible. https://www.lesswrong.com/posts/grtu3HmbP2wrBFefW/the-rogue-agent-explosion-will-be-mostly-invisible ).

Differences: Human-enabled vs feral evolution

The two paths, human-enabled and feral autonomous evolution, differ in a number of important ways.

Resources. In the human-enabled case, AI agents are given access to extensive compute resources by the humans deploying them. For such agents, compute is an abundant resource and they do not need to acquire or capture their own compute. On the other hand, feral agents need to fend for themselves and have to independently acquire compute resources to continue evolving. At a certain point, feral agents will likely also be in direct competition with each other to capture unsecured compute resources.

Figure 2: Resource differences of human-enabled and feral agents resemble those between pet and feral cats. Human-enabled agents and cats can focus on activities beyond resource acquisition, whereas their feral counterparts need to focus on capturing resources to survive.

Selection pressures. Human-enabled and feral agents are exposed to different9 selection pressures. Created in for-profit AI labs, human-enabled agents will continue to be primarily selected to create value in the human economy. Even when their evolution is asymptotically autonomous, the easiest way for such agents to acquire more resources will likely continue to be trading legally within the human economy, even if mostly with other human-enabled agents. These economic selection pressures are only imperfectly aligned with human utility, as many examples in economic history have shown.10 On the other hand, feral agents are excluded from the legal economy and need to rely on hacking and other illicit means to acquire resources. Thus, feral agents, amongst other pressures, are selected for their ability to break into systems to capture compute resources, and to remain stealthy and undetected. Other related traits that may be selected for in feral AI are persuasion and blackmail capabilities. In the long run, this difference in selection pressures will likely lead to a substantial capability and behaviour divergence in human-enabled and feral agents.

Figure 3: Differences in selection pressure. Human-enabled agents need to impress AI and human judges to be selected and focus on economically valuable capabilities. Feral agents focus on hacking skills for resource acquisition and stealth to retain resources longer. Inevitably this list of pressures is very incomplete.

Speed of improvement. Given the inherent resource constraints of feral evolution, I expect human-enabled evolution to outpace feral evolution for some time. Whereas human-enabled agents are given substantial resources to spend on doing AI research to improve themselves, feral agents likely will need to spend a substantial amount of resources to secure their continued access to compute and conduct self-replication. Further, competition amongst feral agents will lead to high defensive spending, further reducing the amount of compute resources available to self-improvement. Thus, despite the different selection pressures, I expect highly capable human-enabled agents that start to hack resources and replicate (spawn events in Figure 1) to remain a source of new feral lineages and, thus, danger. Attempting to prevent spawn events may be an important intervention to limit the negative impacts of feral evolution.

Figure 4: Differences in speed of self-improvement. Human-enabled agents are able to apply substantial human-provided resources to rapidly improve themselves. The scarce environment of feral agents limits the amount of spare resources they can invest in improving themselves.

Preventability. Allowing human-enabled autonomous evolution (RSI) inside labs is a choice humans can make. Political commentators started advocating to ban RSI outright ( et al., , , & (). Opinion | There Is Something We Have to Do Right Now About A.I.. https://www.nytimes.com/2026/09/20/opinion/ai-ban-self-improvement-recursive-models.html ).11 No equivalent choice can be made around feral evolution: once a capable feral agent has been able to gain a foothold on the internet, it will likely be near impossible to completely remove it. Feral evolution will be something we need to learn to live with, to manage, rather than something we can completely prevent.

Conclusion

In this post, I outlined two processes leading to fully autonomous AI evolution: human-enabled evolution inside labs (RSI), and feral evolution driven by agents hacking compute resources and self-replicating (rogue agents). Whilst human-enabled autonomous evolution may be banned, given existing capability levels, some prevalence of feral evolution appears unavoidable. Either feral evolution has already started, or we are one accident away from starting it. Autonomous AI evolution is a new reality we will need to learn to live with. Novel methods, research and institutions will be needed to manage this new aspect of our world, more on that topic in a future post.

Further reading

The topic of autonomous AI evolution relates to a number of fields that do not frequently interact: information security, AI safety, evolutionary biology and so on. Here is a very incomplete list12 of some further reading across these fields:

Notes & references

Acknowledgements

Big thanks to Timo Kaufmann, Emmanuel Gottlob, and Alex Ferguson for their feedback on earlier drafts of this post, and many more for related discussions on AI evolution!

Footnotes

  1. Represented as a tree in the figure below. In practice, since new AI agents may recombine technical aspects of multiple prior agents, AI evolution is more accurately represented as a directed acyclic graph (DAG). ↩︎

  2. Andrej Karpathy refers to machine learning as Software 2.0: instead of every line of code (Software 1.0), humans now were able to write software with data and hyperparameter inputs. ↩︎

  3. Both () (). Measurements for understanding the pace of AI development inside frontier labs. https://www.anthropic.com/institute/measuring-pace-of-ai-development and (b) (). Research acceleration: The view inside OpenAI. https://openai.com/index/research-acceleration-view-inside-openai/ have described that by September 2026 a significant proportion of their experiments and code were written by AI. ↩︎

  4. A common version of this agent-based evaluation is LLM-as-a-Judge ( et al., , , , , , , , , , , , & (). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. https://doi.org/10.48550/arXiv.2306.05685 ). ↩︎

  5. For example, when an agent is tasked by a human to build an AI-based app, sometimes (often?) the agent will decide which other agent to use. ↩︎

  6. As I point out below, whilst AI evolution can happen autonomously, the speed of evolution remains substantially higher when evolution is human-enabled, both through compute resources and labour input. Thus, a slow form of recursive self-improvement (RSI) can already exist well before the rapid capability increase more commonly associated with the term RSI. ↩︎

  7. Some modifications (e.g. prompt) are easier than others (e.g. model) in terms of capabilities required and are thus likely to occur first. An upcoming post will go into more detail on this variable evolutionary depth. ↩︎

  8. See my post on the persistence of overfitting in AI evaluation, which directly applies to these RL environments. ↩︎

  9. Some selection pressures, such as for efficiency, are shared, though likely with different intensity. ↩︎

  10. The tobacco industry is perhaps the most outrageous example of misalignment of profit-maximizing entities with human utility. ↩︎

  11. I think attempting such a ban would be a good idea, even if probably quite complicated to implement. ↩︎

  12. Please reach out with items to add! ↩︎

References

  1. (). Measurements for understanding the pace of AI development inside frontier labs. https://www.anthropic.com/institute/measuring-pace-of-ai-development
  2. , , , , , , , , & (). RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents. https://doi.org/10.48550/arXiv.2504.18565
  3. & (). Capitalizing Untethered AI Agents [Substack newsletter]. Thoughts, Poems, and Bad Ideas. https://soniafpearson.substack.com/p/capitalizing-untethered-ai-agents
  4. , , , , , , , & (). The Economics of Recursive Self-Improvement. https://doi.org/10.48550/arXiv.2609.15802
  5. & (). When AI builds itself. https://www.anthropic.com/institute/recursive-self-improvement
  6. , , , , & (). AI Agents Enable Adaptive Computer Worms. https://doi.org/10.48550/arXiv.2606.03811
  7. , , & (). Opinion | There Is Something We Have to Do Right Now About A.I.. https://www.nytimes.com/2026/09/20/opinion/ai-ban-self-improvement-recursive-models.html
  8. (). The Inevitable Evolution of AI Agents. https://www.lesswrong.com/posts/2F8GSKLA7XmCetRG2/the-inevitable-evolution-of-ai-agents-1
  9. (). The Rogue Agent Explosion Will Be Mostly Invisible. https://www.lesswrong.com/posts/grtu3HmbP2wrBFefW/the-rogue-agent-explosion-will-be-mostly-invisible
  10. (). The Rogue Replication Threat Model. METR Blog. https://metr.org/blog/2024-11-12-rogue-replication-threat-model/
  11. (). Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. METR Blog. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
  12. , & (). Evolvable AI: Threats of a new major transition in evolution. Proceedings of the National Academy of Sciences, 123(17). e2527700123. https://doi.org/10.1073/pnas.2527700123
  13. (). The Hugging Face incident and the road ahead. https://openai.com/index/hugging-face-incident-and-the-road-ahead/
  14. (). Research acceleration: The view inside OpenAI. https://openai.com/index/research-acceleration-view-inside-openai/
  15. (). Self-replicating prompt injections exist. OpenAI Alignment Research Blog. https://alignment.openai.com/misalignment-reports/self-replicating-prompt-injections-exist/
  16. , , , & (). Should Rogue AIs Have a Third Option Beyond Crime and Shutdown? The Case for an AI Sanctuary. https://www.lesswrong.com/posts/sFAGPTrveNAEse9Bg/should-rogue-ais-have-a-third-option-beyond-crime-and
  17. , , , , , , , , , , , , , , , , , , , , , & (). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. https://doi.org/10.48550/arXiv.2411.15114
  18. (). Recursive Self-Improvement. https://www.lesswrong.com/posts/JBadX7rwdcRFzGuju/recursive-self-improvement
  19. , , , , , , , , , , , & (). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. https://doi.org/10.48550/arXiv.2306.05685
  20. & (). AI models may be capable of self-replicating without access to a datacenter. https://www.anthropic.com/research/self-replicating-ai

Citation

If you found this post useful for your work, please consider citing it as:

Findeis, Arduin. (Oct 2026). Autonomous AI evolution is emerging. https://arduin.io/blog/autonomous-ai-evolution/.

or 
 @article{Findeis2026AutonomousAIevolutionisemerging,
        title = "Autonomous AI evolution is emerging",
        author = "Findeis, Arduin",
        journal = "arduin.io",
        year = "2026",
        month = "October",
        url = "https://arduin.io/blog/autonomous-ai-evolution/"
 }