In previous work, we’ve argued that AI that can automate AI R&D could lead to a software intelligence explosion. But just how dramatic would this actually be? In this paper, we model how much AI progress we’ll see before a software intelligence explosion fizzles out. Averaged over one year, we find that AI progress could easily be 3X faster, might be 10X faster, but won’t be 30X faster - because at that speed we’d quickly hit limits on how good software can get.| Forethought
Once AI can automate human labour, *physical* capabilities could grow explosively. Sufficiently advanced robotics could create a feedback loop where automated robot factories build more and better robot factories which build more and better robot factories. In this piece, we examine three stages of an **industrial explosion**: AI-directed human labour, fully automated physical labour, and nanotechnology. An industrial explosion would arise in a world which already has greatly increased cognit...| Forethought
Once AI can automate human labour, *physical* capabilities could grow explosively. Sufficiently advanced robotics could create a feedback loop where automated robot factories build more and better robot factories which build more and better robot factories. In this piece, we examine three stages of an **industrial explosion**: AI-directed human labour, fully automated physical labour, and nanotechnology. An industrial explosion would arise in a world which already has greatly increased cognit...| Forethought
Building on the recent empirical work of Kwa et al. (2025), I show that within their suite of research-engineering tasks the performance of AI agents on longer-duration tasks can be explained by an extremely simple mathematical model — a constant rate of failing during each minute a human would take to do the task. This implies an exponentially declining success rate with the length of the task and that each agent could be characterised by its own half-life. This empirical regularity allows...| Forethought
Automating AI R&D might lead to a software intelligence explosion, where AI improving AI algorithms leads to accelerating progress without any additional hardware. One of the strongest objections to a software intelligence explosion is that AI progress could get bottlenecked by compute: making progress requires compute-heavy experiments, and perhaps beyond a certain point it won’t be possible to accelerate any more without increasing the amount of compute available. In this post, I set out ...| Forethought
Improving model performance by scaling up inference compute is the next big thing in frontier AI. But the charts being used to trumpet this new paradigm can be misleading. While they initially appear to show steady scaling and impressive performance for models like o1 and o3, they really show poor scaling (hard to distinguish from brute force) and little evidence of improvement between o1 and o3. I explain how to interpret these new charts and what evidence for strong scaling and progress wou...| Forethought
AI capabilities have improved remarkably quickly, fuelled by the explosive scale-up of resources being used to train the leading models. But if you examine the scaling laws that inspired this rush, they actually show extremely poor returns to scale. What’s going on?| Forethought
AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments *grand challenges*. These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatical...| Forethought
Once AI fully automates AI R&D, there might be a period of fast and accelerating software progress – a software intelligence explosion. One objection to this is that it takes a long time to train AI systems from scratch. Would retraining each new generation of AI stop progress accelerating during a software intelligence explosion? If not, would it significantly delay a software intelligence explosion? This post investigates this objection. Using a theoretical analysis and some spreadsheet m...| Forethought
AI companies are increasingly using AI systems to accelerate AI research and development. Today’s AI systems help researchers write code, analyze research papers, and generate training data. Future systems could be significantly more capable – potentially automating the entire AI development cycle from formulating research questions and designing experiments to implementing, testing, and refining new AI systems. We argue that such systems could trigger a runaway feedback loop in which the...| Forethought
Today, AI progress is driven by humans, and the rate of progress is roughly constant over time. But once AI itself drives AI progress, this feedback loop could cause the rate of AI progress to accelerate, getting faster and faster over time. In this piece, we explain the conditions under which progress accelerates, and then evaluate whether these conditions hold for three feedback loops by which AI will improve AI: software, chip technology, and chip production. Setting human constraints asid...| Forethought
Once AI systems can design and build even more capable AI systems, we could see an *intelligence explosion*, where AI capabilities rapidly increase to well past human performance. The classic intelligence explosion scenario involves a feedback loop where AI improves AI software. But AI could also improve other inputs to AI development. This paper analyses three feedback loops in AI development: software, chip technology, and chip production. These could drive three types of intelligence explo...| Forethought
In simplified models of intelligence explosions, accelerating AI progress goes to infinity. But in reality, we will hit effective limits on AI intelligence. In this piece, we analyze the room for improvement in three inputs to AI development: software, chip technology and chip production. Setting human constraints aside, we argue that software could increase effective compute by ~10 orders of magnitude or more, chip technology by ~6 orders of magnitude, and chip production by ~5 orders of mag...| Forethought
Today, human efforts drive AI progress. But at some point, all AI progress will be driven by AI. This piece analyzes the transition from human-driven to AI-driven progress. Firstly, we discuss the suddenness of the transition: how many months or years does it take to transition from human to AI-driven progress? Secondly, we turn to the size of the initial speed-up: how much faster is the initial period of AI-driven progress compared to the final period of human-driven progress? We argue that ...| Forethought
Discussions about AI progress often center on the importance of compute scaling and algorithmic improvements to achieve new capabilities, and there is evidence that AI systems are beginning to contribute to AI progress. This paper explores what might happen to the pace of progress in a scenario where AI systems could fully automate AI research. We present insights from interviews with AI researchers from leading AI companies to explore how this scenario would alter the pace of algorithmic pro...| Forethought