Appendices to [AI Tools for Existential Security](/research/ai-tools-for-existential-security). Rapid AI progress is the greatest driver of existential risk in the world today. But — if handled correctly — it could also empower humanity to face these challenges.| Forethought
Exploring how the leading AI country could achieve economic dominance through superexponential growth dynamics. Analysis of trade, technological diffusion, and space resource scenarios that could enable one nation to control >99% of global output post-AGI.| Forethought
In previous work, we’ve argued that AI that can automate AI R&D could lead to a software intelligence explosion. But just how dramatic would this actually be? In this paper, we model how much AI progress we’ll see before a software intelligence explosion fizzles out. Averaged over one year, we find that AI progress could easily be 3X faster, might be 10X faster, but won’t be 30X faster - because at that speed we’d quickly hit limits on how good software can get.| Forethought
How do we compare working on reducing catastrophe with improving the quality of the future? We introduce a simple model (EV ≈ S*F) and use the 'scale, neglectedness, tractability' framework to argue that improving *Flourishing* is of comparable priority to increasing the chance of *Surviving*.| Forethought
How big is the target we need to hit to reach a mostly great future? We argue that, on most plausible views, only a narrow range of futures meet this bar, and even common-sense utopias miss out on almost all their potential.| Forethought
Even if the target is narrow, will there be forces which nonetheless hone in on near-best futures? We argue society is unlikely to converge on them by default. Trade and compromise make eutopias seem more achievable, but still we should expect ‘default’ outcomes to fall far short.| Forethought
Over sufficiently long time horizons, will the effects of actions to improve the quality of the future just ‘wash out’? Against this view, I argue a number of plausible near-term events will have persistent and predictable path-dependent effects on the value of the future.| Forethought
I suggest a number of concrete actions we can take now to make the future go better.| Forethought
Suppose we want the future to go better. What should we do? One approach is to avoid near-term catastrophes, like human extinction. This essay series explores a different, complementary, approach: improving on futures where we survive, to achieve a truly *great* future.| Forethought
Forethought argues improving future quality matters as much as survival: flourishing has greater scale.| Forethought
Once AI can automate human labour, *physical* capabilities could grow explosively. Sufficiently advanced robotics could create a feedback loop where automated robot factories build more and better robot factories which build more and better robot factories. In this piece, we examine three stages of an **industrial explosion**: AI-directed human labour, fully automated physical labour, and nanotechnology. An industrial explosion would arise in a world which already has greatly increased cognit...| Forethought
Once AI can automate human labour, *physical* capabilities could grow explosively. Sufficiently advanced robotics could create a feedback loop where automated robot factories build more and better robot factories which build more and better robot factories. In this piece, we examine three stages of an **industrial explosion**: AI-directed human labour, fully automated physical labour, and nanotechnology. An industrial explosion would arise in a world which already has greatly increased cognit...| Forethought
Building on the recent empirical work of Kwa et al. (2025), I show that within their suite of research-engineering tasks the performance of AI agents on longer-duration tasks can be explained by an extremely simple mathematical model — a constant rate of failing during each minute a human would take to do the task. This implies an exponentially declining success rate with the length of the task and that each agent could be characterised by its own half-life. This empirical regularity allows...| Forethought
It’s plausible that there will soon be digital minds that are sentient and deserving of rights. This raises several important issues that we don’t know how to deal with. It seems tractable both to make progress in understanding these issues and in implementing policies that reflect this understanding. A favorite direction is to take existing ideas for what labs could be doing and spell out enough detail to make them easy to implement.| Forethought
AI progress might enable either an AI system or a human with AI assistance to seize power. Which would be worse? In this research note, I present some initial considerations for comparing AI takeover with human takeover. I argue that AI systems will be kinder and more cooperative than humans in expectation, and that conditioning on takeover makes AI takeover more concerning, but by less than you might think. Overall, it’s plausible that human takeover would be worse than AI takeover.| Forethought
This series contains lists of projects that it could be very valuable for someone to tackle. The projects would be especially valuable if transformative AI is coming in the next 10 years or so, but they are not primarily about controlling AI or aligning AI to human intentions. Most of the projects would be valuable even if we were guaranteed to get aligned AI. Some of the projects would be especially valuable if we were inevitably going to get *mis*aligned AI.| Forethought
This article contains two sections. (1) Backup plans for misaligned AI: If we can't build aligned AI, and if we fail to coordinate well enough to avoid putting misaligned AI systems in positions of power, we might have some strong preferences about the dispositions of those misaligned AI systems. This section is about nudging those into somewhat better dispositions (in worlds where we can't align AI systems well enough to stay in control). A favorite direction is to study generalization & AI...| Forethought
This article presents project ideas relating to helping humanity get better at reaching correct and well-considered beliefs on important issues. If AI capabilities keep improving, AI could soon play a huge role in our epistemic landscape. I think we have an opportunity to affect how it’s used: increasing the probability that we get great epistemic assistance and decreasing the extent to which AI is used to persuade people of false beliefs. A couple of favorite projects are: "Create an orga...| Forethought
It’s plausible that AI will lead to explosive economic and technological growth. Our current methods of governance can barely keep up with today's technological advances. Speeding up the rate of technological growth by 30x+ would cause huge problems and could lead to rapid, destabilizing changes in power. This article is about trying to prepare the world for this. Either generating policy solutions to problems we expect to appear or addressing the meta-level problem about how we can coord...| Forethought
Automating AI R&D might lead to a software intelligence explosion, where AI improving AI algorithms leads to accelerating progress without any additional hardware. One of the strongest objections to a software intelligence explosion is that AI progress could get bottlenecked by compute: making progress requires compute-heavy experiments, and perhaps beyond a certain point it won’t be possible to accelerate any more without increasing the amount of compute available. In this post, I set out ...| Forethought
Humanity is not prepared for the AI-driven challenges we face. But the right AI tools could help us to anticipate and work together to meet these challenges — if they’re available in time. We can and should accelerate these tools. Key applications include (1) *epistemic* tools, which improve human judgement; (2) *coordination* tools, which help diverse groups work identify and work towards shared goals; (3) *risk-targeted* tools to address specific challenges. We can accelerate important ...| Forethought
Advanced AI could unlock an era of enlightened and competent government action. But without smart, active investment, we’ll squander that opportunity and barrel blindly into danger.| Forethought
The shift from scaling up the pre-training compute of AI systems to scaling up their inference compute may have profound effects on AI governance. The nature of these effects depends crucially on whether this new inference compute will primarily be used during external deployment or as part of a more complex training programme within the lab. Rapid scaling of inference-at-deployment would: lower the importance of open-weight models (and of securing the weights of closed models), reduce the im...| Forethought
Improving model performance by scaling up inference compute is the next big thing in frontier AI. But the charts being used to trumpet this new paradigm can be misleading. While they initially appear to show steady scaling and impressive performance for models like o1 and o3, they really show poor scaling (hard to distinguish from brute force) and little evidence of improvement between o1 and o3. I explain how to interpret these new charts and what evidence for strong scaling and progress wou...| Forethought
AI capabilities have improved remarkably quickly, fuelled by the explosive scale-up of resources being used to train the leading models. But if you examine the scaling laws that inspired this rush, they actually show extremely poor returns to scale. What’s going on?| Forethought
I show how a standard argument for advancing progress is extremely sensitive to how humanity’s story eventually ends. Whether advancing progress is ultimately good or bad depends crucially on whether it also advances the end of humanity. Because we know so little about the answer to this crucial question, the case for advancing progress is undermined. I suggest we must either overcome this objection through improving our understanding of these connections between progress and human extincti...| Forethought
AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments *grand challenges*. These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatical...| Forethought
Once AI fully automates AI R&D, there might be a period of fast and accelerating software progress – a software intelligence explosion. One objection to this is that it takes a long time to train AI systems from scratch. Would retraining each new generation of AI stop progress accelerating during a software intelligence explosion? If not, would it significantly delay a software intelligence explosion? This post investigates this objection. Using a theoretical analysis and some spreadsheet m...| Forethought
AI companies are increasingly using AI systems to accelerate AI research and development. Today’s AI systems help researchers write code, analyze research papers, and generate training data. Future systems could be significantly more capable – potentially automating the entire AI development cycle from formulating research questions and designing experiments to implementing, testing, and refining new AI systems. We argue that such systems could trigger a runaway feedback loop in which the...| Forethought
I describe a threat model by which AI R&D capabilities could cause harm, a specific capability threshold at which this risk becomes unacceptable, early warning signs for detecting that threshold, and the protective measures needed to continue development safely past that threshold. I recommend that labs start measuring for the warning signs today. If they observe them, they should pause AI development unless they have implemented the protective measures.| Forethought
There have been recent discussions of centralizing western AGI development, for instance through a Manhattan Project for AI. But there has been little analysis of whether centralizing would actually be a good idea. In this piece, we explore the strategic implications of having one project instead of several. We think that it’s very unclear whether centralizing would be good or bad overall. We tentatively guess that centralizing would be bad because it would increase risks from power concent...| Forethought
Today, AI progress is driven by humans, and the rate of progress is roughly constant over time. But once AI itself drives AI progress, this feedback loop could cause the rate of AI progress to accelerate, getting faster and faster over time. In this piece, we explain the conditions under which progress accelerates, and then evaluate whether these conditions hold for three feedback loops by which AI will improve AI: software, chip technology, and chip production. Setting human constraints asid...| Forethought
Once AI systems can design and build even more capable AI systems, we could see an *intelligence explosion*, where AI capabilities rapidly increase to well past human performance. The classic intelligence explosion scenario involves a feedback loop where AI improves AI software. But AI could also improve other inputs to AI development. This paper analyses three feedback loops in AI development: software, chip technology, and chip production. These could drive three types of intelligence explo...| Forethought
In simplified models of intelligence explosions, accelerating AI progress goes to infinity. But in reality, we will hit effective limits on AI intelligence. In this piece, we analyze the room for improvement in three inputs to AI development: software, chip technology and chip production. Setting human constraints aside, we argue that software could increase effective compute by ~10 orders of magnitude or more, chip technology by ~6 orders of magnitude, and chip production by ~5 orders of mag...| Forethought
Today, human efforts drive AI progress. But at some point, all AI progress will be driven by AI. This piece analyzes the transition from human-driven to AI-driven progress. Firstly, we discuss the suddenness of the transition: how many months or years does it take to transition from human to AI-driven progress? Secondly, we turn to the size of the initial speed-up: how much faster is the initial period of AI-driven progress compared to the final period of human-driven progress? We argue that ...| Forethought
If there is an international project to build artificial general intelligence (“AGI”), how should it be designed? Existing scholarship has looked to historical models for inspiration, often suggesting the Manhattan Project or CERN as the closest analogues. But AGI is a fundamentally general-purpose technology, and is likely to be used primarily for commercial purposes rather than military or scientific ones. This report presents an under-discussed alternative: Intelsat, an international o...| Forethought
Powerful AI will have a huge impact on epistemic processes in our society. How can we influence AI development and usage to make this go better? In this post, I discuss how important this area is broadly, and zoom in on ways to differentially advance and evaluate epistemic capabilities.| Forethought
The long-term future of intelligent life is currently unpredictable and undetermined. We argue that the invention of artificial general intelligence (AGI) could change this by making extreme types of lock-in technologically feasible. In particular, we argue that AGI would make it technologically feasible to (i) perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and (ii) develop AGI-based institutions that would (with high probability) competent...| Forethought
Discussions about AI progress often center on the importance of compute scaling and algorithmic improvements to achieve new capabilities, and there is evidence that AI systems are beginning to contribute to AI progress. This paper explores what might happen to the pace of progress in a scenario where AI systems could fully automate AI research. We present insights from interviews with AI researchers from leading AI companies to explore how this scenario would alter the pace of algorithmic pro...| Forethought
The development of AI that is more broadly capable than humans will create a new and serious threat: *AI-enabled coups*. An AI-enabled coup could be staged by a very small group, or just a single person, and could occur even in established democracies. Sufficiently advanced AI will introduce three novel dynamics that significantly increase coup risk. Firstly, military and government leaders could fully replace human personnel with AI systems that are *singularly loyal* to them, eliminating th...| Forethought