I’ve been thinking for a while that there’s a piece missing from LLMs. There are hints that this hole might soon be filled, and it could drive the next leg up in AI capabilities. Many people have observed that LLMs, for all their abilities, seem to lack “spark”. The new reasoning models are remarkably good […]| Educating Silicon
Science fiction stories are a lot more important than Serious People will admit. Most of us are at some level aiming towards or away from things we read as teenagers. Here are a few stories that live in my head as we’re watching the birth of AI: Excession, Iain M. Banks (1996) What does a […]| Educating Silicon
As Ilya said at NeurIPS, we only have one internet. Once the fossil fuel of existing human-generated data has been consumed, further AI progress requires new sources of information. Broadly speaking, this can happen in two ways: search against a verifier, which trades compute for information, or through direct observation and interaction with the world. […]| Educating Silicon
The previous post looked at how you might invest for a scenario where AI can do most white-collar work (AGI, roughly speaking), without being broadly superhuman (ASI). However, a short transition from AGI to ASI seems plausible, even likely under certain conditions. My focus on the simpler AGI scenario is partly a case of looking […]| Educating Silicon
Recent large language models such as Llama 3 and GPT-4 are trained on gigantic amounts of text. Next generation models need 10x more. Will that be possible? To try to answer that, here’s an estimate of all the text that exists in the world. Firstly, here’s the size of some recent LLM training sets, with […]| Educating Silicon
There’s a nice blog post from last year called Go smol or go home. If you’re training a large language model (LLM), you need to choose a balance between training compute, model size, and training tokens. The Chinchilla scaling laws tell you how these trade off against each other, and the post was a nice […]| Educating Silicon
Everyone knows the one thing you shouldn’t predict about AI is the timelines. So let’s predict AI timelines! Is AI the real deal? Yes. How big a deal? A very big deal. That’s not very helpful. Maybe quantify that a bit? This will have real impact on people’s lives. The media periodically gets overexcited about […]| Educating Silicon
AI is starting to arrive. Those close to the action have known this for a while, but almost everyone has been surprised by the precise order in which things occurred. We now have remarkably capable AI artists and AI writers. They arrived out of a blue sky, displaying a flexibility and finesse that was firmly […]| Educating Silicon
The breeze whispers of a transformation, a time of great trial and tribulation for mankind. We face an adversary whose complexity is almost unimaginable. Its vast computing power makes the human brain look like a toy. Worse still, it wields against us a powerful nanotechnology, crafting autonomous machines out of thin air. Already, more than one percent of the Earth is under its sway. I’m talking of course about the mighty Amazon rainforest.| Educating Silicon
System 3 thinking | www.educatingsilicon.com
At this point it seems basically certain that AI will be a major economic transition. The only real question is how far it goes and how fast. In a previous essay I talked through four scenarios for what the coming few years might look like. In this essay I want to think through how to invest against those scenarios.| www.educatingsilicon.com