Algorithmic Progress in Language Models | Epoch AI
Progress in pretrained language model performance outpaces expectations, occurring at a pace equivalent to doubling computational power every 5 to 14 months.| Epoch AI
Progress in pretrained language model performance outpaces expectations, occurring at a pace equivalent to doubling computational power every 5 to 14 months.| Epoch AI
The Electric Power Research Institute (EPRI) conducts research, development, and demonstration projects for the benefit of the public in the United States and internationally. As an independent, nonprofit organization for public interest energy and environmental research, we focus on electricity generation, delivery, and use in collaboration with the electricity sector, its stakeholders and others to enhance the quality of life by making electric power safe, reliable, affordable, and environm...| www.epri.com
While scaling compute is key to improving LLMs, post-training enhancements can offer gains equivalent to 5-20x more compute at less than 1% of the cost.| Epoch AI
Data movement bottlenecks limit LLM scaling beyond 2e28 FLOP, with a “latency wall” at 2e31 FLOP. We may hit these in ~3 years. Aggressive batch size scaling could potentially overcome these limits.| Epoch AI
If trends continue, language models will fully utilize the stock of human-generated public text between 2026 and 2032.| Epoch AI
We characterize techniques that induce a tradeoff between spending resources on training and inference, outlining their implications for AI governance.| Epoch AI
Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting an...| arXiv.org
The most extraordinary techno-capital acceleration has been set in motion. As AI revenue grows rapidly, many trillions of dollars will go into GPU, datacenter, and power buildout before the end of the decade. The industrial mobilization, including growing US electricity production by 10s of percent, will be intense. You see, I told you it couldn’t be| SITUATIONAL AWARENESS
Projections in the internal document suggest that Microsoft plans to triple the number of GPUs it has in 2024.| Business Insider
The data center, Cumulus Data Assets, sits on a 1,200-acre campus in Pennsylvania and is directly powered by the adjacent Susquehanna Steam Electric Station, which generates 2.5 gigawatts of power.| www.ans.org
On average, a ChatGPT query needs nearly 10 times as much electricity to process as a Google search. In that difference lies a coming sea change in how the US, Europe, and the world at large will consume power — and how much that will cost. | www.goldmansachs.com
In late 2017 we introduced AlphaZero, a single system that taught itself from scratch how to master the games of chess, shogi (Japanese chess), and Go, beating a world-champion program in each...| Google DeepMind
Breakthrough models AlphaProof and AlphaGeometry 2 solve advanced reasoning problems in mathematics| Google DeepMind
Bringing open intelligence to all, our latest models expand context length, add support across eight languages, and include Meta Llama 3.1 405B— the...| ai.meta.com
Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extr…| Engineering at Meta
Triple the output, triple the revenue?| Tom's Hardware
The overwhelming priority of energy policy must be making it easier to build things| Institute for Progress
YouTube is one of the largest, most important communication platforms in the world, but while there is a great deal of research about the site, many of its fundamental characteristics remain unknown. To better understand YouTube as a whole, we created a random sample of videos using a new method. Through a description of the sample’s metadata, we provide answers to many essential questions about, for example, the distribution of views, comments, likes, subscribers, and categories. Our metho...| journalqd.org
President Biden’s Actions to Tackle the Climate Crisis President Biden campaigned on a bold vision of tackling the climate crisis with the urgency that science demands, by building a clean energy economy that benefits all Americans—with lower costs for families, good-paying jobs for workers, and healthier air and cleaner water for communities. As part of…| The White House
The scientific consensus is clear. The world confronts an urgent carbon problem. The carbon in our atmosphere has created a blanket of gas that traps heat and is changing the world’s climate. Already, the planet’s temperature has risen by 1 degree centigrade. If we don’t curb emissions, and temperatures continue to climb, science tells us...| The Official Microsoft Blog
Types and amounts of electricity use in U.S. homes.| www.eia.gov
546.7 million people listen to podcasts! Learn how many podcasters are there with our detailed Podcast statistics.| DemandSage
NVIDIA Collective Communications Library (NCCL)| NVIDIA Developer
