TL;DR Training Large Language Models (LLM) with billions of parameters is computationally intensive and involves large communication in specialized data centers. Nous Research released DeMo, showing how to reduce these communication costs by orders of magnitude, decreasing costs and enabling training with poorer connections and less expensive hardware. This
