如果读者有关注模型架构方面的进展，那么就会发现，比较新的线性Attention（参考《线性注意力简史：从模仿、创新到反哺》）模型都给$\boldsymbol{Q},\boldsymbol{K},...| spaces.ac.cn
Test-Time Training (TTT) models context dependencies by adapting part of the model's weights (referred to as fast weights) during inference. This fast weight, akin to recurrent states in RNNs, stores temporary memories of past tokens in the current sequence. Existing TTT methods struggled to show effectiveness in handling long-context data, due to their inefficiency on modern GPUs. The TTT layers in many of these approaches operate with extremely low FLOPs utilization (often <5%) because they...| papers.cool
Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. We propose a new class of sequence modeling layers with linear complexity and an expressive hidden state. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training ...| papers.cool
在中文圈，本站应该算是比较早关注线性Attention的了，在2020年写首篇相关博客《线性Attention的探索：Attention必须有个Softmax吗？》时，大家主要讨论的还是BERT...| spaces.ac.cn
自从DeepSeek爆火后，它所提的Attention变体MLA（Multi-head Latent Attention）也愈发受到关注。MLA通过巧妙的设计实现了MHA与MQA的自由切换，使得...| spaces.ac.cn
事实上，除了写博客内容，在这几年里，笔者是花了相当一部分时间来做科学空间的“表面功夫”，为此还专门学了一点php、css和js。虽然不敢说精益求精，但总体来说网站的浏览体验应该比前几年要好得多。...| spaces.ac.cn