Login
From:
labml.ai research blog
(Uncensored)
subscribe
Rotary Positional Embeddings with Relative distances (RoPER)
https://research.labml.ai/RoPER.html
links
backlinks
Roast topics
Find topics
Find it!
RoPER, in addition to using relative positions in the attention score calculation, adds relative positional information explicitly to value embeddings.
