Roast topics
Find topics
Roast it!
Roast topics
Find topics
Find it!
Login
From:
Brendan Long
(Uncensored)
subscribe
Shorter Tokens Are More Likely
https://www.brendanlong.com/shorter-tokens-are-more-likely.html
links
backlinks
Tagged with:
misc
ai
I was thinking about LLM tokenization (as one does) and had a thought: We select the next output token for an LLM based on its likelihood, but (some) shorter tokens are more likely. Why? Longer tokens can only complete …
