This is a guest post by Max Buckley, a software engineer at Google and fellow AI researcher1. By some twist of fate, this blog has become the chronicle of the evolution of integer tokenization. In an earlier post in February of 2023, it was discussed how older models: GPT-2 and...
