Background Lots of people experience fiddly behavior when using LLMs. For example: Unironically I found this to be very helpful when prompting LLMs. Giving them spaces and new lines pic.twitter.com/vVuxcCuDzB — anton (@abacaj) November 24, 2023 If you aren’t careful, these can be very hard to debug. This is because of the subtle ways tokenizers work that is not always easy to see by looking at the text. Example The below example demonstrates how things can get confusing and can drift betw...
