explanation re: AI-adjacent@kitten Instead of operating on UTF-8 or something standardized, language models tend to use a special compressed encoding where one input value could represent multiple characters. So common character sequences like 'qu', 'ea', ' the', 'it.' might be represented by a single token while 'ยท' might be multiple tokens (like it is multiple UTF-8 bytes).
Since LLMs ingest and extrude a text a token at a time, the number of tokens in a text corresponds to how much computation it took to process.
This is also the reason why LLMs will happily bullshit that there are no Rs in strawberry, since they do not get presented "strawberry" as a sequence of characters: maybe 'str' or 'rr' or 'ry' are represented as a single token.
1❤️1
