Bye Tokens, Hello Patches

Do we really need to break text into tokens, or could we work directly with raw bytes? First, let’s think about how do LLMs currently handle text. They first chop it up into chunks called tokens using rules about common…