Transformers predict over a representation of a sequence. The same data can be written as bytes, characters, or subword tokens, and these representations may be lossless. Yet, under a fixed context window, they need not expose the same information to the model. This raises a basic question: how does the choice of representation change what a finite-context predictor can achieve?
@article{rami2026effectivecontext,title={Effective Context in Transformers: An Analysis of Fragmentation and Tokenization},author={Jafari Fesharaki, Amirmehdi and Rami, Mohammadamin and Tchamkerten, Aslan},year={2026},month=may,url={https://arxiv.org/abs/2605.13485},}