Tokenization_neurips_2026

Our paper “Effective Context in Transformers: An Analysis of Fragmentation and Tokenization” has been accepted to Neurips 2026! The paper will have a revised title: “Representation in Finite-Context Prediction: An Analysis of Tokenization and Fragmentation”.