Abstract
Sampling-based decoding strategies are widely employed for generating text from probabilistic models, yet standard ancestral sampling often results in text that is degenerate or incoherent. To alleviate this issue, various modifications to a model’s sampling distribution, such as top-p or top-k sampling, have been introduced and are now ubiquitously used in language generation systems. We propose a unified framework for understanding these techniques, which we term sampling adapters. Sampling adapters often lead to qualitatively better text, which raises the question: From a formal perspective, how are they changing the token-level distributions of language generation models? And why do these local changes lead to higher-quality text? We argue that the shift they enforce can be viewed as a trade-off between precision and recall: while the model loses its ability to produce certain strings, its precision rate on desirable text increases. While this trade-off is not reflected in standard metrics of distribution quality (such as perplexity), we find that several precision-emphasizing measures indeed indicate that sampling adapters can lead to probability distributions more aligned with the true distribution. Further, these measures correlate with higher sequence-level quality scores, specifically, Mauve. Show more
Permanent link
https://doi.org/10.3929/ethz-b-000650669Publication status
publishedExternal links
Book title
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)Pages / Article No.
Publisher
Association for Computational LinguisticsEvent
Organisational unit
09682 - Cotterell, Ryan / Cotterell, Ryan
Related publications and datasets
Is supplemented by: https://github.com/rycolab/sampling-adapters
More
Show all metadata
ETH Bibliography
yes
Altmetrics