Do Language Models Prefer Better Demonstrations or Easier Output Tokens?
Main Article Content
Keywords
in-context learning, demonstration selection, label bias, calibration, activation patching
Abstract
Model feedback is often used to choose in-context demonstrations, but a high probability for the correct answer token may reflect token compatibility rather than useful example content. We audit this issue with Qwen2.5-0.5B-Instruct on BoolQ using corrected inputs, frozen samples, and query-level inference. Under the original Yes/No labels, the preferred same-label demonstration improves the held-out margin over a frozen random baseline by 0.2526. Late-layer residual interventions, however, show no reliable advantage for a matched donor over an unrelated donor with the same label. Counterbalanced A/B mappings then make the raw preferred-minus-random effect follow the output token. A new frozen extension compares raw correct-token likelihood, a binary logit margin, and content-free calibration. The raw selector is harmful on average and strongly favors B. Binary-margin selection gives the largest average held-out gain but leaves a smaller bias toward A. Content-free calibration reduces the raw token gap but does not meet the preset equivalence rule. These results show that model-based demonstration preference depends on the full prompt, label vocabulary, and scoring rule. It should not be read as a direct measure of semantic example quality.
References
- [1] T. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1877-1901.
- [2] Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,” in Proc. ACL, 2022, pp. 8086-8098, doi: 10.18653/v1/2022.acl-long.556.
- [3] S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?” in Proc. EMNLP, 2022, pp. 11048-11064, doi: 10.18653/v1/2022.emnlp-main.759.
- [4] J. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for GPT-3?” in Proc. DeeLIO, 2022, pp. 100-114, doi: 10.18653/v1/2022.deelio-1.10.
- [5] O. Rubin, J. Herzig, and J. Berant, “Learning to retrieve prompts for in-context learning,” in Proc. NAACL-HLT, 2022, pp. 2655-2671, doi: 10.18653/v1/2022.naacl-main.191.
- [6] Z. Zhang, S. Lan, L. Song, J. Bian, Y. Li, and K. Ren, “Learning to select in-context demonstration preferred by large language model,” in Findings of ACL, 2025, pp. 11345-11360, doi: 10.18653/v1/2025.findings-acl.592.
- [7] T. Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot performance of language models,” in Proc. ICML, PMLR 139, 2021, pp. 12697-12706.
- [8] J. Wei et al., “Larger language models do in-context learning differently,” arXiv:2303.03846, 2023.
- [9] L. Wang, L. Li, D. Dai, D. Chen, H. Zhou, F. Meng, J. Zhou, and X. Sun, “Label words are anchors: An information flow perspective for understanding in-context learning,” in Proc. EMNLP, 2023, pp. 9840-9855, doi: 10.18653/v1/2023.emnlp-main.609.
- [10] R. Hendel, M. Geva, and A. Globerson, “In-context learning creates task vectors,” in Findings of EMNLP, 2023, pp. 9318-9333, doi: 10.18653/v1/2023.findings-emnlp.624.
- [11] E. Todd, M. Li, A. Sen Sharma, A. Mueller, B. C. Wallace, and D. Bau, “Function vectors in large language models,” in Proc. ICLR, 2024.
- [12] K. Meng, D. Bau, A. Andonian, and Y. Belinkov, “Locating and editing factual associations in GPT,” in Advances in Neural Information Processing Systems, vol. 35, 2022, pp. 17359-17372.
- [13] J. Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, S. Sakenis, J. Huang, Y. Singer, and S. Shieber, “Causal mediation analysis for interpreting neural NLP: The case of gender bias,” in Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 12388-12401.
- [14] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “BoolQ: Exploring the surprising difficulty of natural yes/no questions,” in Proc. NAACL-HLT, 2019, pp. 2924-2936, doi: 10.18653/v1/N19-1300.
- [15] Qwen Team, “Qwen2.5 technical report,” arXiv:2412.15115, 2024.
- [16] A. Holtzman, P. West, V. Shwartz, Y. Choi, and L. Zettlemoyer, “Surface form competition: Why the highest probability answer is not always right,” arXiv:2104.08315, 2021.
- [17] Y. Reif and R. Schwartz, “Beyond performance: Quantifying and mitigating label bias in LLMs,” in Proc. NAACL, 2024, arXiv:2405.02743.
- [18] Z. Yu and S. Ananiadou, “How do large language models learn in-context? Query and key matrices of in-context heads are two towers for metric learning,” in Proc. EMNLP, 2024, arXiv:2402.02872.
- [19] H. Cho, Y. Sakai, M. Kato, K. Tanaka, A. Ishii, and N. Inoue, “Token-based decision criteria are suboptimal in in-context learning,” in Proc. NAACL-HLT, 2025, pp. 5378-5401, doi: 10.18653/v1/2025.naacl-long.278.
- [20] F. Zhang and N. Nanda, “Towards best practices of activation patching in language models: Metrics and methods,” in Proc. ICLR, 2024.
- [21] M. Liu, “In-context fixation: When demonstrated labels override semantics in few-shot classification,” arXiv:2605.08295, 2026.
- [22] B. Cheng and J. Zhang, “Single-position intervention fails: Distributed output templates drive in-context learning,” arXiv:2605.04061, 2026.
Article Sidebar
This work is licensed under a Creative Commons Attribution 4.0 International License.