AI
LLM aggregation accuracy can change dramatically when identical inputs are reordered
A study of 15 language models across five datasets finds that answers to multi-instance classification tasks can vary sharply when the same inputs are presented in different orders. Accuracy differed by roughly 90 percentage points between the best and worst arrangements, with models generally performing better when majority-label examples appeared first. Analysis of reasoning traces and attribution scores suggests disproportionate attention to some positions. The result is a practical warning for systems that summarize or classify collections of independent inputs.
Editorial summary · Publisher article
What matters
- The evaluation used seven ordering patterns while keeping each instance set fixed.
- Models were generally more accurate when majority-label examples appeared at the beginning.
- Reasoning traces emphasized early instances, while attribution was often higher at the beginning and end.
- The authors recommend evaluating multi-instance systems across varied input arrangements.
Context & caveats
- The supplied evidence is an abstract, so implementation details and broader external validation are unavailable.
Why this made the edition
A substantive empirical study identifies a major reliability problem in multi-instance LLM aggregation and tests it across multiple models, datasets and ordering patterns.
Source material