Identical prompts, settings and datasets are supposed to guarantee fair comparison — yet concealed temporal cues in system prompts can slip past that controlled setup.
Somewhere in the plumbing of a benchmark run sits a variable nobody controlled: a date, tucked into a system prompt, quietly steering results.
Benchmarks are the load-bearing structure of the large language model market. Without them, consumers and businesses have no reliable way to judge what progress a model has actually made between versions, or how it stacks up against competitors. Procurement choices, upgrade decisions and competitive positioning all lean on scores that are presumed to be comparable — which is precisely why cracks in evaluation methodology matter well beyond the research community.
The core difficulty is baked into the technology. Large language models are non-deterministic: fed the same input, they will not always produce consistent outputs. Identical benchmark datasets and settings can therefore yield different scores across runs, and a degree of variance has long been accepted as the cost of measuring these systems at all.
The standard defense is exhaustive control — identical prompts, identical model settings, identical benchmark datasets — so that any remaining gap in scores can be attributed to the model itself. But as Unite AI's reporting highlights, seemingly minor differences in the execution environment can persist even under that discipline. Secret dates embedded in system prompts are the case in point: a variable that survives the standard checklist because it hides inside one of the very artifacts researchers believe they have pinned down.
The result is a benchmark whose numbers look clean and are not.
For AI-compute buyers and investors, the practical consequence is a recalibration of trust. Version-over-version progress claims and head-to-head competitor comparisons are only as sound as the evaluation setup behind them, and that setup — it now turns out — can be quietly undermined by something as banal as a date string in a prompt. Until evaluation pipelines account for hidden temporal cues, benchmark numbers should be treated as directional signals rather than ground truth.
萬安算交所 AIXX · 算力硬件实名会员制交易平台 · 进入货源大厅