The hiring score that repeats the same mistake
A repeatable score can still reward the wrong evidence. Test automated hiring assessments for validity before using them to reject applicants.

A hiring score can be consistent every time the system runs and still measure the wrong thing. That is the part of automated assessment I would want explained before a rejection depends on it.
Suppose a candidate has one minute to describe how they would investigate a slow service. One answer identifies a useful measurement and a plausible next step. Another fills the box with fluent generalities. If the longer answer wins, the employer needs evidence that the difference reflects the job skill it meant to assess.
That example is hypothetical. It cannot establish that hiring tools generally reward word count. The available research supports a narrower concern: the behaviour of a language-model judge depends on its model, instructions, and test conditions. A score needs validation in the assessment where someone intends to use it.
The older bias finding needs an update
The 2023 MT-Bench and Chatbot Arena paper examined language models judging other models' responses. It documented weaknesses including verbosity and position bias. It also found substantial agreement with human preferences in its benchmark setting.
That makes it a useful warning about evaluation design. It does not show that an unnamed recruitment product rejects short human answers, or that every longer answer receives an advantage. Chatbot response preference and suitability for a job are different targets.
A June 2026 preprint complicates the simple word-count story. Across its cohort of 21 judges, verbosity bias was small under one pairwise rubric. Some judges combined high repeatability with substantial position bias. Results changed across benchmarks.
This is research on judge models, not a hiring-outcome trial. Its useful lesson for an employer is methodological: running an assessment twice answers a different question from checking whether its scores support the intended hiring decision. A stable mistake remains a mistake.
Try changing the packaging without changing the evidence
I would test a proposed grader with paired answers before letting it reject applicants. Start with responses that subject specialists have independently assessed. Keep those assessments hidden from the model while it scores the answers.
Then construct controlled variants. Shorten an answer without deleting its reasoning. Add polite filler without adding evidence. Swap the order of alternatives. Replace a polished sentence with a plain one that makes the same claim. Record whether those changes alter the score enough to cross the rejection threshold.
These are diagnostic tests, not proof that the whole selection process is fair. They can reveal that a supposedly job-related score depends on incidental presentation. If concise incident reports are part of the job, rewarding unnecessary length would be especially hard to defend.
The evaluation also needs genuinely different answers. An assessment should distinguish an incorrect diagnosis from a correct one, even when both sound confident. Testing only paraphrases could produce excellent consistency while never checking whether the grader recognises competent work.
The UK government's recruitment assurance guide separates several risks in these tools, including validity, bias, and accessibility. Passing a formatting sensitivity check does not settle all three.
A rejection threshold changes the review job
A score used to organise an inbox has a different consequence from a score that removes an applicant. The employer should document where the boundary sits and inspect cases near it, including disagreements between the model and trained reviewers.
Keep the response, the scoring explanation, and the version of the assessment together. Without that record, a reviewer may be unable to reconstruct why an earlier applicant failed after a model update. Store only what the process needs, with a defined retention period and access controls.
Human review also needs time and authority. A recruiter who sees only the score cannot evaluate the answer. A reviewer who is expected to approve every recommendation has little opportunity to correct it.
The ICO's 2026 recruitment report draws on voluntary engagement with more than 30 employers between March 2025 and January 2026. It found that many were likely making solely automated recruitment decisions despite describing human involvement. This was not a representative survey or an audit of every hiring tool. It is evidence that the label attached to a workflow can differ from how decisions actually happen.
Give candidates a route to correct the record
Applicants need to know what the assessment is asking them to demonstrate, how much time they have, and where to raise an access problem or a disputed result. The employer should be able to explain which part of the submitted evidence supported its decision.
I would avoid giving candidates a universal instruction to write 120 words in a minute. That could encourage padding, disadvantage some applicants, and ignore the actual task. A concise answer should stand or fall on its evidence. If a grader cannot explain that difference in tests the employer can inspect, it is not ready to make the rejection decision.