Pairwise LLM judging: where bias enters

Pairwise LLM judging: where bias enters A workflow diagram generated by Archify. EX / Where bias enters 02 / Pairwise LLM judging (main) 03 / Mitigation at each step Length bias · longer answers win · Where bias enters Length bias longer answers win Position bias · swap order, flip verdict · Where bias enters Position bias swap order, flip verdict Self-preference · favors its own output · Where bias enters Self-preference favors its own output Leniency · tends to say correct · Where bias enters Leniency tends to say correct Question · prompt from the eval set · Pairwise LLM judging (main) Question prompt from the eval set Two answers · model A and model B · Pairwise LLM judging (main) Two answers model A and model B Into the prompt · A first or B first · Pairwise LLM judging (main) Into the prompt A first or B first LLM judge · reads both, compares · Pairwise LLM judging (main) LLM judge reads both, compares Verdict · A, B or tie · Pairwise LLM judging (main) Verdict A, B or tie Win rate · aggregated over questions · Pairwise LLM judging (main) Win rate aggregated over questions Control for length · LC AlpacaEval · Mitigation at each step Control for length LC AlpacaEval Grade both orders · count only consistent wins · Mitigation at each step Grade both orders count only consistent wins Rubric, reference · reference-guided · Mitigation at each step Rubric, reference reference-guided Several judges · different model families · Mitigation at each step Several judges different model families Check against humans · κ, π, not just % · Mitigation at each step Check against humans κ, π, not just % Legend input model generating or judging grading step mitigation bias