ExplainBench
Evaluate whether coding-agent explanations are informative, grounded in program behavior, and faithful to the patches they describe.
Evaluation surface
Evaluate agent explanation quality.
ExplainBench assesses the quality of natural-language patch explanations by using them as context for LLMs to answer QA tasks, so explanation quality can be compared independently from patch success.
What should the program do?
Intent questions test whether an explanation conveys the developer-intended behavior behind a bug fix.
What did the patch actually change?
Effect questions test whether the explanation matches execution evidence before and after the agent patch.
Global and local reasoning
The benchmark separates end-to-end program behavior from function-level behavior to reveal where explanations lose precision.
Benchmark pipeline
From patch explanations to comparable scores.
The workflow combines behavioral traces, generated questions, and explanation-only answering to expose when an agent sounds plausible but misses the real patch effect.
Collect traces
Gather developer tests, developer patches, agent patches, and final agent explanations.
Build questions
Derive multiple-choice questions about intended behavior and actual patch effects.
Answer from explanation
Ask a QA model to answer only from the agent explanation and benchmark context.
Score and audit
Compare answers to ground truth and surface explanation gaps for improvement.