Install and run ExplainBench.

Install

The published package supports Python 3.12 (>=3.12,<3.13) and installs the explainbench command.

On macOS or Linux, create a virtual environment and install from PyPI:

1
2
3
4

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install explainbench-cli

On Windows PowerShell:

1
2
3
4

py -3.12 -m venv .venv
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install explainbench-cli

Confirm the installation:

1
2

explainbench --version
explainbench --help

To upgrade an existing installation, run python -m pip install --upgrade explainbench-cli.

Quick start

Create a UTF-8 file named submission.json:

1
2
3
4
5
6
7
8
9

{
  "submission_id": "my-run",
  "instances": [
    {
      "instance_id": "sympy__sympy-15349",
      "explanation": "The patch fixes the sign used in the rotation matrix."
    }
  ]
}

Check the file locally before making any model requests:

1

explainbench checker submission.json

A valid file prints Submission is valid and a short count summary. Invalid files list each problem and return exit code 1.

Evaluation uses a model provider and can make paid requests. For an OpenAI model, set:

1

export OPENAI_API_KEY="your-api-key"

In Windows PowerShell, use $env:OPENAI_API_KEY = "your-api-key".

Run the two intent tasks in lite mode:

1
2
3
4
5

explainbench evaluate submission.json \
  --mode lite \
  --model MODEL_ID \
  --num-generations 1 \
  --output results.json

Replace MODEL_ID with the model served by your provider. Lite mode includes its question data in the package and does not require model_patch.

Submission format

A submission contains one ID and at least one benchmark instance:

1
2
3
4
5
6
7
8
9
10

{
  "submission_id": "my-run",
  "instances": [
    {
      "instance_id": "sympy__sympy-15349",
      "explanation": "The patch fixes the sign used in the rotation matrix.",
      "model_patch": "diff --git a/... b/...\n--- a/...\n+++ b/...\n@@ -1 +1 @@\n-old\n+new"
    }
  ]
}

model_patch is optional when only intent tasks are selected. It is required for e2e.effect, local.effect, full mode, and the local-effect question builder. ExplainBench rejects unknown fields. For effect tasks, use only letters, numbers, periods, underscores, and hyphens in submission_id.

Evaluation modes

ExplainBench provides four tasks:

Lite mode selects e2e.intent and local.intent. Full mode selects all four tasks and requires effect artifacts that match submission_id:

1
2
3
4
5
6
7

question-artifacts/
├── context/
│   ├── e2e_effect__my-run.json
│   └── local_effect__my-run.json
└── ground_truths/
    ├── e2e_effect__my-run.json
    └── local_effect__my-run.json

Run full mode after both effect-file pairs are available:

1
2
3
4
5

explainbench evaluate submission.json \
  --mode full \
  --model MODEL_ID \
  --artifacts-dir question-artifacts \
  --output results.json

The local question builder creates the local_effect pair. You must provide the matching e2e_effect pair before running all four full-mode tasks.

For a custom selection, repeat --task:

1
2
3
4
5

explainbench evaluate submission.json \
  --task e2e.intent \
  --task local.intent \
  --model MODEL_ID \
  --output results.json

--mode and --task are mutually exclusive. The default is five model answers per task instance; use --num-generations to change it. --workers controls concurrent instances, while --generation-workers controls concurrent answers for one instance.

Configuration and credentials

Credentials can be stored in a local .env file:

1

OPENAI_API_KEY=your-api-key

Pass it to an evaluation with --env-file .env. Do not commit credential files.

1
2
3
4
5

explainbench evaluate submission.json \
  --mode lite \
  --model MODEL_ID \
  --env-file .env \
  --output results.json

For repeatable runs, place settings in evaluation.toml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

schema_version = 1

[selection]
mode = "lite"

[evaluator]
model = "MODEL_ID"
num_generations = 1
instance_workers = 2
generation_workers = 1
temperature = 1.0
top_p = 1.0
max_tokens = 8192
max_retries = 5

[paths]
output = "results.json"

[environment]
env_file = ".env"

Run it with:

1

explainbench evaluate submission.json --config evaluation.toml

Paths in a configuration file are relative to that file. Command-line options override configuration values. Run explainbench evaluate --help for sampling, retry, progress, environment, and concurrency options.

Resume and results

ExplainBench writes a checkpoint beside the result while an evaluation runs. For results.json, the checkpoint is results.json.checkpoint.jsonl.

Continue compatible completed work with the same command plus --resume:

1
2
3
4
5

explainbench evaluate submission.json \
  --mode lite \
  --model MODEL_ID \
  --output results.json \
  --resume

Completed work is reused only when the submission and main evaluation settings match. The checkpoint remains when any task instance fails and is removed after all selected work succeeds.

The result JSON records the submission ID, selected mode and tasks, model settings, token usage, per-task means and standard errors, completion counts, per-instance predictions and scores, and error text for failed instances.

Build local-effect questions

The local-effect builder executes code before and after the submitted patch to produce the question files required by local.effect.

Run all stages in a dedicated workspace:

1
2
3
4
5

explainbench question-builder local run submission.json \
  --workspace .explainbench/builds/my-run \
  --output question-artifacts \
  --candidate-model MODEL_ID \
  --resume

Use --candidate-env-file .env when candidate generation needs credentials from a file. The builder can make paid model requests.

Inspect progress:

1
2

explainbench question-builder local status \
  --workspace .explainbench/builds/my-run

List the pipeline stages:

1

explainbench question-builder local stages

The workspace stores source repositories, Docker data, logs, prompts, model responses, and checkpoints. Treat it as private data, keep it while you may need to resume, and never run two builders in the same workspace. Copy the published output to a durable location before deleting the workspace.

Requirements and help

The checker is fully local. Evaluation requires network access and credentials for the selected model provider.

The local-effect builder additionally requires:

• A running Docker service.
• At least 20 GB of free disk space for a first run.
• Network access to package indexes, benchmark Git repositories, Hugging Face, and the Docker image services used by SWE-bench.
• A model key for candidate generation.

The first builder run can take a long time because it may download and build Docker images.

Show help for any command:

1
2
3
4
5

explainbench --help
explainbench checker --help
explainbench evaluate --help
explainbench question-builder local --help
explainbench question-builder local run --help

For detailed configuration and troubleshooting, see the CLI usage guide. Report problems in the GitHub issue tracker, and check published releases on PyPI.