Benchmarks¶
CodeNib serves repository context to coding agents, so it is evaluated on three questions of increasing scope: does a method return the right code, does an agent read the right code while it works, and does the task then get solved? This section separates those layers, states how far each method and benchmark has been checked, and links every published number to its page.
Three layers¶
A layer is a property of the measurement, not of the method. An agent can be scored at L1 on its final answer and at L2 on its whole trajectory. Each layer is a separate claim: a strong localization score does not show that an agent read efficiently, and neither shows that the task was solved.
| Layer | What is scored | Claim it supports | Benchmarks |
|---|---|---|---|
| L1 Localization | The method's final ranked files, symbols, code blocks, or line regions, against locations derived from the golden patch or released by the benchmark | The method finds the code the task depends on | CodeNib Base, CodeNib Synthesis, SWE-bench Lite, Loc-Bench v1, SWE-Explore, Agent Retrieval Bench |
| L2 Agent context | Every source span the agent viewed during its trajectory, and the context it finally kept, against annotated gold context | The agent reads the right code, and how much else it reads | ContextBench |
| L3 Resolution | Whether the patch produced for the task passes the task's tests | The task was solved | ContextBench Pass@1, DeepSWE; see L3 Resolution |
Every method receives the task text and the task's source snapshot: a clean checkout at the base commit, or, for Agent Retrieval Bench, the released corpus chunks. Ground truth reaches the scorer only after predictions are written, and every published result states how failed cases are counted. Harnesses and metrics defines every metric and lists the command that produces it.
Evidence levels¶
Each method and benchmark is labelled with the highest level it has reached. Every level is a separate claim. A pinned contract, for example, says nothing about quality.
| Level | Means | Does not mean |
|---|---|---|
| Tested | CodeNib's test suite covers the method or loader | That a benchmark run exists |
| Runnable | A harness in this repository runs the method on the benchmark and scores it | That a result is published |
| Pinned | Checked against a fixed upstream revision: vendored prompts, tool signatures, or exact agreement with the official scorer | Task quality, or compatibility with another upstream revision |
| Measured | A complete result on a fixed, published case set, presented as a comparison of methods, with failures counted | That the result holds for other repositories, models, or languages, or at another layer |
A Measured cell is always read with its benchmark's layer: Measured at L1 or L2 is not a resolution result. The same levels appear in the context methods matrix.
Benchmarks¶
Each benchmark name links to its card: origin, ground truth, granularity, size, pinned revisions, scorer, and setup. Datasets also covers SWE-bench Verified, SWE-bench Multilingual, SWE-bench Pro, and local JSON inputs, which are used for collection, joins, or custom runs rather than as scored benchmarks.
| Benchmark | Layer | Task | Languages | Entry point | Evidence |
|---|---|---|---|---|---|
| CodeNib Base | L1 Localization | Real issues; find the code the golden patch changed | C/C++, Go, Python, Rust, TypeScript/JavaScript | CodeNibBaseDataset |
Measured |
| CodeNib Synthesis | L1 Localization | Synthetic behavioral, hint, reasoning, and traversal queries | C/C++, Go, Python, Rust, TypeScript/JavaScript | CodeNibSynthesisDataset |
Runnable |
| SWE-bench Lite | L1 Localization | Real issues; find the code the golden patch changed | Python | SwebenchDataset |
Runnable; dataset revision and a 50-case selection are pinned for paired policy runs |
| Loc-Bench v1 | L1 Localization | Real issues, including features, performance, and security; find the code the golden patch changed | Python | LocbenchDataset |
Runnable |
| SWE-Explore | L1 Localization | Rank the source regions an issue needs | Seven languages in the fixed 20-case set | codenib-swe-explore-benchmark |
Pinned; scorer agrees with the official evaluator |
| Agent Retrieval Bench | L1 Localization | Four static tasks; code→test, comment→context, trace→code, edit→ripple | Mixed; multi-language repositories | codenib-arb |
Measured; six methods on all 345 samples |
| ContextBench | L2 Agent context | Real issues; an agent explores, and everything it reads is scored | C, C++, Go, Java, JavaScript, Python, Rust, TypeScript | codenib-contextbench |
Measured; 500 canonical tasks with a research route; leaderboard review pending |
| ContextBench Pass@1 | L3 Resolution | The same tasks; a solver writes a patch, and native tests judge it | C, C++, Go, Java, JavaScript, Python, Rust, TypeScript | scripts/benchmark_context_solver.py, scripts/contextbench_verified.py |
Measured; 388/500 resolved with a research route; leaderboard review pending |
| DeepSWE | L3 Resolution | Solve the task in a container; an external harness runs the solver | Set by the external harness | codenib.eval.benchmarks.deepswe, python -m scripts.guardian.deepswe.results |
Tested; artifact analysis only; no published result |
Method × benchmark matrix¶
Rows follow the three paradigms in context methods: one-shot retrieval (A), fixed LLM workflows (B), and agentic tool loops (C). A cell shows the highest level reached for that pair; "—" means no harness in this repository accepts that pair today. Research scripts count as harnesses, and the cell notes name them. Numbered notes follow each table.
Localization (L1)¶
CodeNib methods¶
| Method | Base | Synthesis | SWE-bench Lite | Loc-Bench | SWE-Explore | ARB |
|---|---|---|---|---|---|---|
| A · BM25 | Runnable1 | Runnable2 | Runnable | Runnable | Runnable3 | Measured4 |
| A · Dense | Runnable5 | Runnable | Runnable | Runnable | Runnable | Measured6 |
| A · Hybrid (RRF) | Runnable7 | — | Runnable | Runnable | Runnable | Measured8 |
| A · Graph expansion | Runnable | — | Runnable | Runnable | Runnable | Measured9 |
| A · Retrieve → rerank | Measured10 | — | Runnable | Runnable | Runnable | Runnable11 |
A · Planner auto |
Runnable | — | Runnable | Runnable | Runnable | — |
| B · Model-planned grep → Jev | Measured | — | — | — | — | — |
| C · CodeNib agent | Runnable | Runnable | Runnable | Runnable | — | — |
- BM25 × Base. Scored only as the first stage of BM25 → Jev.
- BM25 × Synthesis. BM25 and Dense run through
examples/eval_synthesized_queries.pyon rows exported to JSON, scored on file and symbol names only. - BM25 × SWE-Explore. Control scores on the 20 validation cases; not a quality claim.
- BM25 × ARB. All 345 samples; trails ARB's own BM25 because CodeNib's tokenizer does not split camelCase identifiers.
- Dense × Base. Scored only as the first stage of dense → Qwen3-Reranker-4B.
- Dense × ARB. All 345 samples; Qwen3-Embedding-0.6B with no instruction.
- Hybrid (RRF) × Base. Scored only as the first stage of hybrid → Jev and hybrid → Qwen3-Reranker-4B.
- Hybrid (RRF) × ARB. All 345 samples; file-level RRF of the BM25 and dense rankings.
- Graph expansion × ARB. All 345 samples, full-source condition; a graph for 190 of 218 snapshots; lower Recall@20 than BM25 on the samples with a graph.
- Retrieve → rerank × Base. Measured through the frozen-candidate research scripts
benchmark_jev_base.pyandbenchmark_jev_dense.py, not throughRetrieveRerankPipeline. - Retrieve → rerank × ARB.
--method rerankover a completed released-corpus run.
The embedding-model and reranker sweeps on CodeNib Base have complete retained artifacts. They are listed as evidence for the validated models, but their scores are not published on this site, so those cells stay at Runnable.
Reproduced agents and controls¶
These rows run published localization agents, and agents with no CodeNib tool at all, through the same harness, labels, and scorer as CodeNib's methods. Reproduced agents explains what each agent reproduces, how closely it follows upstream, and whether its own backend can be swapped for CodeNib's.
| Method | Base | SWE-bench Lite | Loc-Bench |
|---|---|---|---|
| B · Agentless v1.5.0 | Runnable1 | Runnable | Runnable |
| C · LocAgent | Runnable | Pinned2 | Runnable |
| C · CoSIL | Runnable | Runnable | Runnable |
| C · OrcaLoca SearchAgent | Runnable | Pinned | Runnable |
| C · Claude / Codex without CodeNib | Runnable | Runnable | Runnable |
- Agentless v1.5.0 × Base. Upstream's Python-only file filter limits it to the Python issues.
- LocAgent × SWE-bench Lite. One complete run on a fixed 50-case selection, with the CodeNib backend only and no native comparison arm.
Runnable means the harness accepts the pair, not that the agent was designed for it. The reproduced agents follow upstream policies built for Python repositories.
Agent context and resolution (L2, L3)¶
Only methods with at least one L2 or L3 harness appear here.
| Method | ContextBench (L2) | ContextBench Pass@1 (L3) |
|---|---|---|
| A · BM25 | Runnable1 | — |
| A · Dense | Runnable | — |
| A · Graph expansion | Runnable | — |
| C · CodeNib agent | Measured2 | — |
| C · Claude Agent SDK with CodeNib source tools | Measured3 | Measured4 |
- BM25 × ContextBench. Through the research harness
scripts/benchmark_graph_decisions.py, which also runs dense and graph-expansion arms. Its published 100-task arms rerank with a different decision model, not Jev. - CodeNib agent × ContextBench. Historical 24-task development subset, with source tools only (
read,grep,glob) and no index view.codenib-contextbenchruns the sameAgentRunnerwith its own prompt, budgets, and model, plus thebm25_searchskill, and does not reproduce these numbers. - Claude Agent SDK with CodeNib source tools × ContextBench. All 500 canonical tasks, for a native Opus control and for the complementary Haiku search-pair research arm. The population was already exposed during development.
- Claude Agent SDK with CodeNib source tools × ContextBench Pass@1. 388/500 resolved when the search-pair arm's context feeds an Opus mini SWE-agent repair loop. A leaderboard submission awaits maintainer review.
Research arms¶
Experiments often try configurations that are not methods anyone should choose: a different search budget, a post-processing stage, a router between two workers. Those research arms are listed on their benchmark's results page, each with its components and outcome, and do not appear in this matrix. An arm moves into the matrix only when it becomes a supported CodeNib surface. On ContextBench, for example, the complementary Haiku search pair, the worker router, evidence review, and function completion are all arms; the matrix credits their results to the method they run on and says so in the cell note.
L3 Resolution¶
Resolution is measured by running a solver on the task and grading its patch with the task's own tests. Neither path below is part of the installed package.
ContextBench Pass@1¶
ContextBench's leaderboard requires Pass@1 next to its retrieval metrics. The
codenib-contextbench runner measures context retrieval only, so its results
stay at L2. Resolution runs use research scripts: benchmark_context_solver.py
runs a pinned mini SWE-agent on the retrieved context, and a separate grader
per source (contextbench_verified.py, contextbench_pro.py,
contextbench_polybench.py, contextbench_multiswe.py) tests the patch in the
task's native environment without any model call. Reference patches and tests
enter only the grader.
| Benchmark card | |
|---|---|
| Layer | L3 Resolution: whether the patch produced for the task passes the task's tests |
| Origin | External |
| Task | The same tasks; a solver writes a patch, and native tests judge it |
| Ground truth | Each task's native tests |
| Granularity | Resolved or unresolved per task |
| Languages | C, C++, Go, Java, JavaScript, Python, Rust, TypeScript |
| Size | 500 tasks in the verified configuration |
| Pinned revisions | Same dataset; pinned per-source images, parsers, and grading scripts |
| Scorer | Native graders for the Verified, Pro, Poly, and Multi sources |
| Setup | Pinned task images, a mini SWE-agent checkout, and Docker |
| CodeNib entry point | scripts/benchmark_context_solver.py, scripts/contextbench_verified.py |
| Evidence | Measured; 388/500 resolved with a research route; leaderboard review pending |
DeepSWE¶
An external harness runs the solver in task containers. CodeNib packages only
the analysis of its trial artifacts, and keeps the harness adapter, launchers,
and dashboard under scripts/guardian/deepswe/. No DeepSWE result is
published on this site.
| Benchmark card | |
|---|---|
| Layer | L3 Resolution: whether the patch produced for the task passes the task's tests |
| Origin | External |
| Task | Solve the task in a container; an external harness runs the solver |
| Ground truth | The task's tests |
| Granularity | Pass or fail per trial |
| Languages | Set by the external harness |
| Size | Set by the external harness |
| Pinned revisions | Set by the external harness |
| Scorer | External harness; CodeNib only aggregates its trial artifacts |
| Setup | The external DeepSWE harness and task containers |
| CodeNib entry point | codenib.eval.benchmarks.deepswe, python -m scripts.guardian.deepswe.results |
| Evidence | Tested; artifact analysis only; no published result |
Published results¶
| Result | Layer | Benchmark | Headline | Scope |
|---|---|---|---|---|
| Model-planned grep → Jev | L1 | CodeNib Base, revision 4eb84e2e |
71.40% code-block Recall@5, against 58.62% for the same grep candidates in their original order | Research run with frozen search plans; paired 95% interval +6.44 to +19.67 points |
| Same split, other routes | L1 | CodeNib Base | BM25 → Jev 59.35%; dense → Qwen3-Reranker-4B 63.40%; hybrid → Qwen3-Reranker-4B 65.03%; hybrid → Jev 67.92% | Code-block Recall@5 for each route |
| Fresh product run | L1 | CodeNib Base | 48.37% → 65.57% after Jev (+17.20 points) | All 100 cases attempted; one failed and scored zero |
| SWE-Explore validation | L1 | SWE-Explore, 20 fixed cases | 1,020/1,020 real-output metric values equal to the official evaluator's | Scorer agreement; the BM25 control is not a quality claim |
| ContextBench search | L2 | ContextBench verified, 500 canonical tasks | Complementary Haiku search pair: 77.07% line Recall and 28.16% F1, against 72.83% and 33.95% for a native Opus control; Recall +4.24 points, paired 95% interval +1.59 to +6.41 | Research route on an already-exposed population; different models and billing on each side |
| ContextBench resolution | L3 | ContextBench verified, 500 canonical tasks | 388/500 resolved (77.60%) with the search pair's context and an Opus mini SWE-agent repair loop, at $338.88 recorded search-plus-repair model usage | First attempts only, failures retained; the leaderboard submission awaits maintainer review |
The multi-view system design is described in the CodeNib paper.
Prepare inputs¶
-
Collect
SWE-bench instances
Sample representative tasks across repositories and languages without losing their source metadata.
-
Synthesize
Query datasets
Generate traversal, behavioral, and multi-step queries with explicit curation and verification stages.
-
Prepare
Ground-truth locations
Extract expected files, symbols, and one-based source ranges from benchmark patches.
-
Package
Artifact bundles
Preserve manifests, metrics, diagnostics, and provenance as one verifiable evaluation output.