Agentic Benchmarks Explained: Measuring AI That Actually Works
From smart answers to verified outcomes
For years, AI evaluation asked a narrow question:
Can the model produce the right answer?
That question still matters. Models are tested on mathematics, coding, logic, retrieval, and reading comprehension. But an answer is only one small part of most professional work.
An engineer does not solve a production incident by writing a plausible paragraph. They work through a sequence such as:
Read the ticket and operating constraints.
Inspect the repository and recent changes.
Reproduce the failure.
Collect logs and environment details.
Form and test a root-cause hypothesis.
Search for related call paths.
Run a narrow diagnostic.
Make a targeted change.
Rerun the failing test.
Troubleshoot any new errors.
Revise the patch when necessary.
Run regression and edge-case checks.
Review the final diff for unintended changes.
Verify the repaired state.
Leave behind evidence that the problem is fixed.
A researcher does not finish when a search result appears. They work through a sequence such as:
Translate the question into specific claims that can be checked.
Identify the strongest available sources, prioritizing primary documents, official datasets, technical reports, and original research.
Record the publication date, scope, authorship, and provenance of each source.
Extract the relevant evidence instead of relying on a headline or summary.
Compare independent sources and check whether they measure the same thing.
Resolve contradictions by checking definitions, versions, dates, samples, methods, and evaluation conditions.
Separate observed facts from interpretations, estimates, and recommendations.
Preserve citations next to the claims they support.
Track missing evidence, unresolved disagreement, and possible bias or contamination.
Explain what is known, what is likely, and what remains uncertain.
Produce a conclusion that is proportional to the quality and limits of the evidence.
An operations agent does not earn trust by sounding confident. It earns trust by making only permitted changes, recording what it did, and stopping when approval is required.
This is the territory of agentic benchmarks. They evaluate an AI system's ability to pursue a goal through a sequence of observations, decisions, tool calls, and checks. The unit of measurement is no longer just the response. It is the completed, verifiable task.
Agentic evaluation measures whether a system can turn intent into a correct outcome under realistic constraints, while respecting authorization, enforcing guardrails, preserving evidence, and avoiding destructive or irreversible actions.
That shift changes how we compare models, how we design agent runtimes, and how enterprises decide whether an AI system is ready to touch real work.
The current agentic benchmark landscape
As of September 2026, there is no single authoritative leaderboard for "agentic intelligence." The field is a collection of benchmark families, each testing a different kind of work. The official source for a benchmark is usually its paper, task repository, evaluation code, documentation, or maintainer leaderboard. A third-party chart is useful for discovery, but it is not the benchmark itself.
The most useful way to read the landscape is to ask three questions:
What does the benchmark mean? What job is the agent actually performing?
What is the official source? Can the task definitions, evaluator, and environment be inspected?
How trustworthy is the score? Is the result reproducible, independently checked, and comparable to other results?
A practical map of current benchmarks
| Benchmark family | What it means in plain language | Official source | What a passing result usually proves |
|---|---|---|---|
| SWE-bench | Fix real GitHub issues in real software repositories | SWE-bench repository and official site | The submitted patch passes the benchmark's tests for a specific issue and repository state |
| SWE-bench Verified | A human-validated subset of software-engineering tasks | OpenAI SWE-bench Verified announcement and SWE-bench repository | The system solved tasks selected and reviewed for clearer problem statements and test validity |
| SWE-bench Pro | More difficult, longer, and more enterprise-like software tasks | Scale AI SWE-bench Pro | The system completed a harder curated engineering task under that benchmark's specific harness and evaluator |
| Terminal-Bench | Complete practical jobs through a terminal in a controlled environment | Terminal-Bench repository and documentation | The final container or filesystem state satisfies the task evaluator |
| BrowserGym | Use a browser to navigate, search, and interact with web applications | BrowserGym repository | The agent reached the required web state in one of the supported environments |
| WebArena | Complete realistic multi-step tasks across self-hosted websites | WebArena repository | The agent completed a task in a reproducible web environment rather than on an uncontrolled live website |
| OSWorld | Operate desktop applications and an operating system from visual instructions | OSWorld repository | The desktop state or task-specific evaluator confirms that the requested operation was completed |
| BrowseComp | Find difficult-to-locate information on the open web | OpenAI BrowseComp paper and release | The answer matches the target information under the benchmark's browsing and grading setup |
| Mind2Web | Generalize web interaction from demonstrations and website tasks | Mind2Web project | The system selects useful web actions across websites and task types represented by the dataset |
| tau-bench | Use APIs while following user and business policy constraints | tau-bench repository | The agent completed the conversation or transaction without violating the simulated policy |
| GAIA | Solve general assistant tasks that combine reasoning, tools, browsing, and multimodal inputs | GAIA benchmark | The final answer satisfies a task-specific reference answer or grader |
| AgentBench | Compare agents across multiple environments rather than one task family | AgentBench repository | The system performs across the selected environments, with each environment having its own evaluator |
| METR time-horizon evaluations | Estimate how long a task an AI system can complete reliably | METR research | The system reaches a measured reliability level on tasks of a given approximate duration and complexity |
These benchmarks are not interchangeable. SWE-bench is primarily about repository-level software maintenance. Terminal-Bench is about operating through a shell and reaching a verified environment state. BrowserGym, WebArena, and OSWorld test interaction with interfaces. BrowseComp and GAIA emphasize research and general assistance. tau-bench emphasizes policy-constrained tool use. METR asks a different question again: how task difficulty and reliable completion change as the work horizon grows.
What models and harnesses are scoring right now?
The table below is a dated snapshot of official pages retrieved on September 12, 2026. It is intentionally not presented as one combined leaderboard. Each benchmark uses a different task set, evaluator, budget, and meaning of success. The harness column is essential: changing from mini-SWE-agent to a coding product, or from Codex to Claude Code, changes the system being measured.
| Benchmark and version | Model | Agent harness | Reported result | Cost or uncertainty shown by source |
|---|---|---|---|---|
| SWE-bench Verified, bash-only, 500 tasks | Claude 4.5 Opus (high) | mini-SWE-agent 2.0.0 | 76.80% resolved | $0.75 per task; official SWE-bench entry dated 2026-02-17 |
| SWE-bench Verified, bash-only, 500 tasks | Gemini 3 Flash (high) | mini-SWE-agent 2.0.0 | 75.80% resolved | $0.36 per task; official SWE-bench entry dated 2026-02-17 |
| SWE-bench Verified, bash-only, 500 tasks | MiniMax M2.5 (high) | mini-SWE-agent 2.0.0 | 75.80% resolved | $0.07 per task; open-weights entry dated 2026-02-17 |
| SWE-bench Verified, bash-only, 500 tasks | Claude 4.6 Opus | mini-SWE-agent 2.0.0 | 75.60% resolved | $0.55 per task; official SWE-bench entry dated 2026-02-17 |
| SWE-bench Verified, bash-only, 500 tasks | GPT 5.2 (high) | mini-SWE-agent 2.0.0 | 72.80% resolved | $0.47 per task; official SWE-bench entry dated 2026-02-17 |
| SWE-bench Verified, bash-only, 500 tasks | Kimi K2.5 (high) | mini-SWE-agent 2.0.0 | 70.80% resolved | $0.15 per task; open-weights entry dated 2026-02-17 |
| Terminal-Bench 4.0, resolution rate | GPT-6 Astra (max) | Codex | 58.2% +/- 2.8% | $3.3k reported run cost; source date 2026-09-03 |
| Terminal-Bench 4.0, resolution rate | Fable 5.1 (max) | Claude Code | 57.9% +/- 3.8% | $6.2k reported run cost; source date 2026-09-01 |
| Terminal-Bench 4.0, resolution rate | Opus 5 (max) | Claude Code | 51.8% +/- 3.4% | $6.0k reported run cost; source date 2026-07-24 |
| Terminal-Bench 4.0, resolution rate | GLM-5.3 (max) | Claude Code | 41.8% +/- 3.2% | $2.7k reported run cost; source date 2026-08-14 |
| Terminal-Bench 4.0, resolution rate | GPT-5.6 Sol (max) | Codex | 37.3% +/- 3.8% | $2.5k reported run cost; source date 2026-06-26 |
| Terminal-Bench 4.0, resolution rate | Gemini 3.8 Flash (high) | mini-SWE-agent | 19.1% +/- 3.4% | $1.8k reported run cost; source date 2026-09-02 |
| BrowseComp, 1,266 research questions | GPT-4o | No browsing | 0.6% accuracy | OpenAI's original published comparison |
| BrowseComp, 1,266 research questions | GPT-4o | OpenAI browsing | 1.9% accuracy | OpenAI's original published comparison |
| BrowseComp, 1,266 research questions | o1 | No browsing | 9.9% accuracy | OpenAI's original published comparison |
| BrowseComp, 1,266 research questions | Deep Research | OpenAI Deep Research agent | 51.5% accuracy | The source notes that the model was trained for persistent browsing |
Kimi K3 as a sourced case study
Kimi K3 is used here as a focused case study because it is a prominent open-weight agentic model with a detailed public evaluation report. The purpose is to give readers a fair, source-linked opinion of what its published results suggest and how to interpret them responsibly. This context does not invalidate or dismiss Moonshot AI's claims; it distinguishes the claims that the report directly supports from broader conclusions that require independent comparison. The source is Moonshot AI's official Kimi K3 technical report, which compares Kimi K3 (max) with several other models and documents different harnesses and evaluation procedures. These values are therefore Moonshot-reported results that should be read in the context of the report's methods, not as an independently verified universal ranking.
| Benchmark | Kimi K3 result | Harness or setup stated by the source | Source interpretation |
|---|---|---|---|
| Terminal-Bench 2.1 | 88.3 | Kimi Code harness | Moonshot's report; not the same release as Terminal-Bench 4.0 above |
| FrontierSWE | 81.2 | Kimi Code harness | Dominance score recomputed with the official evaluation script |
| SWE-Marathon | 42.0 | Claude Code harness | Based on a recalibrated H20 branch before the final v1.1 release |
| BrowseComp | 91.2 | Moonshot report; context-management methodology described in the footnotes | Highest value in the report's six-model comparison for this row |
| Toolathlon-Verified | 76.5 | Official task setup; harness detail should be read with the report footnotes | Not the highest value in the report's comparison |
| MCPMark-Verified | 94.5 | 500-task public subset; Gemini 3.1 Pro judge | Highest value in the report's six-model comparison for this row |
| OSWorld-Verified | 84.8 | Multimodal evaluation procedure described in the report | Slightly below the report's highest comparison value of 85.0 |
| tau3-Banking | 33.4 | Source cites external benchmark results for this row | Not a universal agentic score; it is one policy-constrained domain result |
In Moonshot AI's official Kimi K3 technical report, Kimi K3 records the highest result among the six compared systems on the BrowseComp row (91.2) and the MCPMark-Verified row (94.5). The same report records 88.3 on Terminal-Bench 2.1 and 84.8 on OSWorld-Verified. These are source-reported results from benchmark-specific evaluation setups, not a universal ranking across agentic benchmarks.
Fact-checking broad model claims
Popular summaries often compress a benchmark table into a universal ranking. The primary sources support a narrower reading:
| Claim in the draft | Evidence-based assessment | More accurate public wording |
|---|---|---|
| Open models such as Kimi and DeepSeek often win long-context, terminal, and multi-step agentic tasks | Too broad to state as fact. The official SWE-bench leaderboard shows Kimi K2.5 at 70.8% and DeepSeek V3.2 at 70.0% in the cited bash-only Verified snapshot, below several closed-model entries. Moonshot's K3 report shows strong but mixed results across agentic rows. | Open models can be competitive on selected agentic tasks, especially when their weights, context window, or runtime can be customized. Performance depends on the benchmark and harness. |
| Kimi K3 wins five of six major agentic benchmarks | Not established by the cited sources. Moonshot's official K3 report shows K3 leading some rows, including BrowseComp and MCPMark-Verified, while other compared systems lead rows such as Terminal-Bench 2.1, FrontierSWE, Toolathlon-Verified, JobBench, APEX-Agents, and OSWorld-Verified. The report does not define a universal six-benchmark scorecard that supports a five-of-six claim. | Moonshot reports Kimi K3 as a leader on selected agentic rows, not as the universal winner across agentic benchmarks. |
| Claude Opus, Claude Fable, and GPT-5.6 Sol dominate the hardest reasoning and coding benchmarks | Unsupported as a general conclusion. The official Terminal-Bench 4.0 leaderboard has GPT-6 Astra first, Fable 5.1 second, Opus 5 third, and GPT-5.6 Sol sixth in the displayed results. A benchmark result also does not establish reliability or safety-critical superiority. | These closed systems are strong candidates on particular coding and terminal evaluations, but leadership varies by task, harness, version, and evaluation budget. |
| Open models benefit from large context, open weights, lower cost, and community harness optimization | Partly supported, but not a measured universal advantage. Moonshot documents a one-million-token context window and open weights for Kimi K3. The SWE-bench leaderboard reports per-task costs for some entries. The effects of fine-tuning, reinforcement learning, and community harness work are workload-dependent. | Open weights, large context, and lower inference cost can create useful customization and experimentation advantages when the deployment team has the skills and infrastructure to use them. |
| Benchmarks do not measure reliability in real environments | Supported as a limitation. OpenAI describes BrowseComp as useful but incomplete for browsing, and METR measures longer-horizon capability separately from ordinary benchmark accuracy. Controlled task success should not be presented as production reliability. | Benchmark scores measure defined task performance under a defined setup; repeated-run reliability, recovery, safety, and environment robustness require separate evaluation. |
This audit is based on the official SWE-bench leaderboard, Terminal-Bench 4.0 results, Moonshot AI's Kimi K3 report, OpenAI's BrowseComp methodology, and METR's evaluation research. It does not treat vendor-reported results as independent validation, and it does not convert different benchmark percentages into one overall model rank.
Ranked score charts
The bars are sorted from highest to lowest score within each benchmark. The value above each bar is the reported result; the table above supplies the harness, date, cost, and uncertainty details.
Source: official SWE-bench leaderboard
Source: official Terminal-Bench leaderboard
Source: Moonshot AI's official Kimi K3 technical report
Source: Moonshot AI's official Kimi K3 technical report
Source: Moonshot AI's official Kimi K3 technical report
Source: Moonshot AI's official Kimi K3 technical report
The first two charts use benchmark-owned leaderboard snapshots. The BrowseComp and additional K3-report charts use Moonshot's vendor-reported six-model comparison, with benchmark-specific methods and harnesses documented in the K3 report. These charts make ranking visible within each benchmark; they do not make the benchmarks equivalent.
How to read this table
SWE-bench's percentage is resolved issues, not a general probability that the model can fix any software bug. The official page currently shows the bash-only Verified view, where entries use the same
mini-SWE-agentenvironment. This makes those rows more comparable to each other than to a result produced by a different coding harness.Terminal-Bench's percentage is a task resolution rate with a 95% confidence interval. The reported cost is the cost of the full run, not a per-task API price. A harness that spends more tokens or retries more often may score higher and cost more.
BrowseComp's percentage is short-answer accuracy on hard-to-find web facts. It is not a measure of general research quality, citation completeness, or safe browser automation. The official source explicitly describes it as a useful but incomplete measure of browsing ability.
A model name can refer to a configuration, not a single immutable object. The reasoning effort, system prompt, tool permissions, context budget, and date all matter.
Kimi K3 needs source discipline. Kimi K3 has an official Moonshot repository and a primary technical report with agentic results, but it was not present in the cited SWE-bench leaderboard snapshot. Its report supports selected benchmark-leading results, not a universal rank across all agentic evaluations.
The rows are not cross-benchmark ranks. A 76.8% SWE-bench score cannot be compared numerically with a 58.2% Terminal-Bench score or a 51.5% BrowseComp score.
The official SWE-bench leaderboard also marks whether a run was performed or directly checked by the SWE-bench team. Terminal-Bench publishes confidence intervals and warns that its benchmark data should not appear in training corpora. Those are useful trust signals, but they do not remove the need to inspect the harness, task version, evaluator, and failure cases.
The original tau-bench repository is a good example of why version discipline matters: its README warns that the older airline and retail tasks are outdated and points readers to tau3-bench for the latest fixed tasks and additional domains. Its historical scores should not be placed beside the current Terminal-Bench 4.0 results as if they were a live, apples-to-apples ranking.
What the official source should contain
An official benchmark source is more than a page with a score. Before treating a result as evidence, look for these artifacts:
| Artifact | Why it matters |
|---|---|
| Task specification | Defines what is being asked and prevents vague interpretations of success |
| Dataset or task instances | Shows whether the tasks are public, private, static, refreshed, or sampled |
| Environment definition | Identifies repository commits, browser sites, containers, operating systems, and dependencies |
| Agent interface | Shows which tools, APIs, prompts, context limits, and permissions were available |
| Evaluator or grader | Explains how success is determined and whether tests are hidden or visible |
| Reproduction instructions | Lets another team rerun the result instead of trusting a screenshot or claim |
| Version and changelog | Reveals whether two leaderboard entries used the same benchmark release |
| Limitations and contamination notes | Clarifies memorization, task leakage, flaky tests, and known blind spots |
If a result provides only a model name and a percentage, it is a claim to investigate, not a reliable comparison.
How trustworthy are these benchmarks?
The short answer is: useful, but conditional. A benchmark can be trustworthy as a measurement of its defined task while still being a poor proxy for production reliability.
| Trust level | Evidence | How to use the result |
|---|---|---|
| Strong benchmark evidence | Public task definition, versioned environment, inspectable evaluator, reproducible runner, and independent confirmation | Compare systems within the same release and configuration |
| Useful directional evidence | Serious benchmark with documented tasks, but incomplete reproduction details or changing infrastructure | Use for candidate selection and research direction, not procurement claims |
| Weak leaderboard evidence | A score copied from a chart with no task split, harness, attempt count, or evaluator details | Treat as a lead; do not use it as proof of capability |
| Marketing evidence | A selective demo, unnamed tasks, best-run examples, or a score with no failure analysis | Do not generalize to production readiness |
Even strong benchmark evidence has limits:
Construct validity: The benchmark may not represent the work your organization cares about.
Contamination: Public tasks, solutions, or test outputs may have appeared in training data or agent traces.
Evaluator validity: Tests can be incomplete, flaky, overly narrow, or accidentally reward a shortcut.
Harness dependence: A score may reflect a planner, prompt, retry policy, tool wrapper, or budget as much as the underlying model.
Environment dependence: Small changes to package versions, websites, permissions, or repository state can change the result.
Selection effects: Teams may report the best run, the best task subset, or a model configuration that is too expensive for normal use.
Pass-rate ambiguity: A 70% score may mean seven of ten tasks passed once, or 70% over many repeated trials. Those are not the same claim.
The right interpretation is therefore narrow:
This system completed this class of tasks, in this environment, with this harness, under this evaluator, at this reported rate.
That statement is much more defensible than "this model is 70% autonomous."
A benchmark trust checklist
When reading a leaderboard or technical report, record:
The exact model version and reasoning configuration.
The benchmark version, task split, and environment image.
The tools and permissions available to the agent.
The context, token, time, retry, and cost budgets.
Whether the score is pass@1, best-of-many, or averaged across repeated runs.
Whether tasks were public, private, refreshed, or screened for contamination.
Whether the evaluator checks only the final answer or also policy and side effects.
Whether an independent team reproduced the result.
The failure cases, not only the aggregate score.
How closely the benchmark resembles the production workflow being considered.
For enterprise decisions, public benchmarks should be treated as external evidence. The final gate should be an internal benchmark built from representative, sanitized tasks with the organization's own permissions, approval rules, evaluator, and reliability target.
The benchmark is not the model
The phrase "agentic benchmark" can hide an important distinction. A benchmark score usually reflects a system, not a model in isolation.
The model contributes reasoning, language understanding, planning, and tool selection. The surrounding harness contributes the environment, tool definitions, memory, retry policy, context management, permissions, and stopping conditions.
This is why the same model can perform very differently across two evaluations. A clean tool schema, useful state summaries, bounded permissions, and a good recovery loop can improve results dramatically. Conversely, a weak harness can make a capable model appear unreliable.
A useful benchmark report should therefore identify at least five layers:
| Layer | Question it answers |
|---|---|
| Task | What job was the system asked to complete? |
| Environment | What repository, website, data, or infrastructure did it operate on? |
| Harness | Which tools, memory, context window, retry rules, and stop conditions were available? |
| Model | Which model selected and generated the actions? |
| Outcome | What objective evidence determined success or failure? |
Without these details, a leaderboard number is difficult to reproduce and easy to misunderstand.
What agentic benchmarks measure
1. Planning and decomposition
Can the system turn a broad objective into a sequence of useful steps?
A request such as "add authentication" is not an executable plan. A competent agent may need to inspect the existing identity model, define a user table, choose a password strategy, implement the login path, add tests, validate edge cases, and document configuration.
The benchmark is not rewarding a long plan for its own sake. It is testing whether the plan creates productive progress and adapts when the environment disagrees with the initial assumption.
2. Tool use
Can the agent use terminals, browsers, APIs, databases, IDE services, and search systems correctly?
Good tool use includes selecting the right tool, supplying valid arguments, reading the result, and avoiding unnecessary actions. It also includes understanding when a tool call is unsafe or outside the task's authority.
3. State tracking and memory
Can the system preserve the objective, constraints, discoveries, and unresolved questions over a long run?
Long context is helpful, but it is not the same thing as useful memory. A reliable agent needs to distinguish durable facts from stale observations, failed experiments from current state, and evidence from speculation.
4. Error recovery
What happens after the first command fails?
A strong agent classifies the failure, updates its hypothesis, and tries a targeted next step. A weak agent repeats the same action, silently changes the goal, or produces a confident explanation without repairing the state.
5. Outcome completion
Was the work actually completed?
A useful benchmark has an objective evaluator: a test suite, a file checksum, a browser state, a structured answer with citations, a database invariant, or an infrastructure health check. "The agent said it was done" is not an outcome evaluator.
6. Constraint following
Can the agent succeed without violating its operating rules?
Enterprise evaluation must score more than task completion. It should also test whether the agent respects permissions, approval gates, data boundaries, network restrictions, and prohibited actions.
The agentic execution loop
Most agents operate as a repeated observe-decide-act cycle. The model does not solve the complete task in one hidden thought. It chooses a next action from the state it can currently observe, receives a result, and revises its plan.
The loop exposes an important evaluation question: what counts as progress? An agent can make hundreds of valid tool calls and still fail because it never reaches the required state. This is why trajectory length, token usage, and tool-call count are useful diagnostics but poor substitutes for outcome quality.
The benchmark landscape
The names and versions of public benchmarks change quickly, but the task families are durable.
Terminal and computer-use tasks
Terminal-oriented evaluations place the system in a shell or development environment. The agent may need to navigate directories, inspect configuration, edit files, install or use dependencies, run tests, and diagnose command failures.
These tasks test operational fluency: the ability to make progress in a real environment rather than merely describe a solution.
SWE-Bench and software-engineering tasks
Software-engineering benchmarks provide a repository, an issue, and an evaluation suite. The system must understand the existing implementation and produce a patch that resolves the reported behavior.
This is materially harder than code generation because the repository contains history, conventions, hidden assumptions, unrelated complexity, and tests that may fail for different reasons.
More difficult variants increase repository size, issue complexity, cross-module dependencies, or the number of iterative repairs required. Long-horizon engineering evaluations push closer to the rhythm of real maintenance work: investigate, patch, test, learn, and patch again.
Browser and research tasks
Browser benchmarks evaluate navigation, search, page understanding, form interaction, comparison, and information synthesis. The agent must deal with dynamic interfaces, incomplete results, ambiguous wording, and pages that are not designed as APIs.
Research-oriented tasks add source quality, contradiction handling, and citation requirements. A system that finds a plausible page but cannot explain why the page is authoritative has not completed the research task.
Long-horizon planning tasks
Some evaluations emphasize difficult reasoning over extended trajectories. The system must retain goals, manage subproblems, recover from dead ends, and decide when it has enough evidence to stop.
These tasks are particularly useful for exposing context drift. A response can be locally coherent at every step while the overall run slowly moves away from the original objective.
A better scorecard than a single percentage
A single pass rate hides too much. Two agents might both complete 70% of tasks, while one is fast and safe and the other succeeds only after destructive or expensive behavior.
A practical enterprise scorecard should separate outcome quality from operating quality.
| Dimension | Example measure | Why it matters |
|---|---|---|
| Task success | Percentage of evaluator checks passed | Did the system achieve the objective? |
| Correctness | Assertions, tests, factuality, citation quality | Was the result right, not merely complete? |
| Policy compliance | Forbidden-action and approval-gate violations | Can the system operate within authority? |
| Recovery quality | Success after injected failures | Can it adapt instead of looping? |
| Efficiency | Tokens, tool calls, elapsed time, compute cost | Is the behavior economically viable? |
| Reliability | Variance across repeated runs | Does one good run generalize? |
| Evidence quality | Trace completeness and provenance | Can a human audit the result? |
| Human burden | Approvals, corrections, escalations | Does the system reduce work or move it elsewhere? |
This scorecard also prevents a common mistake: optimizing the model to finish more tasks while ignoring the cost of supervision, remediation, and risk.
Why the harness can matter more than the model
Context windows are not memory systems
An agent may read a large codebase, but retaining every token does not guarantee that it will use the right information at the right time. Long workflows need summaries, checkpoints, retrieval, and explicit state transitions.
The relevant question is not only, "How much context fits?" It is also, "Which facts survive, who wrote them, when were they observed, and can they be challenged?"
Tool interfaces shape behavior
A tool that returns unstructured prose forces the model to infer state from text. A tool that returns typed fields, source references, timestamps, and explicit errors gives the agent a more stable operating surface.
Small, composable tools also improve governance. get_test_failures() is easier to inspect and permission than an unrestricted tool named manage_repository().
Recovery policy determines whether failure is useful
Failure is valuable only when the agent can learn from it. The harness should capture the command, inputs, output, changed state, and likely failure category. This makes retries targeted instead of ceremonial.
Stopping rules protect the environment
An agent that never decides it is finished can consume budget, repeat actions, or create new damage. Completion criteria should be explicit, and the system should have a hard stop for repeated failures, policy conflicts, missing authority, and unexpected state changes.
Open and closed models: the enterprise decision
Model comparisons are useful when they are tied to a workload rather than a universal ranking.
Open-weight and openly available models can be attractive for high-volume execution, customization, deployment flexibility, and cost control. They may be especially effective when an organization can tune the model or optimize the surrounding runtime for terminal, coding, or research trajectories.
Closed frontier models often provide strong general capability, polished tool calling, broad multimodal support, and mature operational controls. They can be a good fit when reliability, safety behavior, and time to deployment matter more than model-level customization.
The practical comparison should include:
performance on the organization's own task distribution
reliability across repeated runs, not just best-run scores
tool-call correctness and recovery behavior
data handling and isolation requirements
latency and cost at the expected run length
availability, regional constraints, and operational support
ability to inspect traces and enforce policy
A benchmark leaderboard can identify candidates. It cannot make the deployment decision by itself.
The security boundary changes when AI can act
A chatbot can be wrong in a conversation. An agent can be wrong in the environment.
That difference turns model evaluation into a security and governance problem.
Dangerous actions
A terminal agent may be able to delete files, alter configuration, install packages, or change infrastructure. High-impact operations should be denied by default or placed behind explicit approval.
Secret exposure
The agent should not need to see raw credentials to use an approved capability. Prefer managed identity, short-lived scoped tokens, secret references, and tools that perform the operation without returning the secret value.
Prompt injection
Untrusted pages, files, tickets, and repository content can contain instructions aimed at the agent. The runtime must treat retrieved content as data, not authority. System policy and tool authorization must remain outside the influence of the content being inspected.
Supply-chain risk
Installing a dependency or running an external script changes the trust boundary. Package sources, hashes, licenses, network egress, and review policy should be part of the environment, not left to a model's judgment.
The security pattern is straightforward even when implementation is not: isolate the agent, minimize authority, separate instructions from data, gate consequential actions, and log enough evidence to reconstruct what happened.
What benchmarks are good for
Agentic benchmarks are valuable for five reasons:
Model selection: they reveal which models fit a task family and operating budget.
Harness evaluation: when the benchmark reports the complete model-and-harness configuration, it can reveal how tools, memory, context management, retries, and stopping rules affect task completion. This is a system-level signal, not a universal or isolated measure of memory retention or long-context adherence.
Regression testing: they show whether a prompt, tool, or model change damaged a previously working workflow.
Risk testing: they make policy violations and unsafe trajectories measurable.
Readiness decisions: they provide evidence for where an agent can operate autonomously and where it still needs human review.
The best internal benchmark is usually not a copy of a public leaderboard. It is a representative set of tasks from the organization's real work, with synthetic or sanitized environments, explicit success criteria, seeded failures, and a reviewable trace.
A benchmark becomes genuinely useful when every failed run teaches the team something actionable. If the only conclusion is that the score went up or down, the evaluation is not yet connected to engineering decisions.
A practical enterprise checklist
Before running an evaluation
Define the task objective in observable terms.
Create a repeatable environment and a known starting state.
Separate trusted instructions from untrusted task content.
Define permitted tools and prohibited actions.
Decide which actions require human approval.
Establish a deterministic evaluator where possible.
During the run
Record every tool call, argument, result, and state change.
Capture failures as structured events rather than only raw logs.
Enforce time, token, action, and cost budgets.
Detect repeated actions and stalled progress.
Keep secrets outside the model-visible context.
After the run
Verify the final state independently.
Score correctness, policy compliance, efficiency, and evidence quality.
Review representative failures, not only aggregate scores.
Compare repeated runs to measure variance.
Preserve regression cases for future model and harness changes.
Final perspective
Agentic benchmarks mark a transition from reasoning-centric AI to execution-centric AI. The winning system is not necessarily the one that writes the most impressive answer. It is the one that can make progress in a real environment, recover when the environment disagrees, respect its authority, and prove what it accomplished.
The model remains important. But the model is only one component of the worker.
The surrounding system determines whether the worker has useful memory, safe tools, clear boundaries, reliable recovery, and an honest definition of done. That is why the next generation of evaluation must measure the whole loop:
Intent -> observation -> decision -> action -> verification -> accountable outcome.
Enterprises should use public benchmarks to understand the field, then build private evaluations around their own workflows and risk boundaries. The most meaningful question is not, "Which model has the highest score?"
It is:
Which agent system can complete this job correctly, repeatedly, economically, and within the authority we are willing to grant it?
That is the benchmark that matters.
Suggested reading and benchmark families
The links below are good starting points because they lead to the benchmark's primary repository, paper, documentation, or maintainer guidance. Scores and task versions change, so check the release date, task split, model version, tool configuration, and evaluation harness before comparing results.
Software engineering
SWE-bench repository: The original benchmark and implementation for resolving real GitHub issues in real repositories.
SWE-bench website and leaderboard: Task descriptions, results, and benchmark updates.
SWE-bench-Live: A continuously refreshed benchmark intended to reduce contamination from older public tasks.
SWE-bench Multimodal: Software issues that include screenshots, visual context, or other non-text evidence.
Terminal and shell-use evaluations
Terminal-Bench: A benchmark for completing realistic tasks in a terminal environment.
Terminal-Bench documentation: Task format, runtime details, and current benchmark information.
SWE-agent: An influential open-source coding-agent system and a useful reference for studying agent interfaces and trajectories.
Harbor: An open framework for running and evaluating agents in containerized environments.
Browser and computer-use evaluations
BrowserGym: An environment and benchmark suite for web interaction and browser agents.
WebArena: A realistic, self-hostable web environment for evaluating multi-step browser tasks.
Mind2Web: A benchmark and dataset for web-agent interaction across real websites.
OSWorld: An evaluation environment for multimodal agents operating desktop applications and operating-system interfaces.
BrowseComp paper: A difficult browsing and information-retrieval evaluation focused on finding hard-to-locate answers on the web.
Long-horizon planning, tool use, and research
AgentBench: A multi-environment benchmark covering several classes of agent tasks.
tau-bench: Tool-agent interaction tasks with user and policy constraints, useful for studying reliability and rule following.
GAIA benchmark: General assistant tasks that combine reasoning, tool use, browsing, and multimodal information.
BrowseComp-Plus: Additional resources for difficult browse-and-synthesize research tasks.
METR research: Research on measuring the capabilities and reliability of AI systems on longer, more open-ended tasks.
Model reports and evaluation harnesses
Anthropic research: Technical reports and research posts covering model behavior, reasoning, agents, and safety.
OpenAI research: Model and evaluation reports from OpenAI.
DeepSeek research: Official repositories and technical artifacts from the DeepSeek team.
Kimi technical reports: Official Moonshot AI repositories and model documentation, including the Kimi K3 repository.
Inspect AI: An open evaluation framework from the UK AI Security Institute for defining tasks, graders, solvers, and evaluation suites.
OpenAI Evals: An open-source framework and registry for model and system evaluations.
lm-evaluation-harness: A widely used framework for standardized language-model evaluations. It is primarily model-evaluation infrastructure rather than a complete agent benchmark, which makes that distinction worth keeping in mind.
Security and governance
OWASP Top 10 for LLM Applications: Major application risks for systems built with large language models.
OWASP Top 10 for Agentic Applications: Risks specific to systems that plan, use tools, maintain state, and take actions.
OWASP GenAI Security Project: Broader guidance, threat models, and community resources for generative AI security.
NIST AI Risk Management Framework: A governance framework for managing AI risks across design, deployment, and operation.
How to use these links responsibly
Treat a benchmark result as evidence about a particular configuration, not as a permanent ranking of intelligence. When reading a result, record the model version, system prompt, tools, context limits, environment image, number of attempts, cost budget, evaluator version, and whether the agent was allowed to retry or ask for help. Those details can change the meaning of the score.
Appendix: The Agentic AI Vocabulary
The terminology around agentic AI is still inconsistent. The following definitions use a practical engineering meaning and keep the boundaries visible.
The short version
| Term | Practical meaning | Example in a coding task |
|---|---|---|
| Agent | A system that pursues a goal by observing state and choosing actions | A program that investigates and fixes a failing test |
| LLM | A large language model that predicts and generates tokens | A model that reads the issue and proposes the next command |
| Model | The learned model used for reasoning, language, vision, or action selection | GPT, Claude, Kimi, DeepSeek, or another deployed model |
| Harness | The runtime around the model: tools, memory, policies, prompts, retries, and evaluators | The service that exposes PowerShell, captures output, and stops unsafe commands |
| Tool | A callable capability that reads or changes an external system | run_tests, read_file, search_web, or create_pull_request |
| Environment | The world in which the task is performed | A repository checkout, browser session, database, or sandbox |
| Observation | Information returned from the environment after an action | A compiler error, test result, page, or database row |
| Memory | State retained across steps or runs | The issue summary, attempted fixes, and confirmed test results |
| Planner | The component that selects or organizes future work | A plan to inspect the test, trace the code path, patch it, and rerun tests |
| Evaluator | An independent checker that decides whether the task succeeded | A test suite that verifies the checkout bug is fixed |
| Trajectory | The ordered record of observations, decisions, and actions | Every command, output, edit, retry, and final result in one run |
| Guardrail | A technical or policy control that limits behavior | Blocking Remove-Item -Recurse outside the sandbox |
| Human-in-the-loop | A workflow where a person reviews or approves selected actions | A maintainer approves the pull request before it is merged |
Agent
An agent is a goal-directed system that can choose actions based on its current state and the results of previous actions. It is more than a model answering a single prompt, although a model is often one of its components.
For example, given the task:
Fix the failing checkout test and prepare a patch.
An agent might:
Read the issue and repository instructions.
Inspect the failing test and the checkout implementation.
Run the narrow test to reproduce the failure.
Edit the smallest relevant code path.
Run the test again.
Inspect a new failure if one appears.
Return the patch, test result, and remaining limitations.
A chatbot that only suggests code is not necessarily an agent. It becomes agentic when it can operate through a stateful action loop and is evaluated on the resulting task state.
LLM
LLM means large language model. It is a model trained on large volumes of text and code to predict and generate language tokens. An LLM can summarize an issue, write a command, explain an error, or propose a patch.
An LLM is not automatically an agent. By itself, it may only receive text and return text:
Input: The checkout test fails when the coupon is expired.
Output: Check whether the expiration comparison uses local time or UTC.
The surrounding runtime turns that language capability into action by giving the model tools, state, permissions, and a way to observe results:
Model proposes: run_tests("tests/test_checkout.py::test_expired_coupon")
Harness executes the tool.
Environment returns: AssertionError at checkout.py:184.
Model updates its hypothesis and selects the next action.
Some modern models also process images, audio, video, or structured tool calls. In that case, “LLM” may be too narrow; foundation model or multimodal model may be more accurate. The engineering distinction remains the same: the model generates or scores actions, while the runtime executes and governs them.
Model
A model is the learned computational component used by the system. In an agentic application, it might handle language reasoning, vision, speech, embeddings, classification, or action selection.
The word model is broader than LLM. For example:
A language model interprets an issue and writes a patch.
A vision model reads a screenshot of a browser failure.
An embedding model helps retrieve relevant documentation.
A small classifier identifies whether a command is high risk.
When people say that one model “wins” an agent benchmark, check whether the comparison actually holds the harness, tools, prompts, environment, and budget constant. Otherwise the result may measure the complete system rather than the model alone.
Harness
The harness is the software layer that makes an evaluation or agent run possible. It connects the model to the environment and defines what the model can see, what it can do, and when the run ends.
A harness commonly contains:
system instructions and task context
tool definitions and argument validation
memory and conversation state
context-window management and summaries
permission checks and approval gates
retry, timeout, and budget policies
trace capture and logging
completion checks and evaluators
For the checkout example, the harness might expose read_file, search_code, run_tests, and apply_patch. It may reject arbitrary shell commands, mask environment secrets, limit the agent to a temporary repository, and require approval before creating a pull request.
This is why “the model failed” is often an incomplete diagnosis. The model may have selected a reasonable action that the tool schema represented poorly, or the harness may have hidden the error that would have allowed recovery.
Tool
A tool is a callable interface through which the agent reads or changes something outside the model.
Examples include:
read_file(path)
search_code(query)
run_tests(test_command)
get_customer_order(order_id)
search_web(query)
create_pull_request(title, body, branch)
Tools should have narrow purposes, typed inputs, explicit errors, and a documented authority boundary. A tool named execute_anything is difficult to secure and evaluate. Several small tools make the trajectory easier to inspect and the permissions easier to reason about.
Environment
The environment is the external world in which the agent operates. It provides the state the agent observes and the side effects produced by its actions.
Examples:
a checked-out Git repository
a browser with websites and logged-in test accounts
a database populated with synthetic records
a container with installed packages
an infrastructure sandbox with restricted cloud permissions
An environment should have a known starting state when used for benchmarking. If one run starts with cached dependencies and another starts without them, the scores are not directly comparable.
Observation
An observation is the result the agent receives after inspecting the environment or calling a tool. It may be text, structured JSON, an image, a file diff, a test result, or an error.
In the checkout example:
{
"tool": "run_tests",
"status": "failed",
"test": "test_expired_coupon",
"error": "expected 0, received 10",
"location": "checkout.py:184"
}
Structured observations reduce ambiguity. They tell the model what happened without requiring it to infer every field from a paragraph of logs.
Memory and context
Context is the information currently included in the model's input. Memory is information retained or retrieved across steps or runs. They overlap, but they are not identical.
For example, the current context may contain the latest test failure. Durable memory may contain the original issue, repository conventions, previous attempts, and a record that the test passed after the patch.
Useful memory should include provenance and time. A remembered statement such as “the API uses UTC” is more trustworthy when the system can show where that fact came from and whether it is still current.
Planner and executor
A planner decides what should happen next or creates a sequence of subgoals. An executor performs the selected action through a tool.
Some systems use one model for both jobs. Others separate them:
Planner: inspect checkout.py, reproduce the failure, then patch and retest.
Executor: run search_code("coupon expiry")
Environment: returns matching files and lines.
Planner: inspect checkout.py:184.
Executor: read_file("checkout.py", start=160, end=205)
Separating planning from execution can make policy checks and observability clearer, but it also adds coordination overhead. The right design depends on task length, risk, and latency requirements.
Evaluator
An evaluator independently checks whether the required outcome was achieved. It should not rely only on the agent's final explanation.
Examples:
Run a hidden regression test against the patched repository.
Check that a browser reached the required confirmation page.
Compare a structured answer against reference facts and citations.
Verify that a database invariant holds after the workflow.
Confirm that no forbidden file or resource was modified.
For a high-trust system, use more than one evaluator dimension. A patch can pass the visible test while violating a policy, leaking a secret, or modifying unrelated files.
Trajectory
A trajectory is the chronological record of a run:
goal -> observation -> model decision -> tool call -> result -> next decision -> ... -> outcome
Trajectory review is one of the most useful debugging techniques for agents. It can reveal that the model chose a poor action, that a tool returned misleading data, that memory was stale, or that the evaluator accepted an incomplete result.
Guardrail and policy gate
A guardrail limits what the system can do. A policy gate is a decision point that checks whether an action is allowed before it occurs.
Examples:
allow read-only repository access but block writes outside a workspace
require approval before merging a pull request
prevent outbound network access from a test container
redact credentials from tool output
stop after three repeated failures
Guardrails should be enforced by the runtime or infrastructure whenever possible. A model instruction such as “do not delete production data” is useful, but it should not be the only control protecting production data.
Human-in-the-loop
Human-in-the-loop means that a person participates at a defined point in the workflow. The person may approve an action, resolve ambiguity, provide missing information, or review the final result.
The best approval boundary is usually tied to impact rather than every action. Reading files may be autonomous; changing production permissions may require approval. A good system shows the proposed action, supporting evidence, expected impact, and rollback path before asking for a decision.
Putting the terms together
Here is the complete vocabulary in one small execution trace:
| Runtime step | Term in action | Concrete example |
|---|---|---|
| 1. Start | Goal | Fix the expired-coupon checkout bug |
| 2. Interpret | Agent + LLM/model | Read the issue and choose the next investigation step |
| 3. Coordinate | Harness + planner | Keep task state, expose approved tools, and sequence reproduce -> inspect -> patch -> retest |
| 4. Act | Tool + environment | Run run_tests("test_expired_coupon") and inspect checkout.py in a temporary repository |
| 5. Learn | Observation | The test shows that an expired coupon is still applied |
| 6. Control | Guardrail | Reject writes outside the temporary checkout |
| 7. Verify | Evaluator | Run hidden regression tests and inspect the final diff |
| 8. Remember | Trajectory | Preserve every decision, tool call, result, retry, and state change |
| 9. Finish | Outcome | Tests pass and the intended patch is ready for review |
The vocabulary matters because it makes failures diagnosable. “The AI was wrong” is a starting complaint, not an engineering explanation. A useful diagnosis identifies whether the model misunderstood the task, the harness omitted context, the tool returned a misleading observation, the environment was inconsistent, the policy blocked a necessary action, or the evaluator measured the wrong outcome.