Agent Evaluation & Benchmarking

Deterministic test suites that tell you whether an agent actually works.

What it is

A demo proves that an agent can work once. An evaluation tells you how often it works, on which inputs, and what happens when a tool fails. GTDataWorks writes the scenarios with you, grades every attempt with deterministic code rather than a model's opinion, and keeps a receipt for each attempt, so a result can be re-checked by someone who was not in the room. Where automation is not enough, a person reviews the disagreement and that review is recorded too.

Tool Failure Recovery v2 is a recorded 500-scenario study of agent tool-failure recovery. It retains accepted and failed outcomes and distinguishes evaluation evidence from the successful examples selected for training. The case study documents its protocol, counts, and limits.

500scenarios
629recorded attempts
447accepted scenarios
53retries exhausted

Historical Tool Failure Recovery v2 study. Model attempts exclude 54 synthetic terminal markers. Inspect the study, source checks, and scope.

What you get

  • Scenario setThe inputs, expected outcomes and failure conditions you agreed to test, in a form that can be run again after every change.
  • Grading scriptDeterministic code that scores each attempt the same way every time. You receive it, so you can re-grade the results yourself.
  • Results with per-attempt receiptsA score per model, prompt or agent variant, and for every attempt a receipt: what went in, what came out, the grader's verdict and a SHA-256 hash.
  • Retained-proof reportA finding ledger, an evidence manifest with SHA-256 hashes for every file, and a negative control that shows the graders can fail.

How the work runs

  1. Define the scenarios with you

    What the agent must do, what counts as success, and which failures to inject: timeouts, bad tool output, missing permissions. The scenario count is set by what is being decided, and it sets the price.

  2. Write deterministic graders

    Each scenario gets a grader in code that returns the same verdict for the same output every time. Where a verdict cannot be decided in code, it goes to human review rather than to another model.

  3. Run every variant with per-attempt receipts

    Each model, prompt or agent variant runs the agreed scenario set in the evaluation runner. Every attempt, including retries and failures, is kept with its receipt.

  4. Human review of disagreements

    Attempts where a grader's verdict is in doubt, or where you disagree with it, are reviewed by hand. The review and its reasoning are recorded alongside the receipt.

  5. Retained-proof report

    The results arrive as a report a stranger can re-check: a finding ledger with IDs and dispositions, an evidence manifest listing every file with its size and SHA-256 hash, and a negative control, a deliberately wrong submission the graders must fail.

Pricing

Per evaluation

$500–$2,000

Scoped by scenario count and models compared; includes per-attempt receipts and a retained-proof report.

Prices are in USD and are starting points. The final quote follows a scoping call, where the scenario count and the number of variants compared set the figure.

Where it fits

A good fit when

  • You are choosing between models, prompts or vendors and need a number you can defend, not a demo.
  • You have an agent that calls tools and want to know what it does when a tool fails, before your users find out.
  • The result will be read by someone sceptical: a buyer, an auditor, a technical lead.

Not a fit when

Questions

Can the results be published?

Yes, if you decide to; the results are yours. Because every attempt carries a receipt and the report includes an evidence manifest, anyone you hand the bundle to can check a published number against the attempts behind it.

Does GTDataWorks train on my scenarios or transcripts?

No. Your scenarios, transcripts and results are used for your evaluation and nothing else. The studio draws the same line in its own dataset licensing: evaluation rights are not training rights.

Which models can be evaluated?

Any model you can give API access to, including fine-tunes and self-hosted open models, and whole agents as long as the runner can invoke them. Several variants can be compared in one evaluation; the count is part of the scope.

What is a negative control, and why is it in the report?

A deliberately wrong submission run through the same graders. If the graders pass it, the graders are broken. It is in every report so you can see that the checks are capable of failing before you trust them when they pass.

Get a number you can defend

Bring the agent, the models or the prompts you are deciding between. The scenarios are written with you, and the results come back with receipts.