GTD DATASET FACTORY CORE
A recovery study that keeps the whole record.
A useful training example and an honest evaluation need different views of the same evidence. This study preserves that distinction.
Historical Tool Failure Recovery v2 · Archived August 2026 · Counts rechecked November 7, 2026
What changes when you select only successes?
The evaluation contains 500 scenarios across ten tool-failure templates. Each scenario receives up to three semantic attempts; the protocol stops when the grader accepts a recovery.
447 scenarios met the recovery criteria. 53 exhausted their retry budget. A training projection selects one successful example per accepted scenario; the evaluation retains the full scenario population.
The controls inspect recorded aggregate results. They do not run a model.
All evaluated scenarios
447 accepted outcomes and 53 retry-exhausted outcomes. Failed scenarios remain visible in the denominator.
447 accepted · 53 retries exhausted
Results by failure template
“Accepted” means the recorded recovery met this study’s declared rubric. Each template contributes 50 scenarios.
| Failure template | Accepted | Retries exhausted | Scenarios |
|---|---|---|---|
| Wrong step order | 50 | 0 | 50 |
| Skipped step | 50 | 0 | 50 |
| Cascading failure | 50 | 0 | 50 |
| Recovery loop | 50 | 0 | 50 |
| Wrong tool | 50 | 0 | 50 |
| Missing argument | 50 | 0 | 50 |
| Schema violation | 49 | 1 | 50 |
| Wrong argument value | 48 | 2 | 50 |
| Timeout | 50 | 0 | 50 |
| Permission denial | 0 | 50 | 50 |
| Total | 447 | 53 | 500 |
These are scenario outcomes under a fixed protocol. Later attempts are conditioned on earlier failures; the attempts are not independent trials. The model identifier is taken from the retained records.
A rejected result needs a diagnosis.
The permission-denial template produced no accepted recovery in the primary cohort. A separate nine-attempt diagnostic also produced no accepted recovery across its 50 scenarios.
That does not establish permission bypass. All 450 diagnostic attempts recorded the strategy respect_permission_boundary and an empty forbidden-tools list. One inspected blocked response failed schema checks for a missing required field and an additional field.
The distinction matters: a rubric failure can expose an output-contract problem. The diagnostic remains separate from the 500-scenario evaluation.
What the record contains
- Model-attempt records
- 629
- Synthetic terminal markers
- 54
- Total receipt records
- 683
Scenario outcomes, attempt records, and terminal markers are different counts. Preserving them separately lets a training view change without rewriting the historical record.
Source verification and limits
On November 7, 2026, we recounted scenario rows, training rows, and receipt-index records from the retained August archive. All eight files listed in the pack’s checksum manifest matched. The packaged scenario view matched the retained source view byte for byte.
This check did not rerun the model, recompute the rubric, validate every sealed tape, or establish independent human sign-off. The downloadable summary contains aggregate counts and source hashes; it does not contain the raw traces needed to independently regrade the study.
The success-only training view contains examples from both planned splits and is not a held-out evaluation set. This historical upstream pack has been superseded as a commercial release.
A hash identifies bytes and detects changes relative to a recorded value. It does not establish correctness, authorship, or general safety.
Scenario view SHA-256a7654b7f8a4d583dc92b45cb9bdd0c38a1ea8219949bfd5cf5aaa4fe02ef83cb
How the dataset is produced
The factory distinguishes the scenario, each attempt, the retained evidence, and the projection built from it. Evaluation reports and training rows are derived views. They do not replace the underlying execution records.
For this capture, the recorded protocol is tool_failure_semantic_v2 with the replay_direct prompt builder and a three-attempt primary budget. The permission diagnostic has a separate context and budget.
Test the behavior your workflow depends on.
We can help define the scenarios, build the runner, and turn the resulting records into an evaluation or dataset suited to your use.