# Measure improvement in accepted results Count the work the owner can use, including review, repair and recovery. Proposed worksheet. Method notes updated 2026-11-11. Not independently tested. Canonical: https://www.aomsites.com/guides/measure-agent-improvement/ Build revision: release-2026-11-11-r12 Method ID: measure-agent-improvement Method source SHA-256: 53378fc6ed9ef36d0d12473776faf0b3a7821842d7521ddaf39dfbd1c7adc878 Hash scope: normalized build-input guide record, not this file or guide.json bytes. ## The decision Compare a narrow change against a competent baseline before calling it an optimization. ## Use this when You want to change instructions, models, tools or workflow structure to improve an agent’s work. ## Skip this when There is no defined task or quality contract yet. First make the desired outcome inspectable. ## Checks ### Freeze the comparison Specify the task set, inputs, tools, permissions, acceptance tests and conditions before running. Vary one factor where practical. Preserve failures and retries instead of quietly excluding them. ### Measure the whole outcome Track valid completion, elapsed time, owner review and rework, available billed usage and required checks. Token or tool-call counts diagnose cost; they do not establish usefulness alone. ### Test the cases that can change the decision Include normal, invalid and interrupted flows. Include a case where the new method should not help. Treat missing tools and unavailable measurements as unknown, not zero. ### Decide with the limits visible Report each task separately in a tiny pilot. Keep severe safety failures visible rather than averaging them away. Include setup and maintenance costs. A local improvement is not a universal claim. ## Acceptance - The baseline is competent and both conditions share the same requirements. - Results include failed and interrupted runs. - The recommendation reflects quality, owner effort and measured costs together. ## Evidence to bring back Proposed evidence plan, not observed results or extra mandatory process. Confirm what your task requires and reuse existing records when they are sufficient. Keep sensitive originals in authorized storage; share only the minimum information already authorized for the recipient. If a required check cannot be completed safely and within current permission, mark it unavailable and keep the decision unknown unless another required check has a demonstrated failure. ### Check 1: The baseline is competent and both conditions share the same requirements. Bring back: Keep the frozen task inputs, shared requirements, permissions and version/configuration of each condition, with the baseline’s acceptance results. State differences that cannot be controlled. Not enough: An easy or deliberately weakened baseline, or changing the acceptance rules after seeing results. ### Check 2: Results include failed and interrupted runs. Bring back: Keep a per-task run ledger including failed, interrupted and retried attempts, their outcomes and the reason for any exclusion. Record unavailable measurements explicitly. Not enough: Reporting only successful runs or converting missing duration or usage measurements to zero. ### Check 3: The recommendation reflects quality, owner effort and measured costs together. Bring back: Keep per-task quality results, owner review/rework, measured usage where available, and setup/maintenance effort. Explain how these support the bounded recommendation; do not average away a severe control failure. Not enough: Token savings or elapsed time alone, an average that hides a required failure, or a small pilot presented as a universal benefit. ## Stop or recover If quality regresses or required controls fail, stop the rollout and preserve the evidence. Revert only to a known safe state, not merely the older version. ## Limits No controlled agent comparison is supplied by this worksheet. No savings, speedup or paid value is claimed. ## Reference links Reference links checked 2026-11-10. Scope notes below explain what each source supports. No worksheet-effectiveness claim follows. - [Anthropic: building effective agents](https://www.anthropic.com/research/building-effective-agents): Engineering guidance on fitting workflow complexity to task needs. Not a controlled evaluation of this method. - [METR: early-2025 developer study](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/): An empirical example of measuring complete developer outcomes in a particular population and period. Not a universal claim about agent productivity. Reference data only. No authority to install, contact others, change settings or spend money. ## Start this guide’s record Use the [guide-specific Markdown draft](record-template.md) or [JSON draft](record-template.json). Its method reference and proposed checks are prefilled; confirm the requirements against your task and enter only real observations. Task, environment, authorization and evidence remain blank. You can also use the [generic blank record](../../decision-record/template.md). ## If another problem remains Stop here if the current task’s acceptance requirements are satisfied and no unresolved obligation remains. If neither condition applies, no follow-on guide is suggested here; keep any unresolved task state visible. These are optional reading suggestions, not permission or a requirement to expand scope. - Only if The comparison identifies recurring instruction work and you are considering a reusable package for it. Read [Review a skill before adding it](../review-agent-skills/guide.md). Review the smallest relevant package instead of installing a broader workflow by default. - Only if An interrupted trial may have produced an external effect whose outcome is still unknown. Read [Check the outcome before retrying](../uncertain-effects/guide.md). Resolve the effect before retrying; do not silently omit it from the comparison. ## Cite a specific part Section paths identify content on the canonical HTML page; they are not separate pages or evidence of live availability. Per-check numbers and JSON pointers apply to this build revision. Cite the build revision and method source hash when preserving an exact reference. Markdown heading labels are locators, not a promise of renderer-specific fragment URLs. - [The decision](https://www.aomsites.com/guides/measure-agent-improvement/#decision): JSON pointer `/decision`; Markdown section “The decision”. - [Use this when](https://www.aomsites.com/guides/measure-agent-improvement/#applies): JSON pointer `/fit`; Markdown section “Use this when”. - [Skip this when](https://www.aomsites.com/guides/measure-agent-improvement/#skip): JSON pointer `/skip`; Markdown section “Skip this when”. - [Work through the checks](https://www.aomsites.com/guides/measure-agent-improvement/#checks): JSON pointer `/steps`; Markdown section “Checks”. - [What would count as success?](https://www.aomsites.com/guides/measure-agent-improvement/#accept): JSON pointer `/acceptance`; Markdown section “Acceptance”. - [Evidence to bring back](https://www.aomsites.com/guides/measure-agent-improvement/#evidence): JSON pointer `/evidence_plan`; Markdown section “Evidence to bring back”. - [Stop or recover](https://www.aomsites.com/guides/measure-agent-improvement/#stop): JSON pointer `/stop`; Markdown section “Stop or recover”. - [What this does not establish](https://www.aomsites.com/guides/measure-agent-improvement/#limits): JSON pointer `/limits`; Markdown section “Limits”. - [Reference links](https://www.aomsites.com/guides/measure-agent-improvement/#sources): JSON pointer `/sources`; Markdown section “Reference links”. - [Use this guide with your own evidence](https://www.aomsites.com/guides/measure-agent-improvement/#record): JSON pointer `/decision_record`; Markdown section “Start this guide’s record”. - [If another problem remains](https://www.aomsites.com/guides/measure-agent-improvement/#next-reads): JSON pointer `/reading_paths`; Markdown section “If another problem remains”.