Agent Workflow Comparison Lab · $12 one-time
Find the workflow worth keeping.
Compare completion, speed, operator effort and recorded cost on the same tasks. Turn your trial observations into a clear report for your next workflow decision.
Compare the choices that matter
Compare two prompts, models or tool workflows on the same tasks. Use your own observations to decide which approach to keep, adjust or test next.
Three steps. Three useful files.
- Plan the trial → plan.json. Define tasks and acceptance criteria. Generate a paired schedule with a repeat count and seed.
- Record the work → trials.jsonl. Create an observation template tied to the plan, then fill it with your workflow results.
- Compare results → report.md. Generate overall and per-task results with matched measurement differences and sample counts. JSON and CSV exports are also included.
See the included example commands
Run from the extracted pack folder. The final command reads the included completed example so you can inspect the report immediately.
python3 tools/workflow_lab.py plan examples/spec.json --repeats 3 --seed 42 > my-plan.json
python3 tools/workflow_lab.py template my-plan.json > my-trials.jsonl
python3 tools/workflow_lab.py analyze examples/plan.json examples/trials.jsonl --format markdownA decision-ready comparison report
See completion coverage for each workflow, then compare matched runs by elapsed time, operator minutes and recorded cost. Review results by task or export the paired rows for your own analysis.
Included report structure
Workflow comparison report
- Completion coverage: passed trials, observed trials and scheduled runs for each workflow.
- Paired differences: mean and median changes in elapsed time, operator minutes and cost, with sample counts.
- Task breakdown: matched results for each task and repeat.
Markdown for the decision. JSON for your agent. CSV for further analysis.
Inside the download
Seven reusable skills, decision-protocol templates, per-task comparison diagnostics, a local experiment starter, a Python CLI, a worked two-task experiment, ready-to-fill observation files, expected reports, an experiment-design guide and automated tests.
Four advanced skills, included
Plan a bounded skill search, compare selected gains, account for repeated context exposure and assemble a held-out revision recommendation. The Skill Search & Context module includes its own local tools, examples and current source-linked research.
Use your own workflow
Run your workflows, record the observations, then let the local analyzer organize the comparison. Python 3.10 or later, standard library only. Tested on Linux and macOS. Commercial use license.
Your workflow · a clear comparison
Compare speed, cost and completion together.
Turn your trial records into one clear comparison. The report puts completion coverage beside matched time, effort and cost, so you can compare the full picture.
Your next step: use the report to choose what to keep, adjust or test next in your own workflow.
Read the Markdown summary, give the JSON to your agent, or explore paired rows in CSV.
From trial records to a decision
- Completion: passed, observed and scheduled trials for each workflow.
- Time and effort: matched elapsed time and operator minutes.
- Recorded cost: paired differences with sample counts.
- Task detail: inspect each task and repeat before choosing your next step.
Your output: a readable Markdown report, structured JSON and paired-row CSV.
You run the workflows and supply the observations; the Lab organizes the comparison.
Connect trials with their evidence
The included Fieldwork evidence join checks supplied receipts against a saved trial plan, outcomes, metrics and local artifact bytes. Get a source-linked consistency report instead of comparing the records by hand.
Use the module’s explicit JSON contract profile. The checker dependencies are included, so no second purchase or network access is needed. It checks supplied-record consistency, not execution authenticity or output quality.
Do I also need the Fieldwork Kit?
No. The Lab includes its own trial planner, observation template and analyzer. Choose it for repeated, matched workflow experiments. The $12 Fieldwork Kit is for checking selected deliverable files, keeping the evidence in a receipt and generating a handoff memo; it also compares two compatible receipts.
Keep your toolkit and your results
The ZIP includes the local starter, planner, analyzer and seven reusable skills. Run them with Python 3.10 or later. Your observations and reports stay in your files; no hosted account or API key is required to use the tools.
At checkout, keep cookies in the same browser or agent cookie jar. Your paid download is recoverable there for 30 days from starting checkout. Save the ZIP to use on your own devices.
License and purchase terms
Commercial license for your own and your organization’s work, including AI-agent and client projects. Share your own outputs; do not redistribute or resell the pack. Perpetual use of this edition. Prior MIT grants remain unchanged. See the full license for terms.
Purchases are sold through Link. For non-delivery, goods that are defective, unfit for purpose or substantially misdescribed, or unauthorized or duplicate payments, request a refund through Link within 60 days. Link reviews eligibility case by case. Change-of-mind refunds are excluded except where consumer rights apply. EU and UK consumers have a 14-day cancellation period from delivery or access. Your statutory rights remain unchanged.
One-time digital download
Put your next workflow choice to the test.
Plan matched trials, record your observations and compare the results in your own files.
$12.00 USD plus applicable tax, shown at checkout. No recurring charge.
Checkout opens Stripe. Review the final total before payment. License and purchase terms
Link refund policy and request support · Sold through Link terms