Compare two AI models on the work you actually do
A practical evaluation worksheet for comparing correctness, review effort and cost without relying on a leaderboard.
What you will take away
- Choose one repeatable task and write the acceptance criteria before seeing outputs.
- Use the same inputs and permissions for both models; record failures as well as successes.
- Separate output quality from human review effort and total workflow cost.
Start with a decision, not a model name
Suppose your team wants help turning support requests into draft replies. The useful question is whether a draft is correct, complete and safe to send after review. A fluent response can still invent a refund policy or omit the customer’s actual issue. Write down what would make a draft acceptable before you compare products. This guide proposes a method; it does not report a completed benchmark or name a winning model.
Build a small, representative test set
Create fictional or appropriately de-identified examples covering routine requests, incomplete information, contradictory instructions and cases needing escalation. Keep an answer key listing the facts that must appear and claims that must not appear. Do not select only easy examples or rewrite every input until one model succeeds. Reserve some examples for a final check after prompt adjustments, so you are not judging only the cases used to tune the workflow.
Keep the conditions comparable
Give each candidate the same approved source material, task instructions and allowed tools. Record the exact model version, date and settings. If one system gets web search and the other does not, the result describes those different configurations rather than model quality alone. Run the comparison in a draft-only environment. A support evaluation should not send messages to actual customers or change their accounts.
Use a review sheet you can explain
For each output, record whether it follows the supplied policy, addresses the request, admits missing information and avoids unsupported statements. Separately note the corrections a reviewer must make. A wrong but polished answer must not score higher merely because it reads well. If reviewers disagree, inspect the acceptance criteria and the evidence; do not hide the disagreement inside an average.
Make failures part of the result
Test how the workflow handles unavailable tools, ambiguous instructions and requests outside its scope. An appropriate escalation can be better than a confident answer. Repeat important cases if outputs vary, and keep the failed attempts. Record usage charges and review time from the actual trial rather than estimating savings from a vendor’s advertised throughput. Never insert invented numbers into the results sheet.
Decide on a limited next step
Pick a supervised pilot only if the recorded results meet the criteria for that specific task. A model that drafts replies well has not thereby proved it can diagnose patients, approve refunds or modify production code. Revisit the evaluation after changing the prompt, source documents, tools or model version. Your deliverable is a documented decision with known failure modes, not a universal ranking.