An AI workflow can appear inexpensive during a demo and become costly when it handles real traffic, retries, longer inputs and human corrections. To decide whether an automation is worthwhile, track cost and quality per successfully completed task—not just token price.
This guide builds a simple measurement framework for AI-powered email triage, document extraction, and support drafting.
1. Define one completed task
Before collecting numbers, decide what counts as success. For example:
- Email triage: one inbound email correctly assigned to the intended review queue.
- Invoice extraction: one invoice approved with accurate fields and no duplicate import.
- Support draft: one draft accepted by a human without unsupported claims.
A model response isn't necessarily a completed task. If a reviewer has to fix its output, include the correction time and error in your measurement.
2. Record the important fields
A lightweight event log can include:
| Field | Example | Notes |
|---|---|---|
task_id |
task-102 | Unique identifier |
workflow_version |
3 | Helps compare changes |
model_id |
provider/model-name | Store actual value |
input_tokens |
1,400 | Use provider-reported usage when available |
output_tokens |
250 | Record separately |
duration_ms |
2,800 | End-to-end wall time |
review_outcome |
corrected | Approved / corrected / rejected |
total_cost_usd |
0.006 | Include relevant external charges |
The values here are synthetic. Different AI providers may report or bill cached tokens, tool calls, images, and other usage differently. Always verify the current provider-specific pricing and usage fields rather than hardcoding an assumed rate.
3. Calculate cost per successful task
Suppose a test batch contains 100 tasks. The model usage costs $2.00 and infrastructure overhead allocated to that batch costs $1.00. Human review takes 90 minutes at an illustrative labor cost of $24 per hour.
The total batch cost is:
Model usage $2.00
Allocated infrastructure $1.00
Review: 1.5 hours × $24 $36.00
------------------------------------------
Total $39.00
If 90 tasks were approved as correct, cost per approved task is $39 / 90 = about $0.43. This is merely a calculation example and not a market cost estimate. Change the rates to reflect your own process.
Notice that review time dominates the example. Optimizing token usage alone would have little effect until the number of corrections or time spent reviewing falls.
4. Measure accuracy with a reference set
Create an approved evaluation set before changing prompts or models. For classification, track confusion between categories. For extraction, track correctness by field. For support drafting, use a human rubric including factual grounding, policy compliance and clarity.
Do not use only a model to grade its own output. A second model can help prioritize review, but important acceptance criteria need deterministic checks or qualified human judgment.
Keep a sample of deliberately difficult inputs: multilingual text, vague messages, oversized attachments, malformed data, and adversarial instructions. These cases often reveal more than ordinary success examples.
5. Set budgets and safe fallbacks
Place limits on the number of AI steps, model retries and allowed tool calls per task. When the budget is exhausted, route the task to a manual queue rather than silently discarding it or running endlessly.
Track a weekly dashboard containing successful tasks, correction rate, median and high-percentile latency, total cost, and cost per approved task. Segment by workflow version so a change in model or prompt doesn't hide a regression.
6. Test changes before rollout
Change one significant variable at a time: model, prompt, chunk size, or extraction schema. Run both old and new versions on the same approved test set. Compare quality, response time and total cost.
A cheaper model is not automatically cheaper in operations if reviewers spend more time fixing its mistakes. Similarly, an expensive model is not automatically the right choice if a simple rule-based parser solves the task.
7. Privacy and reliability are part of cost
Incidents, exposure of private information, and incorrect customer messages have costs that don't appear in a model provider's invoice. Apply access controls, redact inputs, retain useful logs without copying secrets, and establish a human escalation path.
When estimating return on investment, include the time spent developing the workflow, maintaining integrations, responding to outages, and reviewing errors. Avoid promising guaranteed savings before running a pilot.
8. A small-business scorecard
After two weeks, answer five questions:
- How many tasks reached the correct final state?
- How often did people need to correct the AI output?
- What did each approved task cost in total?
- How long did the process take compared with the manual baseline?
- Did any error create a customer or privacy risk?
If the pilot cannot answer these questions, expand measurement before expanding automation.
Further reading
The takeaway
Measure the whole workflow, including humans. The right optimization target is a reliable, accurate, affordable completed task—not simply the lowest token price.
Keep the good stuff moving.
One useful workflow at a time. Explore another guide or share this one with someone who needs it.
Explore more guides