How to scope a 30-day AI agent pilot for ops teams
Most AI pilots are scoped like research projects: ambiguous goals, nine months on the calendar, three teams orbiting a vague KPI. The reason 30 days has become our default is that the constraint forces good decisions early.
Here is the playbook we use to scope a pilot that actually clears the bar.
Week 0, pick the right workflow
Before kickoff, write the workflow on one page. Inputs, outputs, who owns each decision, and the volume per week. If the page does not exist, the workflow is not ready. Pick something with high volume, defined inputs, and a reviewable output. Tier-1 ticket triage, vendor onboarding, claims review, and lead enrichment all qualify; we wrote about choosing your first workflow in more detail.
Avoid workflows where the output is a free-form artifact (a strategy memo, a customer email that is the whole product). Save those for v2.
Week 1, connect and map
Goal: agent reads the same data your ops team reads. Connect the system of record (Zendesk, Salesforce, NetSuite, your warehouse). Map the top intents or decision types. Define the confidence threshold and the dollar / risk caps that route to a supervisor.
Deliverable: a shadow-mode agent that produces drafts on real traffic. Nothing ships yet. Supervisors review every output and label it accept / edit / reject.
Week 2, calibrate
The labels from week 1 are the calibration set. Tune the confidence threshold so auto-resolution above the threshold is at or above 95% supervisor agreement. Below the threshold, build the routing pack so the supervisor's review time is under two minutes per case.
Deliverable: a calibrated agent with documented thresholds and a supervisor playbook.
Week 3, go live on routine intents
Auto-resolve the high-confidence intents. Keep ambiguous ones routed. Watch the metrics daily, auto-resolution rate, supervisor agreement, escalation latency. Expect the first surprise: a sub-segment where accuracy drops. Tighten the threshold, do not retrain yet.
Deliverable: the agent live on production traffic for one segment, with a daily metric review.
Week 4, prove the curve
By the end of week 4, you should be able to plot the accuracy curve over the full 30 days. The curve should be going up, that is the part autonomous agents cannot do. If it is flat, you are missing the eval harness, not a better model.
Deliverable: a curve, a documented playbook, and a decision on whether to expand or cut.
What to commit to before kickoff
- One named workflow.
- One named ops leader on the hook for outcomes.
- One reviewer queue you control.
- A clear no-go criterion. If accuracy does not move from baseline by day 30, you cut.
A pilot scoped this way takes the politics out of the rollout. You either ship at day 30 or you have a clean reason not to. Both are valuable.