The AI Rollout Measurement Playbook
Works with
Implementation kit
- The one number charter, with the check that it is a result and not usage
- The four-week baseline log
- The rollout rule, signed before launch
- The weekly log and the day 30 readout
- A worked example, from charter to decision
You rolled out an AI tool, people use it, and nobody can say what changed. This playbook names the one number the rollout is for, measures it before launch, and lets a rule signed in advance decide at day 30 whether to keep it, fix it or kill it.
The full method is below. One number, four weeks of baseline, a signed rule, and one readout.
A rollout judged by usage reads as done the day it ships, whether the number moved or not.
The failure this playbook fixes
How it works
01Name the one number the rollout is for
One number, one owner, written before launch. Pick the result the owner of the number already answers for, such as second meetings per week or the time from an inbound lead to the first touch.
Write exactly what counts and where it is read from, so two people get the same count. Track one usage number beside it, so the readout can tell "not used" from "used and changed nothing".
Watch for a usage number in disguise. Logins, drafts generated and seats activated rise when people use the tool, whether the result moves or not.
02Measure four weeks before
Read the number the same way for the four weeks before launch. Keep the lowest and the highest week.
The gap between them is how much the number swings with no tool at all. Without it, a normal good week reads as the tool working.
03Write what counts as moved, and sign it
The line sits past the baseline's swing, in the direction the number should go. Write the read date, launch plus 30 days, and a minimum count below which the read is too thin to decide.
Write the keep, fix or kill rule now, while nobody has a stake in the result. The owner of the number and the owner of the rollout both sign before launch.
Watch for a line inside the normal swing. A good week then reads as the tool working.
04Run it beside a holdout
A holdout is the strongest comparison. One team, segment or region does the same work without the tool for the same 30 days.
With only one team, compare it against its own baseline. Either way, list anything else planned that could move the number, such as a campaign, hiring or a price change.
05Log it every week
One row a week, about five minutes. The number, the holdout's number, usage, whether it is past the line, and anything on the "what else" list that happened.
06Read it on day 30 against the signed rule
The rule was written before anyone knew the result, so the readout applies it and does not reopen it.
Past the line while the holdout is not means keep. Past the line with the holdout moving too means hold and run 30 more days with a cleaner comparison. Used with the number flat means fix, one change. Barely used means fix adoption or kill. Below the minimum count means keep recording.
07Record the decision and start the next charter
Both owners sign the readout. A kept tool names the next number it should move and gets a new charter.
One moved number earns the next test, never a pass for life.
Name the number before the tool, and let the number decide.
The implementation kit above packages the charter, the baseline log, the signed rule, the weekly log, the day 30 readout and a worked example.
