Many agents are tested straight on real customers with full permissions, and the first serious mistake ends the project. Others are tested so little that there is never evidence to decide.
Who it is for
Small businesses that have already decided to test an agent or an AI automation (their own or a vendor's) and want to do it in two to four weeks with clear rules.
The pilot has three phases: test data, shadow (the agent proposes, a person decides) and limited production. You only move to the next one when the success criterion you wrote before starting is met.
The protocol, step by step
-
Pick a frequent, bounded, reversible case
Frequent so you get evidence fast, bounded so you know what is in and out, reversible so a mistake can be undone. Good example: classifying incoming requests. Bad example: approving discounts.
-
Write success, failure and when it stops
Before turning anything on, fill in this sheet and share it with the team.
Copy and pasteCase: ... Success: the agent gets [X] out of every 10 reviewed cases right for [N] days. Failure: [what counts as a serious error]. It stops if: a serious error happens, or [condition]. Pilot owner: ... Review date: ...
-
Phase 1: test data
Build a set of 20 to 30 real past cases, with sensitive data removed, along with the correct answer your team gave. Run them through the agent and compare. Do not touch production until this phase passes.
Copy and pasteCompare the agent's answer with the correct answer in each case: [PASTE TABLE]. Tell me how many match, which ones fail, what pattern the errors have and whether any would be serious under this definition: [PASTE FAILURE].
-
Phase 2: shadow mode
The agent works on real cases but only proposes; a person decides and executes. Log in each case whether the proposal was correct. This compares the agent against your current human process without risk.
-
Compare against the human process
At the end of phase 2 look side by side: time per case, errors, cases the agent could not solve. If it does not beat the human process on what you defined as success, the pilot is over, and that is also a good outcome.
-
Phase 3: limited production
Let it execute only in a bounded part: one case type, one time window or a share of the cases, with a daily cap and a stop button. Irreversible actions stay under human approval. Review a sample every week.
-
Expand permissions only with stable evidence
Expand one step at a time and only when the success criterion has been met consistently in the previous phase. If a serious error appears, go back one phase.
Common mistakes
- Defining success after seeing the results. Write it before turning anything on.
- Skipping shadow mode because the demo went well.
- Testing with made-up cases. Use real past cases from your business.
- Expanding all permissions at once when the pilot goes well.
Who this is NOT for
If you still do not know whether your process needs an agent, start with the FILTER and DECISION guides in the Vault. This protocol is for when you have already chosen what to test.
Want it applied to your business?
If you would rather not build it alone, book 30 minutes with me and we will review it with your real case. If you still do not know where to start, the map tells you what to automate first in 5 questions.