How to scope your first AI agent pilot

Start with the decision, not the technology

Teams evaluating an AI agent pilot usually start by asking what the agent could do. That is the wrong first question. The right first question is: what decision or handoff, if it happened faster or more consistently, would actually change an outcome someone cares about? A pilot exists to answer one question with evidence: does this approach work here, with our data, our systems, and our people. If you cannot name the decision it informs, the pilot will produce activity without producing an answer.

A useful test is to write down, in one sentence, what you will do differently if the pilot succeeds and what you will do differently if it fails. If both answers are "keep evaluating," you have not scoped a pilot, you have scoped an open-ended project. Pilots need an exit.

Pick a task with these five properties

Not every process is a good candidate for a first pilot. The best first tasks share a specific set of properties. Looking for all five at once filters out most of the tasks that sound exciting but end up stalling.

• Bounded input and output: the task takes a well-defined input (an email, a document, a form submission) and produces a well-defined output (a field filled in, a record updated, a draft written). Open-ended judgment calls are a second-pilot problem, not a first one.
• High frequency, low individual stakes: dozens or hundreds of repetitions per week, where a single mistake is inconvenient but not damaging. This gives you enough volume to see patterns within weeks, and enough safety margin to tolerate early errors.
• A human is already checking the output: if someone currently reviews or approves this work, the agent slots into an existing review step instead of requiring you to build a new safety net from scratch.
• Clear correctness: someone can look at the output and say right or wrong without a debate. Ambiguous judgment calls make it impossible to measure the pilot honestly.
• Existing data access: the information the agent needs already lives somewhere you can connect to, without a multi-month data project first.

Define the boundary before you define the workflow

Before mapping out steps, decide what the agent is not allowed to do. This is the permissions boundary, and it should be written down before any implementation starts, not retrofitted afterward. At minimum, decide whether the agent can read data, propose changes, or make changes directly; which systems it can touch; and who is the named human owner accountable for what it does.

For a first pilot, the safest default is read and propose, not act. The agent drafts, flags, or recommends, and a person approves. This slows the agent down slightly but removes an entire category of risk, and it is much easier to loosen a boundary later than to explain an unintended action after the fact.

Set success criteria you can measure in three to six weeks

A pilot without a deadline becomes a permanent experiment. Set a window, typically three to six weeks depending on task frequency, and define in advance what "worked" means. Useful criteria are usually a mix of accuracy (how often was the output correct or acceptable without edits), time saved per instance, and how much the review burden changed for the human in the loop.

Avoid vague criteria like "the team liked it." Ask instead: out of the last fifty instances, how many needed no correction, a minor correction, or a rejection? That single table tells you more than a satisfaction survey.

Common mistakes that stall pilots

A handful of patterns show up repeatedly in pilots that never reach a clear conclusion.

• Picking a task that is actually several tasks bundled together, so failure in one part is blamed on the whole approach.
• Skipping the boundary conversation, so the pilot gets stuck waiting on a security or compliance review that should have happened on day one.
• No named owner, so when questions come up mid-pilot, nobody has the authority to answer them and the pilot drifts.
• Measuring nothing until the end, so there is no way to tell whether week one and week four looked different.
• Treating the pilot as a demo instead of a test, and declaring success based on how impressive the first output looked rather than how consistent the fiftieth output was.

A simple pilot scoping checklist

Before you start building anything, you should be able to answer all of the following in a single page.

• What is the exact task, including a specific example input and output?
• How often does this task happen, and where does the data currently live?
• What can the agent do on its own, and what requires human approval?
• Who is the named owner if something goes wrong?
• What does the review step look like, and who currently does it?
• What is the measurement window, and what counts as success?
• What happens after the pilot: expand, adjust, or stop?