How to Run a 1-Week AI Agent Pilot That Actually Ships to Production

Quick answer: A 1-week pilot ships when scope is reduced to one workflow, success criteria are quantitative and pre-agreed, the eval set is written before any prompt, and production-readiness work (observability, guardrails, on-call) is included in week one rather than punted to "phase 2." Skip any of those and the pilot becomes a 6-month research project.

The 5-day shape that ships

Day 1: Scope (afternoon)

Pick exactly one workflow. Document its current shape: input format, decision points, output format, edge cases, current owner, current cycle time, current cost-to-serve. Get a one-line quantitative success criterion ("resolve at least 70% of inbound support tickets without escalation") signed by the operating function — not the innovation team.

Day 1-2: Eval set

Build 50-200 examples from real production data. Include common cases (60%), edge cases you've seen in real data (30%), and adversarial examples (10%). Score the eval set with the current human baseline first — that's your floor. Anything the agent ships at must beat that floor on the eval set.

Day 2-3: First version

Build the simplest agent that could plausibly work. One model (Claude Sonnet or GPT-4.5 mini), one or two tools, no fancy retrieval if you can avoid it. Run against the eval set. Iterate prompt and tool definitions. Stop when eval pass rate exceeds the human baseline by a meaningful margin.

Day 3-4: Production wrapper

Wire up: per-input token budget cap, structured logging (every prompt + completion + tool call), output validation (Zod schema, plus an LLM judge for soft validation), error handling and retries with backoff, fallback to human escalation when confidence is low, and a kill switch (env var that turns the agent off without redeploy).

Day 4-5: Production deploy + handoff

Ship to production in shadow mode first (agent runs alongside humans, doesn't affect outcomes). Compare agent decisions against human decisions for 24-48 hours. If shadow mode looks good, flip to live mode for a small percentage of traffic. Document the runbook. Hand off the on-call rotation.

The non-negotiables

1. One workflow. Two workflows means six weeks, not one.
2. Operating-function sponsor. The team that owns the workflow has to have the budget for production. Nobody else can buy production-readiness.
3. Eval set before prompt. Without an eval set you cannot iterate, regression-test, or argue for production.
4. Production-readiness in week one. Token budget, logging, validation, kill switch. None are optional.
5. Shadow mode before live mode. Always.

What we cut

To make 5 days work, we cut: pretty UIs, slick demos, custom dashboards, exotic LLM features (fine-tuning, advanced RAG), and any "phase 2" work. The pilot is the agent — not the wrapper around the agent.

Need help running a pilot? Our AI Kickstart is exactly this playbook, executed by us, embedded with your team. Or book a $425 strategy hour if you want a second opinion on your pilot scope before you start.