🧭 AI pilot to production: the workflow walkthrough


Nov 15, 2026 | Read time: 11 mins.

Kevin Donovan · Architects' Hub
​
Forwarded this email? Subscribe to Architects’ Hub for the next issue.

Tuesday, 10:40 am.

Hey ,
​
Last Sunday we left a hypothetical customer-service assistant with a commitment record and unanswered questions about capacity, data and support.

This week, move the story forward 3 weeks. The team has agreed to a bounded trial on 1 queue. Agents still review every draft. Wider rollout is waiting.

I want to spend this edition with the person on that queue. She has a customer waiting, a plausible answer on her screen and about 90 seconds to decide whether to trust it.

In this edition

  • A 4-second draft and the work its timing leaves out
  • A worked walkthrough and capacity table for the next rollout review
  • 3 practical tests, plus a blank walkthrough for your team
  • Community Corner: Thanks to Abhishek Kumar on designing for the work AI leaves behind -and everyone who chimed in

Let’s get into it.👇

________

🧭

________

​

The case the pilot never met

Picture the customer-service team from last week. The assistant pulls the order, reads the shipment status, and drafts a reply to “where is my order?” The earlier trial produced promising drafts on routine cases.

Tuesday, 10:40. A customer asks about an order. The assistant reads the record and writes, in 4 seconds: “Good news, your order was delivered on Friday.”

The order was split. Shipment 1 did arrive Friday. Shipment 2 has sat at a carrier hub for 6 days. The order system still shows the status from the day it shipped, because the carrier feed updates nightly and failed twice this week.

The customer service rep has 31 cases in her queue. She can send it, fix it (6 minutes in the carrier portal), or hand it to the 1 colleague who already fixes split shipments by hand and was at 110% before the assistant arrived.

During the earlier trial, cases like this were excluded from scoring. An engineer sat beside the support team. Odd cases got pulled aside and “didn’t count.” Today the engineer is on another project and every case counts.

All the figures here belong to our fictional case. The pattern is familiar from work that predates AI: The reporting tool that was right until the source table lagged. The integration that passed UAT on hand-cleaned data. The new part works. The old parts around it were never tested with the new part running.

The pilot measured the model. Tuesday runs a workflow.

Model performance vs workflow performance

Suppose the trial report says reps accepted 92% of drafts with light edits. That’s useful evidence about the included cases. It doesn’t establish that every accepted answer was correct, or say anything about the cases the engineer removed.

The remaining 8% needed more work. Split shipments excluded before scoring sit outside both percentages. Keep those groups separate.

I’d ask for 2 views of performance.

The AI step: how quickly does it draft, and how well does the draft represent the information it receives?

The workflow: how long does the customer wait for a correct answer, and how much work do people do along the way?

An acceptance rate belongs in the first view, alongside an independent quality check. Agent effort, waiting at handoffs, corrections and customer follow-ups belong in the second.

If the report claims “20 minutes saved per case,” ask which cases were compared and where the timing ended. A saving on routine reply preparation can be real. It still can’t tell us how many people the whole queue needs.

For the split shipment, keep 2 clocks. One measures elapsed time from the customer’s question to a correct answer. The other adds up the minutes people spend checking, fixing and recovering. A handoff can add little staff effort and a long customer wait. Both matter.

The 5 jobs around every AI step

For this assistant, I would put 5 jobs beside the AI step. During the trial, the build team helped absorb several of them. Wider use needs an explicit arrangement for each.

  1. Verify. Someone checks the output before a customer sees it. Pilot: a customer service rep with an engineer beside her. Month 3: the same rep, who now trusts the drafts and has 31 cases waiting.
  2. Hand off. Someone routes the cases the assistant shouldn’t touch. Pilot: “those don’t count.” Production: they do.
  3. Fix. Someone corrects the wrong draft and the stale record behind it. Today: 1 person, by hand.
  4. Recover. Someone handles the fallout when a wrong draft went out. The call back, the credit, the complaint.
  5. Support. Someone answers when the carrier feed fails at 09:00 and every draft that morning is wrong. Pilot: a chat channel with 3 engineers. Production: a ticket queue with a 2-day SLA.

Start with the people doing those jobs today. Ask how much time they have available, then estimate what the proposed volume would require.

Here’s an illustrative first pass. 10x means case volume: from 200 to 2,000 cases a week, distributed across 6 teams.

The fix estimate assumes 8% of all incoming cases need a separate human fix, at 6 minutes each:

2,000 × 8% × 6 minutes ÷ 60 = 16 hours a week.

That 8% is a planning assumption for this calculation. We haven’t derived it from the draft acceptance rate. A rep can accept an incorrect draft, and a draft needing edits might never reach the exceptions handler.

The other rows need their own evidence. Support demand may depend on incidents rather than case volume. Recovery depends on how often mistakes reach customers and what it takes to repair them. Verification still has a cost even when people absorb it into their day.

So 28 hours is a partial estimate, with guesses marked.

The first conversation I’d have is with the exceptions handler. Does she have 16 hours available? What comes off her existing queue? If her answer is no, the choices include narrower scope, fewer exceptions, more capacity or a slower rollout.

That’s the work I’d do first in a real engagement: follow the exception to the person carrying it, check their capacity and bring the resulting choice to someone who can fund or change it.

________

🧭

________

​

The walkthrough, filled in

Take 1 routine case and 1 exception to the person who runs the queue. Trace each from the customer’s question to the correct outcome. At each step, separate:

  • Known: what the evidence supports, including the conditions under which it was observed.
  • Assumed: what the rollout relies on without adequate evidence.
  • Untested: the specific behavior or question we still need to examine.

An assumption may lead to an untested question. These columns organize the conversation; adding up their entries doesn’t produce a readiness score.

Normal case: 1 shipment, current record

1. The message reaches the queue.

Known: the trial routed its 200 weekly cases to the right team. Assumed: the proposed routing rules work for additional teams. Untested: their order types and peak arrival patterns.

2. The assistant reads the records.

Known: the lookup returned promptly on the checked cases. Assumed: the order status remains consistent with the carrier. Untested: what happens when the feed is late or records disagree.

3. The assistant drafts a reply.

Known: reps accepted 92% of included trial drafts with light edits. Assumed: acceptance reflects correctness and carries across to the next teams. Untested: an independent quality check and performance on their customer mix.

4. The rep reviews and sends.

Known: review happened with build-team help available. Assumed: the same problems will be caught during an ordinary shift. Untested: catch rate under realistic queue pressure without an engineer pointing out the problem.

5. The customer gets an answer.

Known: the reply was sent. Assumed: it resolved the question. Untested: whether the customer had to write back, and the total time to a correct answer compared with the old process.

Exception: 2 shipments, stale record

1. The assistant reads the order.

Known: the order can have more than 1 shipment. Assumed: the summary status tells the story for every parcel. Untested: how consistently the assistant uses shipment-level evidence when statuses differ.

2. It writes “delivered.”

Known: this particular case produced a confident, incomplete answer. Assumed: it’s rare enough for routine review to catch. Untested: frequency across representative split shipments. One observed failure tells us what can happen, not how often.

3. The rep checks or hands off.

Known: someone can investigate in the carrier portal. Assumed: they have time, access and cover on every shift. Untested: effort per investigation, waiting time and additional load at the proposed volume.

4. Someone corrects the cause.

Known: the customer reply can be fixed manually. Assumed: someone also investigates the stale data. Untested: who receives the feed-failure alert, how they restore service and how the team knows records are current again.

5. The case closes.

Known: the first draft couldn’t safely answer this customer’s question. Assumed: after correction, the case is finished. Untested: repeat contacts, recovery work and whether the next draft repeats the error.

I’d look closely at those last 2 steps. A rep can rescue the customer interaction while the same fault keeps producing tomorrow’s cases. A walkthrough that ends at “human fixed it” would miss that.

The 3 tests still owed before wider use

Pick the questions that could change the scope or capacity decision. Agree acceptance criteria with the service owner before running the tests. The sample sizes and timeboxes below are starting proposals for our example, not universal proof of readiness.

1. Can the assistant recognize the split shipment?

The exceptions handler and delivery engineer review 50 permitted historical cases, covering different status combinations and stale records. Record the draft, the supported answer and any unsupported delivery claim.

A proposed stop rule: if it asserts delivery without evidence for every parcel, keep that case type out of automated drafting until the failure is understood. A clean sample supports the next decision; it doesn’t establish a zero error rate.

2. Will the rep catch a bad draft?

The team lead runs a controlled rehearsal with realistic queue pressure and a mixture of correct and deliberately flawed drafts. Use replayed or synthetic cases in an environment that cannot send to customers. Tell participants it’s an evaluation exercise, but don’t flag which drafts contain errors.

Record missed errors and review time. Agree in advance which errors would block expansion. If people miss a second parcel, try a shipment-count signal and repeat the exercise. Keep necessary review and support in the operating trial.

3. Can the team carry the exceptions?

For 2 weeks of the bounded trial, record every exception, the people involved, their active minutes and time spent waiting. Compare the case mix with the proposed next teams before projecting volume.

Ask the manager how many hours the handler can commit after existing duties. If projected demand exceeds that capacity, expansion needs an accepted change: additional cover, smaller scope or fewer cases reaching that handler. Record who funds it and when it becomes available.

These tests consume people’s time. Put that time on the plan too. If the handler can’t take part this week, the review date should reflect that.

Your blank walkthrough

Book 60 minutes with the operating owner and someone who handles exceptions. Bring a routine case -and a difficult one.

Copy this block for each step:

• Step / intended outcome:

• Known, with evidence and date:

• Assumed by the rollout plan:

• Untested question:

• Who does the work / active minutes / waiting time:

• Next check / owner / agreed date:

• Result that would change the decision:

Use your first 10 minutes to choose the cases and define what “resolved” means. Spend 30 minutes tracing them, then 10 on who carries the surrounding work. Use the last 10 to choose the checks that matter most.

You want to leave with a short list of decisions and tests. Assigning an owner still needs that person’s acceptance.

The yes worth having

Here’s what I want the service lead to be able to say at the next review:

I’m prepared to accept the next team once we’ve checked the split-shipment behavior, rehearsed review without build-team prompts and agreed capacity for the projected exceptions. The feed has a named support owner and a tested manual fallback. We review after 2 weeks. If unsupported delivery claims escape review, or the exception queue exceeds the agreed limit, we pause expansion and return the affected cases to the manual process.

The service lead, sponsor and support owner still need to agree the scope, limits and stop authority. Writing that sentence helps find out whether they mean the same thing by yes.

________

🧭

________

​

Community Corner

​Abhishek Kumar pushed the 5-job discussion back to the design stage. Exception handling needs to be part of the agent architecture from the beginning. As agents take on more work, the exceptions may grow too. His test for automation includes whether the org can sustainably handle the work the agent leaves behind.

​Jordan Corn warned against stopping measurement at the handoff. You can improve the first step and still leave the customer waiting through the rest. In our discussion, he connected that to theory of constraints. For the walkthrough above, I’d keep the clock running through the exceptions queue to the correct answer.

​Pranil Shinde connected that measurement to the investment decision. Strong model results can hide data dependencies, handoffs and exceptions. Making that work visible, measurable and accountable gives leaders a better basis for deciding whether to scale.

________

🧭

________

​

Where to start Monday

Book the hour. Bring 2 cases.

At the first unmeasured handoff, ask who does the work and what they have to wait for. That is a useful place to start.

Next Sunday: the system can act. Which decisions may it make? Someone in our hypothetical review is about to propose letting the assistant issue refunds.

When the walkthrough needs more than an hour

When the model performs and the workflow around it still depends on 1 person fixing exceptions by hand, wider use is a bigger commitment than it looks.

Examining that gap before anyone signs is a large part of what I do in the Compass AI Architecture and Adoption Diagnostic: the use case, its systems and data, the handoffs, and who operates each piece on a Tuesday.

If that decision is yours, reply with the workflow and the 1 exception your team can’t yet handle. I’ll tell you what I’d look at first, and whether it needs 4 weeks or an hour. Two lines is enough.

If you’re the architect being asked to make the pilot real, the known/assumed/untested table is yours to take into the next review. Forward this to the person who has to sign the wider rollout.

The pilot measured the model. Your team will run the workflow.

Until next Sunday,

Kevin


Work with me

I run Compass Architecture Advisory. I help enterprise and technology leaders turn AI ambition and architecture into decisions they can fund, govern and run. Right now that’s a 4-week AI Architecture and Adoption Diagnostic. If there’s a decision like that in front of you, reply to this email with 2 lines about it and I’ll tell you whether it fits.

________

________

Kevin Donovan​
I write Architects' Hub every Sunday for senior architects and technology leaders.
You're getting this because you subscribed at architects-hub.kit.com.
Know someone who’d find this useful? Forward it to them.

​

Korsvejen 9, Viborg, Viborg 8800
​Unsubscribe · Preferences​

Architects' Hub

A Sunday newsletter for senior and enterprise architects. One decision, one practical model, one tool you can use Monday. Written by Kevin Donovan. Join 2,500+ architects. One email every Sunday.

Read more from Architects' Hub
Three doors slightly ajar in a row. Through them, a telephone, a stack of documents with a magnifying glass, and an arrow curving back on itself. A single coin rests on the floor in front of the doors.

Nov. 22 2026 | Read time: 11 mins. Kevin Donovan · Architects' HubForwarded this email? Subscribe to Architects’ Hub for the next issue. The room went quiet Hey , Two Sundays ago, our hypothetical pilot case got its commitment record. Last Sunday it got a walkthrough, and the operating owner signed with 3 tests on record. The AI assistant now drafts delayed-order replies for 3 teams, and the drafts are good. This week someone proposed letting it do more than just draft… In this edition The...

A working terminal with four loose connections labeled Funding, Operations, Data and Authority.

November 8, 2026 | Read time: 9 mins. Kevin Donovan · Architects' HubForwarded this email? Subscribe to Architects’ Hub for the next issue. Hey ,In this edition of Architects' Hub: What the service lead heard when the sponsor said, “We should roll this out.” A commitment record filled in for one example customer-service queue, PLUS a blank copy for yours. How to take a conditional decision back to the sponsor who wants a date. Community Corner: Santiago Jesus Lasso Valenciano on the work that...

A hand-drawn ledger has columns Visible, Committed, Acted and Receipt. Three rows contain circled date markers, with one receipt clipped to the first row. A pencil rests below the sheet.

Nov 1, 2026 | Read time: 10 mins. Kevin Donovan · Architects' HubForwarded this email? Subscribe to Architects’ Hub for the next issue. Where the weeks went Hey Reader, A warning arrives early. Permission arrives late. The recovery gets called heroic. Mike Carroll opens his article on decision latency with a composite plant-floor scene: a machine shows signs of trouble, but the person who can approve a controlled shutdown isn’t available. The line keeps running. Hours later, the bearing...