What It Took to Build Agents That Act Like Real Customers

Where this started

It started as a fun experiment. We pointed an AI shopper at Stitch Fix onboarding and asked it to think out loud like a real customer. It hit a signup error, went looking for help, retried, ran into an account conflict and gave up.

The gaps it surfaced shocked Stitch Fix’s leadership, and they reworked the flow. It also left us with a much bigger question: could we simulate how any customer thinks and acts, well enough that a team would act on what it finds?

Answering it meant solving four problems in order, and building a way to measure each one before we could improve it. We started as UserApproved in ecommerce. Today, as Kinalo, the same work powers growth agents for telehealth.

1. The hands: getting through real websites

Before an agent can explain a journey, it has to finish it. In 2025 that was the hard part. The best published agents finished about half of WebArena tasks, and ours clicked color swatches 16 times trying to pick size M, got stuck behind region popups and Cloudflare walls, and spent 30 seconds on every step.

Getting there meant building the test ground ourselves. We set up frozen and live simulation environments, built our own test stores (clean, cluttered and impulse-driven designs, with bugs planted in them) alongside a bank of real shopping tasks on live sites, and layered DOM reading with computer vision so the agent could handle elements neither could alone. With site maps and recovery actions on top, our own stack reached 88% on our hardest store, at a time when models and third-party browser tools were far less mature.

Two lessons from that work still shape how we build. Check the grader before the agent: in one run, 9 “failures” were tasks the agent had actually finished, and fixing the grader moved the score more than any agent change that month. And know when to stop building: within six months the market caught up, a frontier computer-use model ran our tasks 4.7 times faster than our own orchestration, and we rebuilt around it before handing execution to a third-party hosted browser agent.

Looking back. Agents on today’s frontier models clear 90% on public web benchmarks: Magnitude reports 93.9% on WebVoyager. Execution has become infrastructure, which freed our effort for the brain.

2. The brain: thinking like a real shopper

Richer prompts did not make agents behave like different people. Real stories did. Grounding personas in interviews and observed shopping moved similarity to real shoppers from 19 to 34 out of 50.

It was a deliberate bet. Enterprise VPs offered to sign if we dropped the personas and sold a cheaper QA tool. With a team of two, we chose the harder problem. Our early personas carried 30+ traits that barely changed how they thought and behaved. They made navigation worse, forgot their own frustration a page later, and one announced mid-task that it had a meeting at 2:30.

Four investments moved quality the most:

• Thinking split from clicking. A persona agent wakes only at friction points, so the browsing agent stays reliable.

• Long-term memory. Persistent beliefs, goals and emotion, so frustration on page 3 still shapes the choice on page 4.

• Behavior grounded in data. Real interviews and observed shopping first: a story about a regretted impulse buy shaped behavior more than a field like “Neuroticism: 3.1”. Later, profiles built from each store’s own analytics and customer interviews beat the ones we had researched by hand.

• Models chosen on long sessions. Over 100 to 150 turns, one held its persona at 4.7 out of 5 while another drifted to 2.2.

We measured it against real people. Shoppers took an hour-long psychometric assessment, an observed shopping task and an interview, and we scored the agent against each person’s actions and reasoning. We also shared our long-session evaluations with frontier labs’ computer-use teams, focused on three gaps: persona drift, turning browsing into findings, and stopping too early.

Looking back. The moat was data, not prompts. Our profiles still missed choices rooted in lived experience, like a shopper judging Halara against a competitor she had used the week before. Simulating anyone, anywhere, is a research problem; simulating a defined customer in one vertical is solvable.

3. The judgment: would a CRO expert give the same advice?

Finding a real problem is not advice. A recommendation has to be supported, specific and possible for that store to ship. In October 2025, a CRO expert rated our fixes 2 out of 5. Ten months later, reviewers scored our full report 85 out of 100 on a store where GPT scored 68 and Claude 65.

The early fixes needed Shopify Plus the store did not have, cited screenshots that did not show the problem, and said “compress the images” without saying to what size.

Five CRO experts audited stores on their own, and their reasoning became our rubric: is it true, is it deep, does it help, can a team act on it? AI judges apply that rubric at scale, but only after experts calibrated them, and we audit the graders too. One scorer treated unknown revenue as zero; fixing it moved our score from 54.8 to 68.1 without touching the agent. We score the exact report a customer receives, and every real miss from a customer becomes a new test case.

From a right answer to a good teammate. Copilot answers a team’s questions about its own data, often in Slack, and early versions behaved like a new hire who had not learned the room. “Scratch that” still set off ten tool calls. A plain “yes” launched a mockup nobody asked for. A follow-up quietly reused the last question’s date range, and some answers estimated numbers instead of pulling them. Our fixes took quality passes from 6 of 33 to 32 of 33 on 11 saved journeys and cut tool calls from 86 to 21, and we still held the release for one critical error. The bar now is a teammate’s: resolve what was asked and what was meant, do the work instead of handing it back, ask only the question that changes the answer, and show quietly that it is working before it replies.

Looking back. Frontier models used directly are already sharp analysts. What they lack is what a team needs before acting: evidence a reviewer can check, advice built on the store’s own customers and results, and recommendations that learn from each experiment. That is the gap a vertical agent is built to close.

4. The action: from advice to a tested change

Quality has to survive all the way to the screen a patient sees. When we first brought our agents into telehealth, they did not work out of the box. On test businesses with known facts, our Strategist passed only 15 of 36 trials. It fixated on issues that did not matter to patients and misread the journey from ad to landing page to a long medical intake, missing the broken expectations and the fatigue that make patients quit.

Two agents do this work: the Strategist finds what to fix and sets up the test, and Recovery wins back patients who dropped off. Half of the Strategist’s early drafts promised prices or cancellation terms that did not exist.

Rules about evidence more than doubled the pass rate: behavior can support a hypothesis, but price, access and care claims need explicit proof. Missing data now stays missing instead of turning into zero, and the agent can answer “facts needed” instead of guessing. A three-case pilot on our full product path, against the OpenAI Agents SDK and Deep Agents, gave us no clear reason to switch runtimes, so the effort went into evidence and context.

Two investments moved quality the most. Before each run, we securely pre-analyze how every patient cohort behaves, so the agent starts from context instead of raw events. And every capability in our product, from funnel breakdowns to previewing a change on the live intake, is available to the agent as a tool or skill for investigating and checking its own work.

The hardest lesson came at the last step. One proposed GLP-1 test changed a screen-reader prompt patients never see, so it could not test anything. We rejected it, verified helper text on the screen patients actually use, and now check every test there: the saved assignment, mobile, resume and duplicate submissions.

Looking back. The biggest gains came from evidence discipline and context, not a bigger model or a new harness. The edge was in what the agent knew and could check, not in the loop around it.

What it taught us about evaluation

Our agents got better because our evaluation got better first. Three practices did the most:

1. Start with the data. Real shoppers, CRO experts’ audits and test businesses with a known answer key. No public benchmark measured what our customers needed.

2. Test journeys, not prompts. Each case is a customer goal and the paths a person might take to reach it. Frozen journeys catch regressions, and live cases from customers’ real wins and misses keep the coverage honest.

3. Calibrate the graders. Rubrics make “good” checkable, AI judges apply them at scale, and experts keep the judges honest. Two of our biggest jumps came from fixing graders, not agents.

One rule held everywhere: score what the customer actually receives, and hold the release for one critical error, however good the average looks.

It started with one simulated shopper giving up on a signup form. Today our agents can walk a journey the way a real customer would, explain what went wrong and set up a tested fix. What we are working on now is proving that each fix grows the business.

How to read these numbers

• Each chart comes from its own study. Tasks, sites and graders changed as we learned, so each line shows progress on the problem at the time, not one fixed benchmark.

• Some samples are small: about 14 shoppers in the human study, one store in the CRO head-to-head, three cases in the harness pilot.

• Intake results use synthetic businesses and patients, never real patient data.