Believable Is Not Valid

One of the most dangerous moments in building a simulation system is when the demo starts to feel real.

We experienced this with our early synthetic shoppers. We built rich personas using psychometric dimensions, behavioral traits, and carefully written context. The agents expressed emotions continuously and changed behavior in response to what they encountered. They sounded observant and psychologically active.

Then we compared that behavior with what we observed from shoppers.

In our sessions, people often continued for minutes without an obvious shift in mood or strategy. A stronger signal was usually needed to interrupt that momentum. The agents were more sensitive. A small contradiction or inconvenience could immediately change their emotional state and plan.

The agents were also more attentive to subtle information. They sometimes identified a fine-print conflict that participants appeared not to notice, then treated it as central to the decision. Detecting that conflict could be useful for quality assurance. Reacting to it as a shopper did not necessarily represent observed behavior.

The agents concentrated heavily on completing the assigned task and resolving functional obstacles. The shoppers we observed were also influenced by visual design, aesthetics, novelty, mood, and associations only loosely connected to task completion. They sometimes ignored functional issues that did not interrupt them and spent time on elements the agent considered irrelevant.

We saw uncontrolled variation as well. Important reactions sometimes failed to reproduce across runs even when the persona and environment were unchanged. At other times, an agent invented context to justify its behavior, including one that abruptly decided it had a meeting that existed nowhere in the scenario.

The system was believable. That did not tell us whether it was behaviorally valid.

More precisely, believability can provide face validity: the output looks coherent and recognizable. It does not establish that a real person would behave the same way or that the system can predict what people will do.

Six claims people collapse into one

A fluent interview, coherent persona, or dashboard of simulated responses can invite us to move among several claims:

This is a map of different claims, not a maturity ladder. A system can succeed at one and fail at another.

A synthetic shopper may reveal a broken checkout state even if its emotional narrative is artificial. That is useful journey testing, not behavioral validation. A simulated panel may reproduce an average survey answer while assigning the wrong answers to individuals or subgroups. That is aggregate fit, not individual fidelity.

The mistake is not using plausible output. The mistake is silently upgrading it into evidence about real behavior.

What our own evaluation changed

In an earlier article, I described our progression from rule-based emotion states, to large persona prompts, to a layered architecture grounded in interviews and observed shopping behavior.

Our internal evaluation compared generic agents with personas built from psychometrics, interviews, and behavioral observations. Adding real interviews and observations improved selected similarity measures more than adding increasingly elaborate persona traits. It also exposed persistent gaps, especially in reactions rooted in lived experience.

Those results influenced our architecture. They should not be interpreted as independent proof that the system predicted shopper behavior.

The comparison was specific to our recruited participants, tasks, sites, measures, models, and evaluation process. Some measures involved subjective judgment. The work did not establish transfer to arbitrary purchasing decisions or long-term behavior. It was first-party evidence for an engineering decision, not a universal benchmark.

The honest conclusion was narrower and more actionable:

Within our tested shopping tasks, qualitative and behavioral grounding improved selected similarity measures over a generic agent, but important gaps remained and predictive validity was not established.

Narrower language did not weaken the result. It told us what to test next.

The research supports both optimism and caution

Recent studies show why one verdict on synthetic users would be premature.

Rich grounding can improve bounded performance

A major study created agents for 1,052 people using two-hour interviews, surveys, or both. On held-out General Social Survey questions, agents using interviews and surveys reached accuracy equivalent to 86 percent of the participants’ own two-week test-retest consistency, compared with 74 percent for demographics-only agents. The agents were also tested on personality measures, economic games, and experimental responses. LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals

This is meaningful evidence that person-level grounding can improve performance within measured domains. It is not 86 percent raw accuracy, and it does not establish arbitrary future behavior.

Detailed twins can still fail on novel outcomes

A later preregistered program tested digital twins across 19 studies and 164 outcomes. Each twin had access to more than 500 prior answers from its corresponding person. On new outcomes, the twins’ responses correlated only weakly with the humans’ responses, with an average correlation of 0.20. The full twins were only modestly more accurate than a homogeneous base model and showed systematic distortions including insufficient individuation, stereotyping, representation bias, ideological bias, and hyper-rationality. Digital Twins Are Funhouse Mirrors

The studies do not cancel each other. They ask different questions. One tests held-out responses across defined constructs. The other asks how well detailed twins generalize to new studies and outcomes. A method can improve the first and still struggle with the second.

Simulations can remove the friction that matters

Researchers comparing 1,000 real multi-turn dialogues across 16 domains with simulated users found that the simulations struggled to reproduce communication frictions introduced by real users. The resulting system evaluations could therefore become overly optimistic, and performance varied by domain. Synthetic Users, Real Differences

A simulated user that communicates too clearly, remembers every constraint, and cooperates with the system can produce a clean interaction while hiding failures that matter in deployment.

Similarly, a study replicating 156 psychology and management experiments found that language models often reproduced directional effects but exaggerated their size and turned many human null results into significant ones. The authors position the models as tools for pilot testing and hypothesis work, not replacements for human subjects. Can Large Language Models Replace Human Subjects?

These positive and negative results point in the same direction: simulation can be useful before it becomes a substitute, and usefulness must be evaluated separately from human similarity.

Three recurring mistakes

1. Coherence is treated as behavioral evidence

An agent stays in character, remembers its biography, and explains its actions fluently. That demonstrates internal consistency. It does not show that a real user would take the same action or that the explanation caused the behavior.

Generated reasoning can still help with debugging and hypothesis formation. It should be labeled as simulated or inferred, not observed.

2. Averages and repeated runs hide shared error

A synthetic population can match an average while assigning the wrong responses to individuals or subgroups. Running the same model many times may reduce sampling noise, but it does not remove shared priors, correlated errors, missing context, or a poorly specified environment.

More simulated users create more samples from the system. They do not automatically create more truth about a population.

3. Task success is promoted into prediction

A strong computer-use agent may be more patient, informed, and goal-directed than a real user. Higher completion can improve operational testing while making the agent less representative of abandonment, confusion, and ordinary mistakes.

Even held-out examples provide limited evidence when a new task changes the context, decision, or outcome. Prediction requires a frozen prospective test against real evidence and a credible baseline.

Use the smallest claim that improves the decision

When someone says a simulation works, I now ask three questions:

1. Does it look plausible?

2. Does it improve the decision?

3. Does it correspond to real behavior at the level being claimed?

The first can be judged from the output. The second requires a baseline decision process. The third requires real evidence.

This framing avoids two unhelpful extremes. One dismisses synthetic users because they are not perfect replicas of people, ignoring their value for search, rehearsal, stress testing, and prioritization. The other treats every coherent agent as a replacement for user research, converting model fluency into unjustified certainty.

The productive position is between them:

Use simulation to expand and prioritize what might be true. Use observed behavior, human research, and experiments to establish what is true.

The next article will introduce the Explore, Test, and Predict permissions and show where a simulation claim can fail across grounding, representation, interaction, traces, validation, and decision use.