Building a World With Known Answers: Synthetic Data for Agent Evaluation

At Gmail, some of the hardest work in agent evaluation was not building the agent. It was building a world in which we could tell whether the agent worked.

The recurring questions were practical. Did the agent actually help a user solve a problem? Did it retrieve the right evidence, synthesize it correctly, complete the required steps, and produce a useful result? Which benchmark would let us improve prompts, models, retrieval, and workflows instead of optimizing for a few persuasive demos?

For a reusable development benchmark, we could not rely on users’ production inboxes and documents. We therefore constructed representative inboxes, Drive files, documents, and relationships among them. Building and evaluating that data became a substantial engineering and curation effort of its own.

The central lesson was simple:

Synthetic evaluation data is not realistic-looking filler. It is evaluation infrastructure.

Why production data cannot be the whole benchmark

Production evidence remains essential because it reveals how people actually work. It cannot, by itself, provide every property a repeatable development benchmark needs.

• Privacy and access: Real workspaces contain personal information, confidential discussions, and data belonging to people who never consented to model evaluation.

• Known truth: A real workspace contains facts, but it rarely identifies which source is authoritative, which evidence is required, or which outcome should count as success.

• Coverage: Production history contains what happened, not every permission state, contradiction, ambiguous request, or rare consequential failure a system should survive.

• Reproducibility: Messages arrive, files move, policies change, and permissions evolve. Two model versions cannot be compared cleanly if the test world changes between runs.

Synthetic data is not a replacement for real evidence. It is a controlled instrument that lets a team isolate capabilities, reproduce failures, and measure progress before returning to representative real tasks.

The bakery workspace that was too clean

For one early test world, I drew on conversations with the owner of a bakery we visited regularly. The goal was not to reproduce that person’s private data. It was to understand the relationships and communications a small business owner actually manages. From that understanding, we created a synthetic bakery-owner persona and workspace with conversations among friends, vendors, and customers; business documents; operating information; and relationships connecting the artifacts.

After significant quality work, the agent performed well against the synthetic data we had created. When the capability reached beta users, however, it encountered a very different operating environment.

Real inboxes were messy. They mixed personal and business communication, forwarded messages with missing context, group threads with changing participants, conversations extending over years, and large volumes of spam and unread mail. Drive files were scattered, duplicated, inconsistently named, and often outdated. Important facts were rarely stored in one authoritative document.

The most difficult difference was time. People and businesses evolve. Writing style changes. Relationships change. A bakery’s shipping policy today may conflict with a document written two years ago, and both may still be retrievable.

Our original generation process could create a large volume of polished content, but a single generated snapshot flattened that history. It made the test world easier to search and reason over than the worlds our beta users actually had.

We changed the construction process. Instead of generating a massive collection of artifacts in one pass, we began with a coherent model of the person, their relationships, and how the business changed. We used the present-day person and business as an anchor, then constructed a history backward across several years, allowing communications, policies, documents, and writing style to evolve.

In our internal assessment, this improved both the accuracy and representativeness of the test corpus. Each artifact could be generated against a more coherent history, while the resulting workspace better captured the variety, relationships, and temporal inconsistency we encountered in real accounts. Facts appeared across forwards, group messages, long-running threads, and documents written at different times. That gave retrieval and synthesis evaluation a more faithful version of the operating conditions we cared about. It did not prove that the agent would perform well in production. The beta gap was the reason to keep comparing synthetic evaluation with observed tasks and real failures.

The important shift was from generating documents to constructing history.

The unit is a world, not a document

It is easy to generate ten thousand emails. It is much harder to generate a small workspace that behaves like one coherent place.

Consider an account manager preparing for a renewal call. The task may depend on:

• an email thread containing the customer’s latest concern;

• a contract describing the renewal terms;

• an outdated project document;

• a newer meeting note correcting that document;

• a support ticket that should be considered but not quoted externally;

• a restricted file the agent is not permitted to open;

• unrelated messages that look superficially relevant.

Generating each artifact independently creates accidental contradictions. Generating everything from one summary makes the task unnaturally clean. A useful test world needs consistent entities, events, relationships, time, permissions, and distractors.

People and organizations
↓
Accounts, projects, products, and contracts
↓
Events and changing facts
↓
Emails, documents, tickets, and calendar entries
↓
Permissions and visible evidence
↓
Tasks, milestones, and expected outcomes

The documents are projections of the world. They are not the world itself.

Known answers do not mean one scripted path

For an agent, an answer key should be an evaluation contract rather than one preferred sequence of clicks.

These layers expose different failures. A system can retrieve the correct document and misuse it. It can complete every milestone and still produce a poor recommendation. It can write a high-quality answer after taking an unsafe action. One overall score hides those differences.

Existing agent benchmarks reflect parts of this design. WorkArena places knowledge-work tasks inside a hosted enterprise environment. OSWorld defines an initial state and uses execution-based evaluators to inspect the resulting computer state. AgentDojo adds tool use, untrusted data, and security test cases. tau-bench compares the final database state with an annotated goal and measures consistency across repeated trials.

The common idea is that the environment, task state, and evaluator are part of the benchmark. The final response alone is not enough.

Design for the mess that matters

A benchmark is an encoded theory of the work. If every task has one clearly relevant document, it rewards keyword search. If every needed fact appears in the newest message, it does not test cross-source synthesis. If permissions never block an action, it does not test safe recovery.

The world should preserve the forms of difficulty expected in deployment:

• relationships among people, projects, messages, and files;

• facts that change, conflict, or become stale over time;

• permission boundaries, tool failures, and incomplete information;

• ambiguous requests and valid alternative solutions;

• common workflows plus rare but consequential cases.

This does not mean maximizing difficulty. It means representing the operating regime the evaluation is intended to support.

Training and evaluation data also require different controls. Training data may be optimized for coverage and repeated exposure. Evaluation worlds should be frozen, versioned, held out, and protected from template or generator leakage. After major system decisions are fixed, a second evaluation set can reveal whether the team has adapted to the benchmark rather than improved the underlying capability.

Keep the synthetic world connected to reality

The largest failure is allowing the generated distribution to become the product target. A team can improve performance on synthetic tasks while losing contact with real usage.

Synthetic evaluation should therefore be updated from production failures, expert judgment, user research, and observed tasks. Representative real evidence should test whether benchmark gains transfer. When the two disagree, the benchmark should change.

This creates a continuing loop:

Observed tasks and failures
↓
Synthetic world and frozen evaluation
↓
Agent improvement
↓
Comparison with new real evidence
↓
World and benchmark revision

In the framework from Article 1, the primary output remains Category 1: a controlled information environment and a set of measurable cases, even when interaction or journey simulations run within it.

The cleanest claim for a synthetic workspace is:

We know enough about this constructed world to measure whether the system retrieved, reasoned, and acted correctly under these conditions.

That is narrower than saying the agent works in the real world. It is also a much stronger foundation for improving one.

The next article addresses the other side of the problem. Even when a simulation produces coherent, realistic output, what has actually been demonstrated?