Skip to main content

Engineering

What We Learned From Benchmarking Web Research Agents

August 22, 2026

We recently spent some time benchmarking TinySearch against several other ways of giving an AI agent access to the web.

We are not publishing the full benchmark setup, task bank, or competitor-level results yet. There are a few reasons for that. The harness is still evolving, some of the measurements need stricter normalization, and frankly, a lot of the most useful findings are currently more valuable as engineering input than as leaderboard material.

What we can share is what the process taught us.

The short version is that the final score was not the interesting part.

The interesting part was watching an agent actually use each research tool, call by call, and seeing where time, tokens, and information were being wasted.

That ended up being a much better way to understand TinySearch than simply asking whether it passed a task.

We wanted to test the tool the way people actually use it

A web research tool is rarely used by a human in isolation. It sits inside an agent loop.

The model decides what to search for, reads the result, decides whether it has enough evidence, opens another page, reformulates a query, or gives up. The quality of the final answer depends on the whole interaction.

So instead of testing endpoints with handcrafted queries and comparing raw JSON, we put the same capable model in front of each system and let it solve the same research tasks with a realistic reasoning setting.

That matters because the interface changes the behavior of the model.

A tool that returns excellent information but encourages three unnecessary follow-up calls may be worse in practice than a slightly weaker retriever that lets the model finish immediately. Likewise, a tool that returns huge amounts of vaguely relevant text can look impressive in a retrieval benchmark while quietly making the agent slower and more expensive.

Anthropic makes a similar distinction in its guidance on agent evaluations. An agent eval is not just a prompt followed by an answer. It is the full trajectory, including tool calls, intermediate results, and the eventual outcome. Their recommendation to inspect transcripts closely matched what we found in practice: the trace often tells you more than the grade itself. See Demystifying evals for AI agents.

The first lesson: a good answer can still hide a bad research process

This became obvious very quickly.

Two systems can both produce the correct answer while taking completely different paths to get there.

One may identify an authoritative source immediately, extract the useful passage, and stop.

Another may search several times, open a handful of pages, carry thousands of extra tokens through the context window, and eventually reach the same conclusion.

If you only score the final answer, those runs are identical.

They obviously are not.

For a production agent, we care about at least five things at once:

Latency and reliability belong in the same conversation too.

Tavily wrote about this directly when describing the development of its own research agent. Their team says careful monitoring of agent traces provided higher-signal feedback than optimizing a single eval score, and highlights token usage, latency, reliability, and failures as meaningful production metrics. That is very close to what we found. See Building Deep Research: How we Achieved State of the Art.

The biggest surprise was how expensive early context can become

One of the most useful things we learned was that returned tool tokens are not just a one-time cost.

Imagine the first web search returns 4,000 tokens. The model reads them and makes another tool call. On the next turn, much of that earlier context is still present. Then it calls another tool. Then another.

A bloated result early in the trajectory can get paid for repeatedly.

This changes how we think about search output.

A search endpoint should probably not try to be a research report. Its job is discovery.

It should give the agent enough information to answer one question: which source should I inspect next?

That argues for compact search results with a clear title, URL, short snippet, useful metadata, and very little else.

Then the reading step can do the heavier work and return citation-grade evidence for the specific question being asked.

In other words:

search -> compact candidate discovery
read   -> focused supporting evidence

The separation sounds obvious, but it is surprisingly easy to blur the two.

More evidence is not automatically better evidence

Another pattern showed up repeatedly: broad retrieval can inflate apparent quality.

If a research tool returns ten sources and several thousand tokens of text, there is a good chance the answer keywords will appear somewhere in the payload. That can make simplistic relevance metrics look excellent.

But the agent may only need one paragraph from one source.

This is why we have become much more interested in evidence precision than raw evidence volume.

The question is not just:

Did the tool retrieve the fact?

It is also:

How much irrelevant material did it make the model read before finding that fact?

For TinySearch, this reinforced a direction we were already moving toward: tighter query-focused extraction, stronger deduplication, and smaller evidence packets.

A page reader should not return a cleaned version of everything vaguely related to the query. It should try to return the smallest set of passages that can support the answer.

That is a much harder objective, but it is also much closer to what an agent actually needs.

Page structure matters more than we expected

A lot of web extraction still treats a page as a long sequence of text chunks.

That is often wasteful.

Many research questions map naturally onto structure:

Once you look at traces, it becomes obvious how silly it is to return an entire table when the answer is contained in one row.

This points toward more structure-aware extraction rather than simply larger semantic chunks or more aggressive crawling.

HTML headings, tables, lists, feeds, metadata, and document hierarchy are useful signals. They let a tool compress information before the model ever sees it.

Structured web sources are underrated

One particularly useful lesson was how often a machine-readable source can be better than the human-facing page.

Release histories are a good example. A rendered release page may be dynamic, partially loaded, noisy, or difficult to parse. An Atom or RSS feed may expose exactly what the agent needs: title, URL, publication time, and ordering.

This is a good reminder that web research does not always mean rendering the prettiest representation of a page.

Sometimes the best source is the boring structured one sitting next to it.

We are now looking more carefully at things like:

These are cheap signals, and they can save a surprising amount of downstream reasoning.

Ambiguity should be surfaced, not hidden

One of the more important failure patterns was entity ambiguity.

If a user asks about a project, company, package, or product with a generic name, a search engine may return multiple perfectly legitimate entities with the same or similar names.

The dangerous failure is not getting zero results.

The dangerous failure is finding the wrong entity, then confidently collecting excellent evidence about it.

Once an agent commits to the wrong interpretation, later searches often become confirmation searches. It starts asking more specific questions about the incorrect target and can build a very convincing answer on top of a bad initial assumption.

That suggests a research tool should help the model notice collisions early.

Not with a giant entity-resolution system, necessarily. Even a lightweight signal that says “multiple distinct entities appear to match this query” can be enough to change agent behavior.

This is one of those cases where explicit uncertainty is more valuable than another page of evidence.

Empty search results should be treated as a failure state

Another practical lesson was almost embarrassingly simple.

A search that returns nothing useful should not look like a successful tool call.

When an agent submits a constrained or oddly formatted query and gets no usable results, it often spends another reasoning turn reformulating the same intent.

The search layer can usually handle that more cheaply itself.

A bounded internal fallback can normalize the query, relax unsupported syntax, try another configured backend, or explicitly return a zero-result state.

That lets the model reason about information instead of debugging search-engine syntax.

This is also why tool interfaces matter so much. Anthropic has written separately about designing tools specifically for agents and evaluating those tools with models in the loop. The basic point is simple: agents are only as effective as the tools they are given. See Writing effective tools for AI agents.

We are becoming more conservative about adding tools

A natural reaction to a benchmark is to add more capability.

More endpoints. More browser controls. More knobs. More fallback modes.

Our traces pushed us in almost the opposite direction.

TinySearch already has enough capability to solve a wide range of research tasks. The bigger opportunity is to make the existing path more selective and predictable.

That means fewer actions, smaller outputs, clearer metadata, better failure signals, and stronger evidence selection.

There is a real cost to giving an agent a giant tool surface. Every extra parameter and tool description consumes context and creates another decision the model may have to make.

The ideal interface may be surprisingly small.

Something close to:

search(query, domains?, limit?, max_tokens?)
scrape_urls(items, max_tokens?)

with most of the messy fallback behavior handled inside the system rather than delegated to the model.

The Model Context Protocol makes it easy to expose tools to agents. The harder engineering problem is deciding how much control should actually be exposed.

Benchmarks are becoming part of the development loop

The most useful change for us is not a single feature. It is the workflow that came out of this exercise.

When we now find a bad behavior in a trace, we can preserve it as a regression case.

Then an engineering change has something concrete to beat.

Did the new extractor preserve evidence completeness while returning less text?

Did a search fallback remove an unnecessary agent turn?

Did metadata parsing let the model answer a freshness question without opening another page?

Did entity-collision detection prevent a confident wrong-target answer?

That is much more useful than shipping a feature because it sounds clever.

Anthropic makes the same broader point in its eval guidance: once you have a stable task bank, you also get regression tracking for latency, token use, cost, and errors. Those measurements become more valuable over time because every change can be compared against a known baseline.

What we are changing in TinySearch

We are keeping the exact roadmap intentionally vague for now, but the direction is clear.

The next improvements are less about making TinySearch do more things and more about making each research step carry more useful information per token.

We are focusing on areas such as tighter evidence extraction, better handling of structured sources, stronger source and entity signals, clearer metadata, smaller search payloads, and fewer unnecessary agent turns.

The benchmark gave us some confidence in the core idea behind TinySearch. A relatively small research tool can give an agent very strong evidence without embedding a second research agent behind the API.

It also showed us where the current implementation is still wasteful.

That is probably the best outcome we could have asked for.

A leaderboard can tell you whether something won a test.

A good trace can tell you what to build next.

If you want to see how TinySearch works today, read the TinySearch architecture overview or install it from the docs.

Back to all posts