Ideas
Why Raw Webpages Are Bad Context for AI Agents
The easiest way to give an agent web access is also the worst one: fetch a page, paste the whole thing into the prompt, and let the model figure out what mattered. It works for a demo. It falls apart the moment the page is a real webpage instead of a tidy article, because most of what a browser renders was never meant for a model to read.
What’s actually on a typical page
Open the raw HTML or even the rendered text of an average webpage and count what isn’t the content you wanted. Navigation menus repeated in the header and footer. Cookie banners and newsletter prompts. Related-article widgets, comment sections, ad slots, and tracking scripts that render as invisible noise but still take up characters. On a long article, the paragraph you actually need might be a few hundred tokens surrounded by several thousand tokens of everything else.
An agent that pastes the whole page into its prompt pays for all of it. Every token of navigation and boilerplate is a token not available for instructions, other tool results, or the model’s own reasoning, and it’s a token the model has to read past before reaching anything useful. Longer context windows have made this cheaper to get away with, not free: retrieval quality tends to degrade as the useful signal gets diluted by noise, even before you count the API cost of the extra tokens.
It also breaks citations
There’s a second cost that’s easy to miss. When a model is handed a full page and asked to answer from it, the citation it produces is a claim about which part of that page mattered, made after the fact, by a model that had to infer the structure itself. Ask it to point to the specific sentence that supports its answer and you’re often trusting a paraphrase of a paraphrase.
Compare that to a pipeline that ranks and selects the relevant passage before the model ever sees the page. The citation isn’t a guess about what mattered; it’s the actual text that was retrieved because it mattered. That distinction matters more as agents get used for things where a wrong or unverifiable citation has a real cost.
What a page should become before it reaches a prompt
Treat a page as material to process, not content to forward, in four steps. Crawl it and extract the readable text, stripping the navigation and boilerplate that never should have counted as content in the first place. That extraction step can itself be query-aware: filtering a page’s content against the actual question, rather than just stripping known boilerplate patterns, catches noise a generic filter would miss, and drops content below some minimum length rather than accepting whatever survives. Split what’s left into chunks small enough to rank independently and sized to a consistent token target, since a page’s relevant paragraph and its irrelevant one shouldn’t be scored as a single unit, and a small overlap between adjacent chunks keeps a sentence from being cut in a way that strands its own context. Rank those chunks against the actual query, not just the page’s overall topic, and drop near-duplicates so the same point retrieved from two pages doesn’t crowd out a different one. Return the smallest set that answers the question, capped per source so a single long page can’t dominate the result, with the source URL still attached to each chunk.
That’s the shape TinySearch uses: search, crawl the pages worth opening, chunk what comes back, rank the chunks, and return ranked evidence instead of raw pages. The crawling and extraction underneath it comes from Crawl4AI, which does the specific job of turning a URL into clean markdown; TinySearch adds the search and ranking steps around that extraction so the result a model receives is already narrowed to what’s relevant.
The result is unglamorous, and that’s the point
None of this is a clever trick. It’s a compression step that has to exist between the web and a model’s context window, the same way a search engine has to rank results before a person can use them. The alternative, pasting raw pages and hoping the model sorts it out, works until the page is long, the question is specific, or the citation needs to hold up to a second look. Ranked, chunked, source-attributed evidence is more work to produce and considerably less work for everyone downstream of it, including the person reading the answer. The same logic applies past web pages, too: it’s the same reasoning behind treating any tool’s context as a budget rather than a dump.