Ideas
How to Reduce Token Usage in AI Agent Workflows
Token usage isn’t only a line on a bill. It’s the ceiling on what a local model can hold in mind at once: how much room is left for instructions after tool output fills the prompt, how quickly an agent can respond once the context grows past a few thousand tokens, and, for anything running on local hardware, how much of that context a smaller model can actually use before its own attention starts to degrade.
Context as a dump vs. context as a budget
The default habit, especially early in building an agent, is to treat context as a dump: fetch everything that might be relevant, paste it all in, and let the model sort it out. That works right up until the agent needs to do more than one thing at once. Add a second tool result, a longer conversation history, and a system prompt with real instructions, and the dump approach runs out of room, usually while still including material the model never needed.
Treating context as a budget means asking, before anything gets added to the prompt, whether it earns its place. That’s a retrieval question as much as a prompting one, and it applies to every source of context an agent pulls in, not just web search.
Where the tokens actually go
A few habits account for most of the waste:
Retrieving too many documents for a question that only needed one or two good sources. Pasting whole pages or files instead of the specific section relevant to the query. Skipping a source cap, so one unusually long document crowds out three shorter, more relevant ones. Re-sending the same context on every turn instead of retrieving fresh or recalling it only when needed.
Each of these is fixable without changing what the agent can do, only how much it costs to do it. Retrieve fewer documents by ranking harder before deciding what to fetch. Split documents into chunks small enough to rank and select individually, instead of scoring a whole page as one unit. Set a token quota per source so no single document can dominate the context. Distinguish between context that needs to persist and context that was only useful for one turn.
Where this shows up in TinySuite
TinySuite’s two tools are both built around that same assumption: the best context is usually smaller than the easiest context to produce, and it’s worth spending an extra retrieval step to get there.
TinySearch applies this to web research. Instead of returning whole pages, it chunks and ranks crawled content against the query and returns only the passages that scored well, each with its source URL attached. The budgets are explicit and server-controlled rather than left to the calling model to guess: a search call returns a default of ten backend-ordered results, and each item a scrape_urls call opens is capped to a default token budget, so a single overly long page can’t quietly eat the rest of the context window. A model reading that output spends its tokens on evidence, not on the navigation and boilerplate of pages it never needed in full.
TinyContext applies the same idea to memory. Recall doesn’t resend everything an agent has ever saved; it ranks stored memories with hybrid BM25 and embedding search, then trims the result to a token budget before returning it. Durable facts like user preferences live in a separate, always-attached profile block with its own budget, so an agent doesn’t burn part of its per-query memory budget re-retrieving something that should just always be present. In TinyContext’s own benchmark, that trimming cut input tokens by 96.7% compared to resending 300 stored memories raw across eight test queries, with recall staying in the tens of milliseconds even as the store scaled from a few hundred to several thousand memories, which is the kind of gap that shows up directly in cost and latency once an agent has been running long enough to accumulate real memory.
The budget mindset generalizes
None of this is specific to search or memory. Any tool that hands an agent context, a database query, a file read, an API response, benefits from the same discipline: rank before you retrieve, chunk before you rank, cap before you return. The alternative isn’t wrong so much as it’s paying for tokens the model was never going to use, on every single call, for as long as the agent runs.