Releases

Link Discovery Without a Crawler: TinySearch 0.6.0

August 19, 2026

There’s an awkward gap between the two things web tooling usually offers an agent. Scraping works on URLs you already have, which is fine until the page you fetched turns out to be a hub that points at the page you actually wanted. Crawling closes that gap by following links itself, but it decides how far to go and how much to read, and an agent that hands off that decision has handed off its context budget too.

TinySearch 0.6.0 takes the middle path. A scrape of an HTML page now comes back with a short list of links found on that page, ranked against your query. TinySearch doesn’t follow any of them. It hands them to the model and lets the model decide whether a second scrape_urls call is worth making.

Why not just crawl

The case for autonomous crawling is that it saves a round trip. The case against it is everything that round trip was buying you.

An agent loop accumulates context at every step. Each tool result stays in the conversation, gets re-sent with the next request, and competes with everything else the model needs to hold onto. A crawler that follows links on its own is making spending decisions against that budget without knowing what the agent is actually trying to answer, or how much room is left. It finds pages, and pages are exactly the thing that’s expensive to put in a prompt.

The model, on the other hand, knows both. It knows the question, it knows what it already learned from the first page, and it knows whether “Pricing” or “API reference” is the link that moves it forward. That’s the decision worth keeping in the loop, and it costs one short list of candidates to keep it there.

So related_links is deliberately not a crawl. It’s discovery without traversal.

What comes back

Every HTML scrape now includes a <related_links> block alongside the page content:

<related_links>
  <link rank="1">
    <url>https://example.com/getting-started</url>
    <text>Getting started guide</text>
  </link>
</related_links>

Just the URL and its label, capped at scrape_max_links candidates, eight by default. No page content, no previews, no scores in the prompt. The point is to be cheap enough that including it never has to be a judgment call, so the agent can look at the list, ignore it, and move on when nothing there is useful.

How the list gets ordered depends on how you called the tool. With a focused query, candidates are ranked using the same hybrid BM25 and embedding retrieval TinySearch already runs over content chunks, scored against each link’s visible text plus the surrounding page context rather than the URL alone. In page-order mode, when the query is omitted or set to "*", links keep the order they appeared in on the page and aren’t scored at all, which also means that path never initializes the embedding model.

Document scrapes don’t produce related links. A PDF or DOCX goes through a different extraction path with no anchor tags to collect, so related_links is empty for those items rather than misleading.

The part that was harder than it looked

Link-dense pages are the interesting case. A news homepage or a docs index can carry several hundred anchors, and embedding all of them to rank eight is a waste.

The obvious fix is to take the first hundred and rank those. The problem is that DOM order has nothing to do with relevance. Navigation, cookie notices, and social icons cluster at the top of the markup, and the one link that actually answers the question is frequently far down the page. Truncating before ranking drops it before anything has scored it.

So the pre-filter is lexical first. BM25 runs across every candidate on the page, which is cheap because it’s pure token counting with no model involved, and only the top hundred by that score go on to be embedded. A relevant link buried at position 400 still gets pulled into the pool. Ranking cost stays bounded, and the bound stops being a blind spot.

This release also fixes a ZeroDivisionError in exactly that path. A page with more than a hundred links whose labels and surrounding context all tokenize to nothing, image-only navigation being the realistic case, produced an empty BM25 corpus and a division by an average document length of zero. The pre-filter now detects that there’s nothing lexical to rank and falls back to page order instead of crashing the scrape.

Bring your own browser

The other addition in 0.6.0 is unrelated but has been asked for repeatedly. TinySearch ships with Playwright’s bundled Chromium and installs it during tinysearch setup. Now you can point it at a browser you run yourself:

{
  "browser_cdp_url": "http://browser:9222"
}

Server processes accept TINYSEARCH_BROWSER_CDP_URL for the same setting. When either is present, TinySearch connects over the Chrome DevTools Protocol through Crawl4AI instead of launching or installing anything of its own.

Two reasons this matters. The first is that a bundled Playwright Chromium is fairly identifiable. Its network fingerprint doesn’t match a real Chrome release, and anti-bot systems have gotten specific about noticing, so pages that a normal browser renders fine can come back empty or blocked. Attaching to a browser you control means the executable, profile, proxy, and fingerprint are yours to configure, and TinySearch stays out of a fight it isn’t built to win. The second is simpler: a persistent browser can hold a logged-in session, and reusing one across scrapes beats launching a fresh instance per run.

Treat the endpoint as what it is. CDP is privileged remote control of a browser, so keep it on loopback or a private network, require authentication if it crosses a host boundary, and don’t put port 9222 on the public internet. If TinySearch itself runs in Docker, remember localhost means the TinySearch container, not your host. The setting is also deliberately operator-only: PUT /config can’t change it even when configuration writes are enabled, so an HTTP client can’t quietly repoint the browser. Set it in the environment or the config file and restart.

Upgrading

pip install --upgrade tinysuite-search

Nothing here is breaking. Related links show up automatically on HTML scrapes, and setting scrape_max_links to 0 turns them off if you’d rather not spend the tokens. The external browser is opt-in and does nothing until you set an endpoint.

The configuration reference has the full key list. If you’re arriving at TinySearch cold, the overview post covers what it is and how the search-then-scrape flow fits together.

Back to all posts