Engineering
What Crawl4AI Does, and What TinySearch Adds on Top of It
Crawl4AI is one of the dependencies TinySearch is built on, not a rival to it. It’s worth saying plainly because “crawl4ai vs X” is a common search, and the honest answer for TinySearch is that there’s no versus. TinySearch’s crawling step runs on Crawl4AI. The interesting question is what TinySearch adds around it, and why that gap exists in the first place.
What Crawl4AI is for
Crawl4AI is an open-source, Apache-licensed Python library for turning webpages into clean, LLM-ready markdown. It drives a real browser through Playwright, strips out navigation and boilerplate, and can execute JavaScript, take screenshots, generate PDFs, and route around anti-bot defenses. It’s fast at the one thing it’s built for: given a URL, hand back readable content a model can use.
What it does not do is search. Every entry point into Crawl4AI needs a starting URL or a domain you already know. Its AsyncUrlSeeder can discover pages within a known domain from sitemaps and Common Crawl, then rank them against a query with BM25, but that’s filtering a set of URLs you’ve already enumerated, not searching the open web from a bare question. AdaptiveCrawler works the same way: it takes a start_url and a query, then decides when it’s read enough of that site to stop. Neither tool answers “where on the internet would I find this.”
That’s the gap TinySearch fills. When a question doesn’t come with a URL attached, something has to turn it into candidate pages first.
Where TinySearch uses Crawl4AI directly
TinySearch’s crawl step is Crawl4AI running underneath it, and several of TinySearch’s own configuration settings pass straight through to Crawl4AI’s machinery rather than reimplementing it. crawl_fit_markdown_mode chooses between Crawl4AI’s bm25 and pruning content filters, or turns filtering off entirely. When BM25 filtering is selected, crawl_bm25_threshold (default 1.5) and crawl_bm25_language control the same query-aware filter Crawl4AI uses to trim page noise before content moves further into TinySearch’s pipeline, and crawl_pruning_threshold controls the alternative, query-independent pruning filter for cases where no query context is available. crawl_fit_min_chars sets a floor on how much filtered content has to survive before TinySearch accepts it, which catches pages where aggressive filtering left too little to be useful. TinySearch isn’t reimplementing extraction; it’s configuring the extractor it depends on, then deciding what happens to the output.
Crawl4AI does have chunking strategies of its own, along with content filters like PruningContentFilter for noise removal, and its Adaptive Crawling mode adds confidence scoring across coverage, consistency, and information saturation. That’s genuinely useful for deciding when a crawl of a known site has gathered enough. It’s a different job from ranking chunks across many different pages against one query and returning the smallest useful set, which is what TinySearch’s rerank step does after Crawl4AI has finished extracting.
The pieces TinySearch adds
Three things sit on top of Crawl4AI in TinySearch’s pipeline.
The search step comes first. TinySearch queries DDGS by default, or a bundled SearXNG instance in the Docker deployment, to turn a question into a ranked list of candidate URLs before any crawling happens. Only the top-ranked results, search_max_results_to_keep in TinySearch’s configuration, five by default, actually get passed to Crawl4AI for extraction, so the crawler is never asked to fetch more than the search step judged worth opening.
Chunk ranking comes after crawling. Once Crawl4AI has returned filtered markdown for the pages worth opening, TinySearch splits that content into chunks (crawl_max_chunk_tokens, 300 by default, with crawl_overlap_tokens of context shared between adjacent chunks) and ranks them against the original query with embeddings fused against BM25 through weighted reciprocal rank fusion, not just the content-level filtering Crawl4AI already did within each page. A near-duplicate cutoff drops chunks that are too similar to ones already kept, and a per-source cap keeps one unusually long or repetitive page from dominating the final evidence set.
The MCP interface wraps the whole thing. search(query), scrape_urls(items), and get_current_datetime() are tools any MCP client can call over stdio without writing Python or standing up a server.
Crawl4AI does ship its own MCP support, but it lives on the Docker server as a secondary feature rather than a maintained standalone project, with endpoints for tools like md, html, screenshot, crawl, and execute_js. There’s no search tool among them, and no stdio path: it’s Docker or nothing, with auth on by default since its 0.9.0 hardening pass. That’s a reasonable design for a general-purpose crawling server. It’s a heavier setup than TinySearch needs for the narrower job of turning a question into cited evidence.
Where each one is the right tool
If you already have a URL and want the cleanest possible extraction, including JavaScript execution, screenshots, or PDF generation, use Crawl4AI directly. That’s genuinely its strength, and TinySearch doesn’t try to compete with it on extraction depth. TinySearch’s own crawl settings expose only a slice of what Crawl4AI can do, the slice that matters for turning a search result into a ranked evidence chunk.
If you’re starting from a question and need an MCP tool that turns it into cited, chunked evidence without a search API key or a hosted account, that’s the gap TinySearch was built to close, using Crawl4AI as the extraction layer underneath. Install TinySearch to see the two working together, or read more about how TinySearch’s full pipeline fits together.