How to Implement Custom AI Search on Your Business Website

Most teams shipping AI search on a public website treat it as a chatbot problem. It is not. It is a retrieval problem with a generation layer bolted on top, and the retrieval layer is where site-facing implementations quietly fail. The user types four words into a box, expects a grounded answer in under two seconds, and leaves if the top hit is wrong. That is a tighter loop than an internal knowledge assistant and a much smaller error budget.

The commercial case still holds. Site search users convert at 4.63% versus 2.77% for non-searchers per Econsultancy, retailers with advanced search see roughly 2x higher desktop conversion than those with basic search, and yet about 72% of ecommerce sites fail basic user expectations. The gap is an engineering gap. What follows is a build guide for closing it.

Start With a Source Inventory, Not a Model

Before picking an embedding model or a vector store, catalog every corpus the search box will answer from. For a typical business site that includes marketing pages, product or service detail pages, pricing, docs, a help center, blog posts, changelogs, PDFs, and often a Zendesk or Intercom knowledge base sitting behind an API. Each source has a different update cadence and a different authority level.

Record four fields per source: canonical URL pattern, refresh interval, size in documents, and trust tier. Trust tier matters because grounding rules downstream will prefer canonical pages over blog speculation when both match a query. A pricing page is tier one. A three-year-old blog post is tier three. Without this metadata baked into the index at ingest time, the retriever has no way to prefer the right one.

The inventory also decides your crawl strategy. Public marketing pages are fine over a standard fetch; a gated help center needs authenticated extraction, and JavaScript-heavy product pages need headless rendering with proxy rotation to avoid rate limits during a full recrawl. Search.co's own writeup on proxies and vector search covers the extraction side in more detail.

Chunking Policy Is a Configuration Decision

Chunk size is the single largest lever on retrieval quality after model choice, and there is now enough public benchmark data to skip the folklore. In the Vecta 2026 evaluation across 50 academic papers, recursive 512-token splitting scored 69% accuracy and outperformed more expensive semantic-chunking approaches, which averaged 43-token fragments too small to give the LLM enough context to answer. Microsoft Azure's guidance is similar: 512 tokens with roughly 25% overlap as a starting point.

For a business website, split by document type rather than applying one policy globally:

• Marketing and product pages: chunk by semantic section (H2 boundary), then cap at 512 tokens. Keep the page title and breadcrumb in every chunk's metadata.
• Docs and help articles: 512 tokens with 15% overlap, preserving code blocks as atomic units.
• Blog posts: 384 tokens with 10% overlap. Blog prose is looser and benefits from slightly tighter windows.
• PDFs: extract by page, then split. Store page number as metadata for citation.

Every chunk carries at minimum: source URL, page title, section heading, trust tier, last-modified timestamp, and a stable chunk ID. The metadata is what powers grounding filters later.

Hybrid Retrieval Beats Vector-Only in Practice

Vector search alone looks good in demos and fails on the specifics that public sites get asked about: SKUs, model numbers, error codes, plan names, exact prices. Embedding models are trained to generalize meaning, not to guarantee that a query for "SKU 4471-B" retrieves the document containing that literal token. BM25 does that job. The production pattern is to run both in parallel and fuse the results.

Reciprocal Rank Fusion is the default fusion method for a reason. It is parameter-free, robust across score scales, and combining BM25 with dense retrieval via RRF improves recall and precision over either method alone across every subset tested in recent benchmarks. The typical k value for long candidate lists is 60; for shorter top-10 lists, k in the 1–10 range weights the top ranks more heavily.

A reasonable production config for a site search box:

• BM25 index over title, headings, and body, with a modest boost on title matches.
• Dense index using a 1536-dimension embedding model with cosine similarity.
• RRF fusion with k=60, taking top 20 from each side and merging to a candidate pool of 30.
• Optional cross-encoder rerank on the top 30 to produce the final top 5 that goes to the LLM.

The rerank step is the largest single quality gain available. In the same benchmark, adding a cross-encoder to hybrid retrieval delivered +17.2 percentage points MRR@3 over unreranked hybrid. It also adds latency, which is why the next section matters.

Grounding Rules Decide What the Model Is Allowed to Say

Grounding is where site search either becomes a trust asset or a liability. The rule is simple: the model may only assert facts that appear in the retrieved chunks, and every claim must be citable back to a source URL the user can click. Implement this at three layers.

First, in the prompt. Instruct the model to refuse rather than guess when the retrieved context does not contain an answer, and to attach a chunk ID to every sentence it generates. Second, in the retrieval filter. Enforce trust-tier ordering so that when a pricing question hits both the pricing page and a blog post from 2023, only the pricing page is passed to the model. Third, in a post-generation citation check. A lightweight verifier compares each generated claim against its cited chunk and drops answers that fail.

The payoff is measurable. Retrieval-side techniques like hypergraph-driven RAG have been shown to improve accuracy by an average of 12.3% over direct LLM use in domain benchmarks. On a site facing customers, the equivalent lift is the difference between a search box that gets escalated to support and one that doesn't. Search.co's RAG pipelines writeup goes deeper on grounding patterns.

Set a Latency Budget and Enforce It

Public search boxes have a hard latency ceiling. Users abandon around two seconds and consider anything above four seconds broken. The budget has to be allocated up front, not discovered in production.

A workable allocation for a hybrid-retrieval pipeline with reranking and a streaming LLM response is roughly: 40ms for query preprocessing and embedding, 120ms for parallel BM25 and vector retrieval, 180ms for cross-encoder reranking on 30 candidates, 60ms for grounding filters, and the balance for first-token latency from the LLM. Streaming the response matters more than raw generation speed because perceived latency ends at first token, not last.

Cache aggressively. A query-to-answer cache with a short TTL absorbs the long tail of duplicate queries that any public site sees. An embedding cache on frequent queries removes the embedding call entirely for cache hits. Neither is exotic; both are commonly skipped.

A Rollout Checklist That Actually Ships

Ship in stages. A dark-launch phase runs the new pipeline in parallel with existing site search, logging both result sets without changing what the user sees. This produces a baseline diff and catches regressions before any customer touches the system. Follow with a 5% traffic slice, then 25%, then full cutover, with automated rollback tied to two metrics: zero-result rate and cited-answer rate.

Zero-result rate is the cheapest early warning. Well-optimized ecommerce search engines run closer to 5%, while poorly optimized sites exceed 20%. If the new pipeline pushes zero-results up, something in the retriever config is wrong. Cited-answer rate — the share of generated answers where the verifier passes — should stay above 95%. Below that, tighten the grounding filter or expand the candidate pool.

Instrument everything. Log the query, the top 30 candidates with their fused scores, the final chunks passed to the LLM, the generated answer, and the click-through on citations. This is the dataset that lets you tune chunk sizes and rerank thresholds against real user behavior rather than synthetic evals. Vendor questions to ask before signing anything are covered in the enterprise web research contract guide.

1. 11. Dark launchRun new pipeline in parallel, log both result sets · User sees no change · Baseline diff shows no regressions
2. 22. 5% traffic sliceRoute a small share of live queries to the new pipeline · Zero-result rate stable vs. baseline
3. 33. 25% traffic sliceBroaden exposure, watch cited-answer rate · Cited-answer rate stays above 95%
4. 44. Full cutoverNew pipeline serves 100% of queries · Automated rollback armed on both metrics
5. 55. Tune on real logsAdjust chunk sizes and rerank thresholds from logged behavior · Ongoing

Where the Work Concentrates

The visible surface of AI site search is a text box and a streaming answer. The work is everywhere else: the source inventory that decides what gets indexed, the chunking policy that decides how it gets split, the hybrid retriever that decides what comes back, the grounding rules that decide what the model may say, and the latency budget that decides whether any of it ships. Skip any one of those and the box works in staging and fails in production.

The public numbers on conversion lift and query deflection are real, but they accrue to teams that treat this as pipeline engineering rather than model selection. The model is the last decision, not the first. Everything upstream is what determines whether it has anything true to say.