Migrating off Bright Data and Glean to a Unified Research Stack

Most consolidation pitches start with a tidy slide: one vendor, one invoice, one throat to choke. The reality on the ground is less tidy. A team already runs Bright Data or Oxylabs for proxy traffic, Glean or Vectara for retrieval, a vector store it half-owns, and an orchestration layer somebody built in a hackathon. The contracts renew on different dates. The embeddings are tied to a model nobody wants to keep. The scraping budget is lumpy. And the people asking for the switch are not the people who have to run it.

This is a cutover guide for that team. It covers what actually breaks when a proxy vendor and an enterprise search vendor get folded into one pipeline, what the exit clauses tend to hide, and the parallel-run budget needed before the old stack can be powered down. The goal is not to argue that consolidation is correct in every case. It is to make the switch legible enough that the decision gets made on numbers rather than vibes.

Why the Two-Vendor Pattern Started Showing Strain

The split made sense when scraping and retrieval were separate problems. Bright Data and its peers sell proxy bandwidth; the web scraping market is worth USD 1.56 billion in 2026 and growing at a 17.39% CAGR toward USD 3.49 billion by 2031, with a scaled infrastructure tier of Bright Data, Zyte, Apify, and Oxylabs. Glean sells permission-aware search over internal systems and has grown fast: Glean reached $300M in ARR in May 2026, up from $208M at end of 2025, at a $7.2 billion valuation. Different jobs, different buyers, different procurement cycles.

What changed is that both jobs now feed the same downstream agent. A competitor-pricing crawl, a SEC filing, and a Confluence page end up in the same prompt. Keeping them in separate vendors means two retrieval stacks, two auth models, and two sets of embeddings that do not share a schema. Finance has started noticing. Intuit's 2026 Enterprise Technology Benchmark Report found 73% of senior US business and finance leaders agree that consolidating their tech stack is the fastest path to a healthier bottom line, and Spendesk's Top CFO Tools Report 2026 found more than 90% of finance teams now use six or fewer tools, with 41% on just one to three.

The engineering case is sharper than the finance one. A Zapier survey of over 500 enterprise leaders found 70% of enterprises have not moved beyond basic integration for their AI tools, and three in four have already experienced negative outcomes from disconnected AI. Separate retrieval and extraction layers are a leading contributor. For a broader read on where this is heading, our earlier piece on AI and retrieval merging covers the architectural direction.

Mapping the Current Stack Before Touching Anything

No migration survives contact with an unmapped stack. Before any RFP goes out, inventory three things: the extraction surface, the retrieval surface, and the connectors between them. Larridin's State of Enterprise AI 2026 research found the average enterprise uses 23 different AI tools, but only 38% maintain a complete inventory of what is actually running. The gap is where migration budgets die.

A workable inventory covers:

• Proxy traffic: monthly GB by proxy type (residential, ISP, datacenter, mobile), success rate by target domain, and the share of volume routed through managed APIs like Web Unlocker versus raw proxies.
• Retrieval surface: number of connected sources, document count, average document size, embedding model and dimensionality, re-index cadence, and permission model.
• Downstream consumers: every agent, notebook, or dashboard that reads from either system, with the query volume each generates.

That inventory becomes the parallel-run contract. Anything not on it will not be migrated on day one, and anything on it that nobody can name an owner for is a candidate to retire rather than port.

The Contract Exit Traps That Actually Cost Money

Bright Data, Oxylabs, and Zyte sell on consumption, which makes exit mechanically simple but financially sticky. Residential proxy pricing moves with committed volume: pay-as-you-go rates sit around $8/GB on Bright Data's own tiers, dropping to roughly $5/GB on the $1,999/month plan, per published Bright Data pricing for mid-2026. A team that signed a 12-month commit for a lower per-GB rate owes the remainder of that commit whether it uses the bandwidth or not. The common exit trap is signing a large annual commit two months before deciding to consolidate.

Glean exits are structurally different. The pricing is quote-based and seat-anchored; buyer intelligence puts Glean at approximately $75 per user per month, with enterprise deployments commonly starting around 100 seats. Three contract clauses deserve a legal read before notice is given:

• Data export format and window. Permission-aware indexes are not portable by default; exports often ship as flat JSON without the ACL graph.
• Connector decommission. OAuth tokens issued to the incumbent should be revoked on the source systems, not just inside the vendor UI.
• Minimum seat ratchets. Many enterprise seat contracts prohibit mid-term reductions, so parallel-run periods pay full seat count on both sides.

Our prior write-up on vendor questions before signing covers the clauses to flag at contract time, which is the cheapest moment to fix them.

Re-Embedding Is the Line Item Everyone Underestimates

Switching retrieval vendors usually means re-embedding the corpus against a model the new stack supports. The token math is friendlier than it sounds. OpenAI's text-embedding-3-large runs $0.13 per million tokens standard and $0.065 per million via Batch API, and a 10,000-document corpus runs around $0.65 at standard rates. The number people miss is not the embedding call itself. It is the extraction cost to rehydrate every source document, re-chunk it, and re-run quality checks against the new index.

For a mid-sized research corpus (say, 5M documents averaging 2K tokens), the embedding bill is modest. The crawl-and-rehydrate pass against the open web portion of the corpus is where parallel-run budget gets consumed, and where a BYOK model on the new stack changes the math: tokens bill at the provider rate rather than a vendor markup. For teams routing across models, the breakdown in routing GPT-5, Claude, and Gemini by task covers how to keep that cost from drifting during the cutover.

A Parallel-Run Plan That Will Actually Finish

Parallel-run is the phase where consolidation projects quietly die. The old stack keeps serving production. The new stack indexes the same sources. Every query hits both. Results are compared. The exit date slips because nobody owns the kill criterion.

A plan that finishes has four pieces, in order:

1. Shadow traffic, read-only. Mirror a sampled share of production queries to the new stack. No user sees the result. Compare retrieval recall and latency.
2. Dual-write on extraction. Both proxy stacks fetch; the new one writes to a staging index. Costs run high here. Budget for 1.5x to 2x of the old stack's monthly spend over this window.
3. Narrow cutover by workload. Move one agent or one dashboard first, usually the one with the clearest success metric. Competitor pricing crawls are a good first candidate because the ground truth is checkable.
4. Documented rollback criterion. A single sentence, agreed before cutover: "If first-attempt answer accuracy on the validation set drops below X for Y consecutive days, roll back." Without that line, every regression becomes a committee meeting.

Keep the parallel run short. Thirty to sixty days is enough for most teams. Longer windows cost real money and rarely surface new information after week four. Internal enterprise search systems achieve roughly a 10% first-attempt success rate versus Google's 95%, per the Slite Enterprise Search Survey 2026, so a validation set anchored on first-attempt accuracy is the metric that will tell the truth fastest.

What a Unified Stack Should Actually Deliver

Consolidation is worth doing when it collapses the handoffs that generated the two-vendor pattern in the first place. In practice that means: one auth model across proxy and retrieval, one schema for extracted and internal documents, one routing layer that picks the model per task, and one place where results are versioned and audited. The pipeline-first architecture that pairs proxy extraction with the ingestion stack is designed for exactly that collapse.

The payoff is operational, not just financial. McKinsey Global Institute research found knowledge workers spend about 20% of work hours — roughly 1.8 hours a day or 9.3 hours a week — searching for internal information and tracking down colleagues. A unified stack does not solve that on its own. It removes the structural reason a research question has to be asked twice: once of the internal index, once of the open web.

When to Keep the Split

Not every team should consolidate. Three patterns argue for keeping the two-vendor model:

• Scraping volume is heavy enough that a dedicated proxy contract genuinely beats a bundled rate, and the team has the engineering to manage rotation, retries, and compliance in-house.
• The internal search deployment is already wall-to-wall across the company, with hundreds of connectors mapped and permission graphs that would be expensive to rebuild.
• Regulatory posture requires physical separation between external data acquisition and internal knowledge, with distinct audit trails.

For everyone else, the question is sequencing rather than principle. Consolidate the layer whose contract renews first, run the other in parallel for a quarter, then decide. The failure mode to avoid is cutting both vendors in the same month and discovering the rollback plan was a slide, not a system.

The Cutover Window Closes Faster Than It Opens

Most migrations stall because the window to run two stacks in parallel is narrower than it looked at kickoff. Budgets tighten mid-year. The incumbent proxy contract renews. A new agent gets greenlit on whichever stack is live that quarter. The teams that finish the cutover are the ones that wrote the kill criterion first, picked a single workload to prove it on, and treated re-embedding as an extraction cost rather than a model cost. Everything else is paperwork.