Implementing Internal Search on Private Corporate Databases

Most "enterprise AI search" demos read the same way: a public web crawl, a vector store, and a chat box. That shape is fine for marketing copy and competitor dashboards. It falls apart the moment the index has to span Postgres, Snowflake, SharePoint, Confluence, and a dozen S3 buckets whose contents a given employee may or may not be cleared to read.

The hard part of internal search on corporate databases is not retrieval quality. It is making sure the retrieval layer honors the same access rules the source systems already enforce, strips regulated fields before they touch a model, and keeps tenant data from mixing in a shared index. Under a BYOK deployment, where the customer's own LLM keys are in play, there is no vendor to blame if a chunk leaks. The pipeline has to be right.

What does that pipeline actually look like, connector by connector?

Why Internal Search Is a Different Problem

Public-web RAG optimizes for coverage. Internal search optimizes for correctness under constraints. IBM estimates roughly 90% of enterprise-generated data is unstructured, and BARC's 2026 survey found that only 29% of respondents fully know where their AI-relevant unstructured data lives, with 70% reporting that less than half of it is discoverable for analytics or AI. That is the problem space: large, poorly mapped, and governed by rules that live in the source systems rather than the search layer.

The economic pressure is pushing the same direction. On-premises or edge enterprise AI inference reached 55% in 2026, up from 12% in 2023, and 70% of enterprises now say they prioritize internal LLMs over public options. BYOK and private inference are the default assumption, not a premium tier. The BYOK deployment model changes the trust boundary: the search platform never sees prompts or completions in cleartext, which also means it cannot rely on the model vendor's post-hoc filters to catch a permissions bug. Enforcement has to happen before the chunk is retrieved.

Connector Patterns That Preserve ACLs

Every connector in a serious internal-search stack has to answer three questions at ingestion time: what is the canonical ID of this object, what are the permissions attached to it, and how will those permissions be refreshed. The implementation differs by source.

Postgres and Snowflake

Relational sources are the easiest case because the permissions model is already in the database. The standard pattern is to co-locate chunks and embeddings with a tenant_id and an acl_groups column, enable row-level security on the chunk table, and run similarity search as the authenticated role rather than as an admin. pgvector inherits Postgres RLS directly, so a CREATE POLICY clause filters the vector index scan the same way it filters any other SELECT. For Snowflake, the equivalent is row access policies plus secure views over the embedding table.

One pgvector-specific trap: with HNSW indexes, filtering runs after the approximate nearest-neighbor scan, so a very selective tenant filter on a large shared table can return fewer than k rows. pgvector 0.8.0's iterative index scans keep walking until enough rows pass the policy, which is the right default for multi-tenant workloads.

SharePoint and Confluence

Document stores are harder because permissions are maintained in a separate identity graph and change constantly. The current Azure AI Search SharePoint connector detects and updates ACL changes incrementally on each successful indexer run for items with unique permissions, and extends ingestion to list items, ASPX site pages, and SharePoint groups. Microsoft 365's Confluence connector takes a cruder approach: full crawls detect deletions and sync ACL changes every 24 hours by default, while incremental crawls every 15 minutes do not pick up ACL changes. Any pipeline that reads from Confluence has to either shorten the full-crawl interval, subscribe to webhooks, or accept a staleness window during which revoked users can still see snippets in results.

The practical rule: store the source ACL on the chunk as a group ID set, resolve the caller's group membership from the IdP at query time, and intersect them before similarity search. Resolving at query time avoids the stale-group problem that one-time ingestion snapshots create.

S3 and Object Stores

S3 has no row-level security to inherit, so permissions have to be synthesized at ingestion from bucket policies, prefix conventions, and object tags. The connector should emit an explicit ACL per object even when the bucket is "team-wide," because team membership drifts. Treat object-tag changes as an invalidation event and re-emit the ACL rather than re-embedding the content.

Hybrid Retrieval Earns Its Keep Inside the Firewall

Dense vector search alone underperforms on the kinds of queries internal users actually type: ticket numbers, SKUs, internal project codenames, legal clause references. Hybrid retrieval, which combines sparse keyword scoring (BM25) with dense embeddings, closes that gap. Hybrid RAG improves NDCG by 26–31% over dense-only approaches on benchmark datasets, and Anthropic's Contextual Retrieval, which fuses contextual embeddings with contextual BM25, cut top-20 retrieval failure by 49%, or 67% with reranking.

The implementation detail that matters for internal corpora: run the sparse and dense passes against the same ACL-filtered candidate set, not against two separately scoped indexes that can drift. Fuse with reciprocal rank fusion, then apply a cross-encoder reranker that has also been given the ACL context. Reranking is where cost shows up, so this is where routing logic from a cost-aware model router pays back: use a small local reranker for high-volume queries and reserve a frontier model for the long tail.

PII Masking and Tenant Isolation Under BYOK

BYOK removes the vendor's model from the trust equation but adds obligations to the ingestion layer. Three controls carry most of the weight.

• Field-level masking at the connector. Classify columns and document fields during the first crawl using a schema registry plus a lightweight PII detector. Replace detected values with deterministic tokens so joins still work downstream, and keep the mapping in a separate KMS-scoped store that the retrieval path never reads.
• Per-tenant embedding namespaces. Even with RLS, a shared embedding model trained or fine-tuned on cross-tenant data is a leakage risk. Keep tenants in logically separate collections or use a tenant-salt in the embedding input, and never fine-tune a shared retrieval model on raw tenant corpora.
• Query-time identity propagation. The caller's identity, groups, and tenant must travel with the request from the application through the retriever to the reranker. A single service account that "has access to everything" turns every bug into a breach.

The downside of getting this wrong is now priced in public reporting. The 2026 IBM Cost of a Data Breach report put the global average at $4.99 million and the US average at $11.5 million, with mean time to identify and contain a breach at 247 days. Internal search that bypasses the source systems' ACLs is a direct contributor to that number, because retrieval pipelines are a fast way to turn a single misconfiguration into a corpus-wide exposure.

Operational Shape of the Pipeline

A workable internal-search stack has roughly six moving parts, and each has an owner:

1. Connectors with ACL preservation and incremental sync, one per source system.
2. A normalization layer that produces canonical chunks with stable IDs, ACL metadata, and PII tokens.
3. Embedding workers that write to tenant-scoped collections in a vector store. The vector database market is projected to grow from $3.02B in 2025 to $8.71B by 2030, which reflects how central this tier has become.
4. A hybrid retriever that enforces ACL filters before scoring and fuses sparse and dense results.
5. A reranker tier with cost-based routing between local and hosted models.
6. An audit log that records query, caller, retrieved chunk IDs, and the ACL decision for every request.

The audit log is the control that most teams underweight. It is also the one that regulators, security reviewers, and the engineer debugging a complaint at 2 a.m. all actually need. Teams exploring the broader design pattern will find more context in the internal search thesis and in the proprietary infrastructure argument.

Where This Leaves Operators

Internal search is not a thinner version of web search. It is a different workload with different failure modes, and the correctness bar is set by whichever source system has the strictest permissions. The pattern that holds up is boring on purpose: connector-based ingestion with ACLs preserved on every chunk, row-level access control at the retrieval boundary, hybrid retrieval for the queries users actually type, and a BYOK posture that keeps the model vendor out of the trust path. Teams that treat the connector layer as the interesting problem, rather than the vector store or the model, tend to ship systems that pass a security review on the first pass. Teams that do not, tend to spend a quarter rebuilding after one.