Hybrid Vector and Keyword Search in a Lakehouse: The Complete Architecture and Evaluation Guide
How to run hybrid vector and keyword search inside a lakehouse: dual indexing, filter pushdown, RRF versus score fusion, index freshness, retrieval evaluation, cost drivers and a phased adoption sequence.
The decision in front of you is not whether to add vector search. It is how to run semantic and lexical retrieval over data that already lives under your lakehouse's governance, permissions and refresh model, without standing up a parallel copy of the corpus in a separate search system that nobody owns. This guide works through that decision in the order a design review will raise it: what hybrid retrieval actually is, how a hybrid query executes, how to architect it inside the lakehouse, how to fuse two ranked lists, how to keep indexes fresh, how to evaluate retrieval quality before you commit, where the cost comes from, and what a phased adoption sequence looks like. It is written for the architects, data platform leads and procurement reviewers who will sign off on the design.
What Hybrid Retrieval Is and Why One Method Is Not Enough
Hybrid retrieval runs a lexical query and a dense-vector query over the same corpus in parallel, then merges the two ranked lists into a single result set. The lexical path scores documents with a term-weighting function such as BM25, which ranks by query-term occurrence, rarity across the corpus and document length; the vector path scores by embedding similarity. Neither path is a superset of the other, and the difference is mechanical rather than a matter of tuning.
Lexical search wins where the query contains a token that must match exactly. Azure AI Search's hybrid overview puts product codes, highly specialized jargon, dates and people's names in that category, and credits vector search with finding conceptually similar content even when no keyword appears in the inverted index. Databricks makes the complementary point from the embedding side: a pre-trained embedding model was trained on external data and has no explicit knowledge of your data, while a keyword index trained on your corpus knows your names, product keys and identifiers (Databricks hybrid search GA).
Each path therefore has a characteristic failure. A lexical-only system misses the user who asks about "sneakers" when the document says "shoes" — the Balancing the Blend study describes exactly this synonym gap as the primary limitation of full-text search. A vector-only system misses the user who pastes an internal SKU, a contract number or a version string that the embedding model never saw during pre-training. Hybrid retrieval is risk reduction against both failure modes, not a novelty.
How a Hybrid Query Executes: Two Paths, One Result Set
A hybrid query is a pipeline, and each stage has an architectural consequence. Query text arrives; the same text is tokenized for the lexical path and embedded for the vector path; each path retrieves its own candidate set of size k; filters are applied; the candidates are fused into one ranked list; an optional reranker rescores the fused list; the top-n results return to the application.
The two paths score with different functions and produce scores on different scales. Azure documents the contrast directly: full-text scores use BM25 and have no upper limit, while vector scores from HNSW sit in a bounded range that depends on the similarity metric — 0.333 to 1.00 for cosine, 0 to 1 for Euclidean and dot product (Azure hybrid ranking). Fusion is the step that reconciles them, which is why the choice of fusion method matters more than it first appears.
Latency follows the same parallel structure. Because the fusion step must wait for every path to complete, a system's P99 tail latency is set by its slowest component, not by the average of its paths — the Balancing the Blend authors state this as a consequence inherent to parallel execution, and measure a four-path configuration at 3.7× the latency of a full-text-only path on one dataset. Adding a path buys accuracy with tail latency, and the tail is what your users experience under load.
Architecting Hybrid Search Inside the Lakehouse
The architecture question is where the index lives relative to the governed data, and how the two retrieval paths stay comparable.
Shared Chunk Identity and Filter Pushdown
Two details break more hybrid implementations than any fusion choice. First, chunk identity: if the lexical and vector paths index different chunk boundaries, a document that ranks first in both lists appears as two unrelated rows, and fusion treats them as separate candidates — the same content competes with itself and neither copy accumulates the combined score. Derive both indexes from one chunking pass with a stable, deterministic chunk ID, and make that ID the join key at fusion time.
Second, filter ordering. A filter must be pushed into both retrievers before top-k truncation. If the vector path returns its k nearest neighbours unfiltered and you discard non-matching rows afterwards, a tenant-scoped query can return fewer than k results, or worse, leak candidates across tenants into a reranker. Azure's own documentation notes that filters can be applied at the beginning or end of query processing and recommends testing which behaviour suits your queries — but for multi-tenant isolation, pushdown is the safer default.
Co-Located Index Versus Dedicated Search Tier
Constraint | Co-located index | Dedicated search tier |
|---|---|---|
Permission model | One; inherits table grants and row-level security | Second model to provision and audit |
Freshness | Tied to table commits; incremental refresh | Depends on a sync pipeline you operate |
Latency tuning | Bounded by the lakehouse engine | Mature, independently tunable |
Operational surface | One system to run | Two systems, plus the pipeline between them |
Data copies | None beyond the index structures | A second copy to govern and reconcile |
Read the table against your own constraints rather than as a verdict. Co-location is the stronger fit when tenant isolation, row-level security and audit trails are the binding requirements, because the index inherits the controls you already enforce. A dedicated tier earns its place when sub-100 ms serving latency at high query concurrency is the binding requirement and the corpus is stable enough that a sync pipeline is not a constant source of drift.
Fusion Methods: Reciprocal Rank Fusion Versus Score Normalization
Fusion has two families, and the choice between them is a choice about what you trust: rank position or score magnitude.
Databricks-reported result on an internal dataset designed to represent its customers' datasets: chunks needed to reach recall 0.9 were 50 for pure dense retrieval and 40 for hybrid search. Recall here means the answer chunk appears in the top num_results. Vendor-reported, not a general benchmark. Source: Databricks, Hybrid Search GA.
When Rank-Based Fusion Is the Right Default
Reciprocal Rank Fusion scores each document as the sum of 1/(k + rank) across the lists it appears in. In the original RRF paper, Cormack, Clarke and Büttcher fixed k = 60 during a pilot investigation and did not alter it during subsequent validation; Azure documents the same 1/(rank + k) form and notes that experiments show the algorithm performs best with a small k such as 60. Because RRF reads only rank position, it is indifferent to the fact that BM25 scores are unbounded while cosine similarity sits in a narrow band — no normalization, no training, no calibration per corpus.
The paper's claim is scoped to what it tested: across the TREC collections and the LETOR 3 dataset used in that study, RRF outperformed Condorcet Fuse, CombMNZ in all but one case, and the best individual ranking, with the exception of one TREC run that was produced with a human in the loop. That is a strong result on those collections, not a guarantee for your corpus.
The cost of RRF is that it discards magnitude. A document that is a near-perfect vector match and a weak lexical match scores much like a document that is moderate on both. For most retrieval applications that trade is acceptable, and when one path is noisy it is actively desirable.
When Score Normalization Earns Its Complexity
Weighted score fusion normalizes each path's scores to a common range and combines them with weights you choose. It earns the calibration effort in three situations: one path is demonstrably more reliable for a query class you care about, such as exact identifier lookup; you hold relevance judgments dense enough to tune weights against without overfitting; or you need a minimum-score threshold to suppress low-confidence results, which requires comparable scores to threshold on.
The constraint to plan for is that normalized scores are comparable only within one corpus and one embedding-model version. Change the model and the score distribution shifts, so weights tuned against the old model must be re-validated. The two families are also frequently layered rather than exclusive: Databricks describes its hybrid implementation as RRF-based with scores normalized so the maximum is 1.0, which makes a document scored near 1.0 legible as one that both retrievers rated highly.
Whichever method you pick, assess each path alone first. The Balancing the Blend study reports a "weakest link" effect in which a weak path can substantially degrade overall fused accuracy, sometimes leaving the hybrid below its best constituent path.
Keeping Indexes Fresh as Lakehouse Data Changes
Freshness is where a hybrid index stops being a design diagram and becomes an operational commitment. The question is not whether the index can be rebuilt but what it costs to keep it current, and what a reader sees while it is being rebuilt.
Incremental Embedding and Change Detection
Incremental embedding means re-embedding only rows that were inserted or updated since the last refresh. Snowflake's Cortex Search documentation describes the mechanism and the billing consequence together: embeddings are computed each time a row is inserted or updated, processed incrementally in the evaluation of the source query, so embedding cost is incurred only for added or changed documents. The same page notes that if no changes are identified during a refresh, virtual warehouse credits are not consumed because there is no new data to refresh.
The expensive case to plan for is the one incremental logic cannot help with: a change to the embedding model or the indexed schema forces a full re-embed of the corpus. Budget for it as a discrete project, not as a refresh.
Atomic Index Publication and Query Consistency
Readers must see either the old index or the new one, never a half-built state. The pattern that delivers this is to build the new index version alongside the live one, catch up the writes committed during the build, and publish with a single metadata commit. Onehouse describes exactly this for vector search on open table formats: the rebuild creates a new index version alongside the current one, picks up table writes made during the rebuild, and a single metadata commit makes it available, so queries use the current version until the new one is online atomically.
The lakehouse supplies the consistency primitive underneath. Apache Iceberg documents serializable isolation, meaning table changes are atomic and readers never see partial or uncommitted changes, plus time travel for reproducible queries against an exact snapshot (Iceberg documentation). Pin an index build to a snapshot and the build has a well-defined read point even while writers keep committing.
Evaluating Retrieval Quality Before You Commit
Retrieval quality is measurable before production, but only if you choose the metric to match the decision and measure the paths separately before you measure the blend.
Building a Query Set That Reflects Real Traffic
Sample from actual query logs, then stratify. At minimum, separate exact-identifier queries (where the lexical path should win), natural-language questions (where the vector path should win), multi-hop questions that need several chunks, and filtered queries that exercise tenant or date constraints. Include the long tail rather than only the head, and hold out a slice you do not look at until the final comparison.
The trap is evaluating on queries the team wrote. Those over-represent well-formed questions with known vocabulary and under-represent the malformed, abbreviated and identifier-heavy queries that dominate real traffic — which is precisely the population where the two paths diverge.
Reading the Metrics: Per-Path, Per-Class, Per-Depth
Read the numbers in three cuts. First, compare each path alone against the fused result: if fusion does not beat the better single path on your query set, the weaker path is dragging the blend, the "weakest link" effect reported in Balancing the Blend. Second, break results out by query class, because an average hides a lexical path that wins every SKU lookup and a vector path that wins every paraphrase. Third, evaluate at the depth your application actually consumes — top-5 for a chat answer, top-50 for a reranker — since gains at rank 100 do not reach the user. Pair nDCG with a precision-oriented check, because nDCG does not penalize bad documents in the list.
Where the Cost Comes From and What a Buyer Should Verify
Hybrid search bills along five separate axes, and vendor quotes rarely map onto them one-to-one. Decompose any proposal into embedding at ingest, index build and refresh, serving, storage, and query-time candidate work before comparing totals, because a low headline rate on one axis can sit next to an at-rest charge on another.
Databricks Lakebase benchmark on MIRACL English with gte-large-en-v1.5 1024-dimension embeddings: vector index size grows roughly linearly with corpus size. Vendor-reported measurements for one corpus and one embedding model; not a general sizing rule. Source: Databricks, Sizing Lakebase Search.
The Five Cost Centers of a Hybrid Index
Cost center | What drives it |
|---|---|
Embedding at ingest | Tokens embedded; incurred on insert and update, not on unchanged rows |
Index build and refresh | Warehouse compute consumed only when changes exist |
Serving | Often billed per GB per month of indexed data, whether or not queries arrive |
Storage | The materialized index and its intermediate structures |
Query-time vector work | Candidate pool size per path, and cache hit rate |
The serving row is the one buyers miss. Snowflake's Cortex Search documentation states that serving compute is incurred per GB per month of uncompressed indexed data "even if no queries are served during a given period." An idle corpus is not a free corpus.
A Buyer's Verification Checklist
Copy these into the RFP and require a written answer for each:
Which billing dimensions apply, in what units, and which are metered at rest?
What triggers a refresh, and is refresh compute billed when nothing changed?
What does a full re-embed cost after an embedding-model change, and who pays for it?
Is the per-path candidate pool tunable, and what quality loss does a smaller pool cause?
Will the vendor run your own labeled query set in a pilot and report per-class results?
On candidate-pool sizing, there is a vendor-reported reference point: in Databricks' Lakebase sizing study, cutting the pool from 1,000 to 200 candidates preserved 90.5% of top-10 results at one-third the page reads. Treat that as a hypothesis to test on your corpus, not a default.
A Reference Architecture and Adoption Sequence
Assembled end to end, the pipeline is short: governed source tables in the lakehouse, a chunking step that assigns stable row identities, dual indexing over those chunks (lexical plus vector) with filters pushed into both, a hybrid query that fuses the two candidate lists, an optional reranker, and the application on top. Every design decision earlier in this guide lands somewhere in that chain — chunk identity at the join, filter pushdown at retrieval, fusion at the merge, atomic publication at refresh.
The deployment question is where the index lives. Co-locating it with the governed tables gives one permission model, one transaction path, and refresh tied to table commits; a dedicated search tier buys tuning maturity and low-latency serving at the cost of a second copy and a sync pipeline. Singdata's lakehouse is one option for teams that want the index alongside governed data rather than in a separate tier — see the Singdata lakehouse overview and, for the comparison itself, unified search and analytics versus a standalone vector database. Evaluate it against the same criteria you would apply to any vendor: filter semantics, refresh behavior, and whether the candidate pool is tunable.
Adopt in phases rather than all at once:
Pick one corpus and build a labeled query set stratified by query class.
Measure the lexical path and the vector path separately; record per-class results.
Add fusion and confirm it beats the better single path on your data.
Tune candidate pool and refresh lag against measured quality and cost.
Expand to further corpora only after step 3 holds.
Step 3 is the gate. If fusion does not beat your best single path, adding more paths will not fix it — the weakest link will keep setting the ceiling.
Frequently Asked Questions
Do I need both paths, or can I start with one?
Start with one if your queries are homogeneous. If you serve both exact identifiers and natural-language questions, a single path will fail one class, and hybrid is the cheaper fix than two separate systems. Measure each path alone first so you know which class you are losing.
What is the difference between RRF and weighted score fusion in one sentence?
RRF combines rank positions using a reciprocal-rank formula with a constant k, commonly 60, and needs no score normalization or relevance labels, while weighted score fusion normalizes each path's scores and combines them with weights you choose and validate.
How fresh can the index be?
As fresh as your refresh schedule allows. Incremental embedding means only changed rows are re-embedded, so a tight target lag adds refresh compute, though full rebuilds can still occur more often than the soft index-build interval (Snowflake Cortex Search); a model or schema change is the expensive case and forces a full re-embed.
What does hybrid search cost at rest?
It depends on the vendor's billing model. Some bill serving compute per GB per month of indexed data whether or not queries arrive; others bill compute only while it runs. Ask explicitly, because for a mostly idle corpus that difference can be the largest line in the bill.
How many candidates should each path retrieve?
Enough that the fused top-n is stable, and no more. The right number is empirical: shrink the pool until top-n overlap with a larger pool starts to fall, then keep a margin above that point.
Can I use hybrid search with filters and multi-tenancy?
Yes, but treat filter placement as a behavior to test. Pushing the filter into both retrievers before top-k truncation is the safer default for multi-tenant isolation; post-filtering a vector path can return fewer than k results, and a tenant filter applied to only one side can let unfiltered candidates into the fused list.
Is a dedicated vector database required?
No. Vector and BM25 indexes now run inside lakehouse table formats, warehouse services, and Postgres extensions. A dedicated vector database is justified when you need its specific tuning and serving characteristics, not as a default.
Further Reading and Sources
Cormack, Clarke & Büttcher, "Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods" (SIGIR 2009) — the original RRF paper; the k=60 pilot and the TREC comparisons.
Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond — the derivation of BM25, term saturation, and document-length normalization.
Microsoft Learn — Hybrid search ranking (RRF) — the 1/(rank+k) form and the score ranges for BM25, HNSW, and RRF.
Microsoft Learn — Hybrid search overview — the mechanism-level case for combining vector and keyword paths.
Databricks — Hybrid Search GA — the RRF-based implementation and the vendor-reported recall comparison.
Databricks — Sizing Lakebase Search — candidate-pool sizing, cache ratio, and cost measurements.
Snowflake — Cortex Search overview — the cost breakdown, including serving billed per GB per month.
"Balancing the Blend: An Experimental Analysis of Trade-offs in Hybrid Search" — the weakest-link finding and the latency cost of parallel paths.
Järvelin & Kekäläinen, "Cumulated gain-based evaluation of IR techniques" (2002) — the origin of DCG and nDCG.
Lucidworks — Evaluation metrics — the @k convention and the recall bound of k/total relevant.
Apache Iceberg documentation — serializable isolation, snapshot reads, and time travel for consistent index builds.