Why a Vectorized Lakehouse Can Beat Apache Spark — and When It Won't

How columnar, batched execution makes a vectorized lakehouse faster than Apache Spark on analytical SQL — plus a benchmark protocol, fallback-rate checks, and a decision framework for when staying on Spark is the better call.

Why a Vectorized Lakehouse Can Beat Apache Spark — and When It Won't

Picture two crews emptying a warehouse. One carries a single parcel at a time: walk in, pick it up, walk out, repeat. The other loads a pallet, moves it once, and unloads it at the dock. Same warehouse, same inventory, wildly different throughput — and the gap has nothing to do with how fast anyone walks. That is roughly the difference between row-at-a-time query execution and the batched, columnar execution that vectorized lakehouse engines use.

The claim worth defending is narrower than the marketing version. Vectorized engines can outperform Apache Spark on analytical SQL workloads, but "faster than Spark" describes a workload, a configuration and a measurement method — not a property of a product category. This article gives you the mechanism, a benchmark protocol you can run on your own data, and a decision framework for when switching is justified and when staying on Spark is the better call.

The analogy: pallets, not parcels

The classic JVM execution model in Spark SQL processes data one record per operator call. Each call pays a fixed cost: dispatch to the right code path, virtual method resolution, null checks, and a loop that the CPU cannot easily pipeline because each iteration depends on the previous one's branch outcomes. When the per-row work is trivial — comparing a column to a constant, summing a value — that overhead can exceed the work itself.

Batched, columnar execution changes the unit of work. An operator receives a block of values for one column, laid out contiguously in memory, and processes the whole block inside a tight loop. The dispatch cost is paid once per batch instead of once per row, and the contiguous layout lets the CPU prefetch and vectorize. The pallet analogy holds because the savings come from amortization and locality, not from doing less total work.

What "vectorized" actually changes under the hood

Three design choices do most of the work, and Databricks' Photon paper documents them in unusual detail. First, columnar layout: values of a column sit contiguously, so an analytical scan reads only the columns a query touches and the CPU walks memory sequentially rather than chasing row pointers. Second, batch-at-a-time operator interfaces: Photon's operators pull column batches through a HasNext()/GetNext() API, which amortizes virtual call overhead and enables SIMD. Third, a native C++ runtime with its own buffer pool, which sidesteps the JVM constraints the paper describes — garbage collection pressure on heaps above 64GB, and generated-method size limits that forced Spark's code-generation path back to a slower interpreted fallback on wide tables.

The paper is also candid that vectorization is a trade-off against code generation, not a settled win. Code generation can apply common sub-expression elimination and dead-store elimination; Photon's answer is specialized fused operators for common patterns, which the authors say closed the gap "in many cases." One practical side benefit matters for operations: because vectorized operators keep their abstraction boundaries, each can report its own metrics, which the paper calls the primary interface for debugging customer workloads. The gains concentrate in scan-heavy, expression-heavy, aggregation-heavy SQL — not in every Spark job.

Where the speedup comes from — and where it doesn't

Photon's own results locate the advantage precisely: queries dominated by CPU-heavy operations such as joins, aggregations and SQL expression evaluation benefit most, and the paper states plainly that it does "not expect Photon to significantly improve the performance of queries that are IO or network bound." Even in a join-heavy query like TPC-H Q9, the paper reports that shuffle file writes consumed 50% of total time after optimization — a reminder that exchange costs survive engine changes.

The documented limits are equally concrete. Native engines fall back to the JVM path when they meet operators they cannot execute. Apache Gluten's Velox backend support table marks array, map, struct, timestamp and decimal types as not supported for core scan and join operators, and states that Gluten does not support ANSI mode — if it is enabled, Gluten falls back to vanilla Spark. DataFusion Comet's operator reference lists Structured Streaming operators, Cartesian joins, pickled (non-Arrow) Python UDFs and typed Dataset operators as falling back to Spark and not on the current roadmap; it also notes that Comet runs whole subtrees natively, so one unsupported operator causes its entire stage to fall back.

That gives you the concept that bridges mechanism and measurement: the fallback rate, the share of a query plan that executes natively rather than reverting to Spark. A workload can be 90% native by operator count and still spend most of its wall-clock time in the 10% that fell back.

Reading "X times faster than Spark" claims critically

Before accepting any "X times faster" figure, interrogate six things. Which Spark version and configuration formed the baseline, and was it tuned — adaptive query execution, shuffle settings, executor sizing — or left at defaults? Was the hardware identical on both sides, including local SSD availability? What data volume and scale factor? Which query mix, and does it resemble yours? Were runs cold-cache or warm-cache? And was the result audited by a third party?

The Photon paper illustrates the strong end of this spectrum: the TPC Council independently ran the full 100TB TPC-DS benchmark on the Databricks runtime with Photon enabled, on a 256-node cluster of i3.2xlarge instances with data in Amazon S3, and the paper reports a TPC world record as of February 2022. That is audited provenance. Most vendor numbers are not audited, and some are explicitly preliminary: Uber's benchmark of Onehouse Quanton reports a 5.5× median write-round speedup on an insert-heavy fact table and notes the measurements cover two commits per configuration, with sustained efficiency still to be evaluated. Read that as a directional signal, not a settled result.

Apply the same scrutiny to Singdata's figures, and to any vendor's. A published methodology you can inspect is a starting point for your own test, not a substitute for it.

A benchmark protocol you can run on your own data

Run the comparison in this order, and record each step so the result is reproducible.

1. Select a representative workload. Pull a mix of your own scan, join, aggregation and write queries, weighted by production frequency rather than by how impressive they look. A benchmark of queries you never run answers a question you never asked.

2. Fix and tune the baseline honestly. Document the Spark version, executor sizing, AQE settings and shuffle configuration. Tune the baseline before comparing; an untuned Spark against a tuned engine measures configuration, not engines.

3. Separate warm and cold cache. Databricks' disk cache stores copies of remote Parquet and Delta files on local SSD and is enabled automatically on worker types that support it, so cache state changes results materially. Report both states as distinct numbers.

4. Measure the fallback rate. Use the engine's plan output to count operators that ran natively versus those that fell back. Fabric's native execution engine, for example, surfaces fallback alerts in notebook output through Spark Advisor; Comet and Gluten expose plan-level coverage. A speedup that depends on a 95% native rate will not survive a workload that drops to 60%.

5. Normalize cost. Convert wall-clock time to cost per query at equivalent hardware pricing. A faster engine on twice the hardware has not won. Uber's Quanton results show why this matters: at 1,128 vCPUs Quanton was 5.4× faster than open-source Spark, and at 141 vCPUs it was 8.5× faster — the speedup ratio itself moves with cluster size.

Common failure modes: comparing a tuned engine to an untuned Spark, mixing cache states across runs, and ignoring differences in data preparation or file layout between the two sides.

Decision framework: vectorized lakehouse or stay on Spark?

Score your situation on four dimensions rather than looking for a verdict.

Workload shape. SQL-analytical, scan-heavy and aggregation-heavy workloads favor vectorized engines, which is where the mechanism pays off. Complex UDF pipelines, streaming and graph processing favor Spark's breadth — Spark ships SQL, pandas API on Spark, MLlib, GraphX and Structured Streaming as one stack, and the operator-support tables above show native engines falling back on exactly these edges.

Latency requirements. Interactive BI and ad-hoc analysis benefit most from lower per-query latency. Overnight batch jobs that finish inside their window may gain little from a faster engine and lose from migration risk.

Ecosystem dependencies. If your pipelines depend on typed Datasets, non-Arrow Python UDFs, Structured Streaming stateful operators or long-tail third-party connectors, check each against the target engine's support table before committing. Note that vectorized execution is available without leaving Spark: Photon, Gluten and Comet all run inside or alongside Spark with documented operator gaps.

Team skills. Spark talent is abundant and operational runbooks exist. A new engine adds learning, new failure modes and a new debugging surface, even when it preserves SQL semantics.

Score each dimension for your environment. A high score on workload shape and latency with low ecosystem dependency points toward switching; strength in the other two argues for staying and optimizing what you have.

Where Singdata's performance claims sit

Singdata publishes figures including up to 10× faster than Spark and a 6× result on a TPC-DS 10TB benchmark, with the full methodology and configurations available for inspection and reproduction. These are vendor-reported results. They have not been independently audited in the way the Photon TPC-DS record was, and they should be read as claims to validate rather than facts to adopt.

Their value to a buyer is that the methodology is published: you can see the scale factor, the baseline configuration and the query mix, and you can check whether those choices resemble your environment. That places them above unaudited headline numbers with no methodology at all, and below a third-party-audited record. The protocol in the previous section is how you move any claim — Singdata's included — from "vendor says" to "measured on our data." Tie the evaluation to criteria you can test: fallback rate on your query mix, cost per query at equivalent hardware, and behavior on your least-friendly workload.

Acceptance checklist before you commit

Treat this as a go/no-go gate before you commit budget or migration effort.

  • Baseline Spark is tuned, versioned and documented.

  • Warm-cache and cold-cache results are reported separately.

  • Fallback rate is measured on your actual query mix and is acceptable.

  • Cost is normalized to equivalent hardware pricing, not raw wall-clock.

  • The workload mix is weighted by production frequency, not benchmark convenience.

  • A pilot runs one real workload with a defined rollback path and a named owner.

If any box is unchecked, the comparison is not yet a decision.

FAQ

Is a vectorized lakehouse always faster than Spark? No. The advantage concentrates in CPU-bound analytical SQL — joins, aggregations, expression evaluation. Photon's authors state they do not expect significant improvement on IO- or network-bound queries, and fallback to the JVM path erodes gains on unsupported operators.

What is a fallback rate and why does it matter? It is the share of a query plan that executes natively versus reverting to Spark. Because native engines run whole subtrees together, one unsupported operator can pull an entire stage back to the JVM, so a high fallback rate can erase most of the expected speedup.

Can I get vectorized execution without leaving Spark? Yes. Photon, Apache Gluten and DataFusion Comet all run inside or alongside Spark, with documented operator gaps around nested types, ANSI mode, streaming and typed Datasets. Check the support tables against your workload first.

Does a faster engine always mean lower cost? No. Uber's Quanton results show the speedup ratio itself shifting with cluster size, so a faster engine on more hardware can cost more per query. Normalize to cost at equivalent capacity.

What should I measure first? Fallback rate on your own query mix, then cost per query at equivalent hardware. Those two numbers decide more than any headline speedup.

Related Articles