46 min read

How to Get the Fastest Query Performance on Apache Iceberg in a Cloud Lakehouse

A practical guide to Apache Iceberg query performance in a cloud lakehouse: planning vs. scan costs, engine, catalog, layout and maintenance layers, a benchmark audit protocol, file sizing and compaction targets, and a procurement acceptance checklist.

How to Get the Fastest Query Performance on Apache Iceberg in a Cloud Lakehouse

"Fastest query performance on Apache Iceberg" is not a product attribute you can buy off a datasheet. It is an outcome produced by four interacting layers: the query engine, the catalog and metadata path, the table layout, and the maintenance discipline applied to that layout. Change any one and the same table can move from sub-second to minutes without a single byte of data changing.

This guide is written for CTOs, data engineers, data architects, and procurement teams evaluating cloud lakehouse platforms on object storage with Iceberg v2 and v3 tables and mainstream engines. It covers the mechanisms that determine latency, a reproducible protocol for auditing any vendor's benchmark on your own data, layout and compaction targets with the conditions attached to them, a procurement acceptance checklist, and a calibrated reading of vendor claims — including Singdata's own published figures, which this article treats as unvalidated vendor-reported claims rather than established fact.

Refreshed 14 November 2026. Iceberg, engine, and cloud guidance all move; re-check version-specific defaults before you commit to a configuration.

What "Fastest Iceberg Query Performance" Actually Measures

A query's wall-clock duration is the sum of several distinct costs, and each is governed by a different part of the stack. Iceberg's own design goal is narrow and specific: scan planning that fits on a single node, so that finding the right files does not require a distributed job. That mechanism sets a floor on planning cost. It does not set your execution time, your storage throughput, or your concurrency ceiling.

Planning latency versus scan throughput versus execution

Planning latency is the time spent turning a SQL predicate into a list of files and splits. It is governed by manifest counts, manifest-list filtering, metadata caching, and the number of planning threads the engine allocates — Trino, for example, exposes iceberg.planning-threads and iceberg.split-manager-threads, both defaulting to double the coordinator's processor count.

Scan throughput is how fast those selected files are read: file sizes, how well partition and column statistics pruned the file list, and object-store request latency all bind it. Execution is what the engine does with the rows afterwards — vectorization, join strategy, shuffle volume. A benchmark that reports one number for all three tells you nothing about which one you can fix.

Why a single "seconds" figure hides the bottleneck

A headline duration aggregates conditions that are rarely stated. Was the first run cold, with metadata and Parquet footers fetched from object storage, or warm, with the engine's metadata cache populated? Was it a single query or forty concurrent ones competing for the same coordinator? Trino's iceberg.metadata-cache.enabled defaults to true, and its Parquet footer cache defaults to none — so two deployments of the same engine version can produce materially different numbers on identical data. Report cold and warm separately, or the figure is not comparable to anything.

Where Iceberg Query Speed Comes From: Engine, Catalog, Layout, Maintenance

Treat these four layers as the spine of any performance investigation. Each contributes a distinct mechanism, and each constrains what the others can achieve.

The engine layer: pruning, vectorization, and caching

Iceberg hands every engine the same metadata: a manifest list carrying partition value ranges, and manifests carrying per-file partition tuples plus column-level value counts, null counts, and lower and upper bounds. The engine's job is to exploit it. Iceberg documents that filtering data files by upper and lower bounds at planning time can deliver a 10x performance improvement in some cases by eliminating splits without running tasks. Beyond pruning, engines differ in vectorized execution, cost-based optimization from table statistics (Trino's iceberg.table-statistics-enabled defaults to true), and cache design. Those differences are why the same table can be fast on one engine and slow on another.

The catalog and metadata layer

The catalog — Hive Metastore, Glue, JDBC, REST, or Nessie, all selectable in Trino via iceberg.catalog.type — resolves table names to the current metadata file. The manifest list then acts as an index over manifests, so planning does not require reading every manifest. That design degrades gracefully only while manifest counts stay bounded. A table with thousands of small manifests, or hundreds of retained snapshots, forces more metadata reads per query and shrinks the benefit of any cache.

The table layout layer: partitioning, sorting, file sizing

Layout determines how much metadata pruning can eliminate. Hidden partitioning means predicates on a source column are automatically converted to partition predicates, so users do not need to know the layout to get pruning. Partition evolution lets you change the spec as query patterns shift without rewriting history. File sizing matters too: AWS prescriptive guidance notes that as file counts grow, increasing file size alleviates the expansion of metadata, and recommends setting write.metadata.metrics.column.* to full only for the columns you actually filter on.

The maintenance layer: compaction, snapshot expiry, statistics

Maintenance keeps the other three layers from degrading. Rewriting data files consolidates small files; rewriting position-delete files reduces manifest metadata and the overhead of opening many small delete files. Snapshot expiration and orphan file removal bound metadata growth. The computeTableStats action collects number-of-distinct-values statistics into a Puffin file that engines can use for cost-based optimization. Skip this layer and the engine and layout work you did erodes quietly.

How to Audit a Vendor's Iceberg Benchmark in Your Own Environment

No vendor benchmark, audited or not, answers the question you actually have: how will this perform on my tables, my query mix, and my concurrency? The only reliable answer comes from a controlled test on your own data. The protocol below is designed so that a vendor's published methodology can be checked against it rather than trusted.

The six questions every benchmark must answer

Ask each of these before reading the result. Which engine version and tuning state formed the baseline — defaults or tuned? Was hardware identical on both sides, including local SSD availability, which changes cache behaviour materially? What data volume and scale factor? Which query mix, and does it resemble yours? Were runs cold-cache or warm-cache, and were both reported? And was the result independently audited, or is it vendor-reported? Each answer changes the number, and a benchmark that omits any of them is directional at best.

Designing a fair A/B test on your own data

Freeze a snapshot so both sides read identical data. Capture the table's metadata profile first — file counts, size distribution, manifest counts, snapshot count — because that profile, not the row count, drives planning cost. Run the same query set cold and warm, and measure planning and execution separately rather than as one total. Record concurrency explicitly. The common failure modes are predictable: one side tuned and the other at defaults, queries cherry-picked to favour a strength, and cache state left unreported.

Reading vendor-reported figures without being misled

Classify every claim before you weigh it. An audited result — such as a TPC benchmark run and verified by the TPC Council — sits at the top. A vendor benchmark with published methodology and configurations sits in the middle: you can inspect the scale factor and baseline, even though no third party verified it. An unaudited headline number with no methodology sits at the bottom. Then map the vendor's workload to yours: a 1TB complex-query result says little about your 100GB dashboard workload, and vice versa.

File Sizes, Partitioning, and Compaction Cadence: Targets and Conditions

The numbers below are documented defaults and starting points, not universal rules. Each carries conditions, and each should be validated against your own file-count growth and query predicates before you treat it as a target.

Target file and row-group sizes

Iceberg's write.target-file-size-bytes defaults to 512 MB and write.parquet.row-group-size-bytes to 128 MB. AWS guidance presents the same two values as starting points and notes that files may be written smaller than the target. Note the engine caveat: Trino's iceberg.target-max-file-size defaults to 1GB, so the effective target depends on which engine writes the table. Larger files reduce metadata volume and per-file overhead; they also raise the cost of rewriting any single file, so the right target depends on your update pattern.

Partitioning and sort-order choices

Layout should follow your query predicates, not your ingestion convenience. Partition on the columns your filters actually use, and prefer hidden partitioning so predicates convert automatically. Bucket transforms help when a partition column has high cardinality. Sort orders, combined with Iceberg's column statistics, sharpen file pruning — AWS guidance notes that sorting plus statistics can make pruning more effective, and that compaction is how you apply a sort order to data written before it existed. Partition evolution lets you change the spec without rewriting history, so a wrong early choice is recoverable.

Compaction and snapshot-expiry cadence

Iceberg's rewrite action exposes documented thresholds you can start from: min-file-size-bytes at 75% of target, max-file-size-bytes at 180%, min-input-files at 5, and delete-ratio-threshold at 0.3 (Spark procedures reference). Snapshot expiry defaults to a 5-day maximum age with at least 1 snapshot retained (configuration). Set cadence from observed file-count growth per partition rather than a fixed calendar: instrument the growth rate, then compact when a partition's small-file count crosses your trigger.

Why Planning Latency and Metadata Bloat Matter as Much as Scan Speed

Every query pays a planning cost before it reads a byte. Iceberg's design keeps that cost low by pruning metadata files as well as data files, so planning fits on a single node even for multi-petabyte tables (Iceberg performance). The corollary is that when metadata grows — more manifests, more snapshots, more small files — planning inflates while scan performance looks unchanged. A table that was fast at 10,000 files can become slow at 500,000 without any change to the queries or the engine.

How metadata grows and how it slows queries

Three mechanisms compound. First, small-file proliferation: each write that produces many tiny files adds manifest entries, and AWS guidance notes that as file counts grow you should increase file size to avoid exponential metadata expansion (AWS prescriptive guidance). Second, snapshot accumulation: each commit adds a snapshot and metadata version, and defaults keep snapshots for up to 5 days. Third, manifest fragmentation: Iceberg merges manifests on write by default (commit.manifest-merge.enabled=true, target 8 MB, minimum 100 manifests), but heavy concurrent writes can outpace merging. Each of these increases the manifest bytes a planner must read and reduces the hit rate of any metadata cache.

Symptoms and diagnostics

Watch four signals. Rising planning time with flat scan time points at metadata. Growing manifest counts per snapshot (query the manifest list) indicate merging is behind. Falling metadata-cache hit rates — Trino exposes iceberg.metadata-cache.enabled and a Parquet footer cache — show the planner re-reading files it should have cached (Trino Iceberg connector). A widening gap between p50 and p99 latency under concurrency suggests planning threads are saturated. Each maps to a specific action: rewrite data files, expire snapshots, tune commit.manifest.min-count-to-merge, or raise planning threads.

How to Read Singdata's 4x/10x/TCO Claims

Singdata publishes performance and cost positioning that a buyer should treat as vendor-reported claims, not established facts. The right response is neither dismissal nor adoption: extract the stated configurations, identify what is not disclosed, and convert each claim into a hypothesis your own environment can test. This section does that for the figures most likely to appear in a procurement conversation.

What the claims say and what they don't

Singdata's published comparison reports benchmark durations across SSB-FLAT 100GB, TPC-H 100GB, and TPC-DS 1TB, with a stated note that no special tuning was applied and that all platforms used default settings (Singdata comparison). The same page reports cost figures per workload and positions Singdata as 3x–6x lower in cost than Snowflake. The homepage separately cites a 6x result on TPC-DS 10TB and a 50% TCO reduction (Singdata). What the material does not disclose: third-party audit status, the full query-level methodology, and whether the competitor configurations were tuned beyond defaults. Those gaps are what your acceptance test must close.

Turning vendor claims into your own acceptance test

Map each headline to a testable hypothesis. The TPC-DS duration claim becomes: on your 1TB-shaped workload, at equivalent hardware, does the platform complete the query set faster than your current engine? The cost claim becomes: at your actual query volume and concurrency, what is cost per query? The "no special tuning" caveat becomes a control: run both sides at defaults first, then at each side's best tuning, and report both. The vectorized lakehouse comparison sets out the same six-question protocol and is a useful companion for structuring the test.

Procurement Acceptance Checklist for Iceberg Query Performance

Each item below should be verifiable in a joint test or from documentation, not accepted from a slide. Mark each pass, fail, or not demonstrated.

Metadata and planning checks

  • Planning latency measured on a table with realistic manifest counts (not a freshly loaded table).

  • Metadata caching behavior documented and observable (cache hit rate or equivalent metric).

  • Catalog compatibility confirmed for your current metastore (Hive, Glue, JDBC, REST, or Nessie).

  • Planning threads and split-generation threads configurable.

  • Behavior under concurrent planning demonstrated, not asserted.

Layout and maintenance checks

  • write.target-file-size-bytes and row-group size configurable per table.

  • Compaction tooling available, with documented thresholds and scheduling.

  • Snapshot expiry and orphan-file removal available and schedulable.

  • Table statistics collection (NDV/Puffin) supported for cost-based optimization.

  • Partition evolution supported without full rewrite.

Benchmark and evidence checks

  • Benchmark provenance stated: who ran it, on what hardware, at what scale factor.

  • Query mix published and comparable to yours.

  • Cold and warm cache reported as separate numbers.

  • Third-party audit status disclosed, or explicitly stated as unaudited.

  • Vendor willing to run a joint test on your data with your query set.

FAQ

Does Iceberg itself make queries fast? No. Iceberg provides the metadata structures — manifest lists, partition summaries, column-level bounds — that let an engine prune files quickly. Whether a query is fast depends on the engine exploiting that metadata, the table layout giving it something to prune, and maintenance keeping metadata small. Iceberg is a necessary condition, not a sufficient one.

Is a bigger file always better? No. Iceberg's default target is 512 MB, but the right size depends on your scan patterns and storage throughput. Very large files reduce metadata but can hurt when queries touch narrow slices. Treat 512 MB as a starting point and validate against your workload.

How often should I compact? Set cadence from observed file-count growth, not a calendar. Instrument small-file counts per partition and compact when a partition crosses your trigger. Iceberg's rewrite defaults (75% of target as the minimum, 180% as the maximum, 5 input files) are reasonable thresholds to start from.

Do vendor speed claims transfer to my environment? Rarely at face value. Headline figures aggregate specific hardware, scale factors, query mixes, and cache states. The only reliable way to know is to run the same protocol on your data.

What single metric best predicts user-perceived latency? Planning time plus execution time at your p95 concurrency, measured warm. A single "seconds" figure that hides planning, cache state, and concurrency tells you little about what a user will experience.

Further Reading and Sources

Primary references for the mechanisms and defaults cited in this guide:

Refresh date: 14 November 2026. Re-check engine versions, Iceberg release notes, and vendor methodology pages before relying on any figure; defaults and capabilities change between releases.

Previous posts

https://www.singdata.com/

https://www.singdata.com/company

https://singdata.com/trending/powerbi-date-difference-dax-calculations/