1h 24m read

TPC-DS Benchmark Performance and Customer ROI: A Buyer's Guide to Reading Results and Building a Defensible Business Case

A practical guide to reading TPC-DS benchmark results: what the standard measures and omits, how to spot official versus vendor-run claims, which scale factor to demand, and how to turn price/performance into a defensible ROI model with replaceable inputs.

TPC-DS Benchmark Performance and Customer ROI: A Buyer's Guide to Reading Results and Building a Defensible Business Case

Last refreshed: November 2026. TPC-DS specification version 4.0.0 and the TPC's Dynamic Pricing Model are current as of this date; re-check the TPC documentation page before relying on any figure in a procurement decision.

Most enterprise data platform decisions now arrive with a TPC-DS number attached: a bar chart, a throughput figure, a price/performance ratio, and a multiple against a competitor. Someone did run a benchmark. The harder question is what that number licenses you to conclude about your own workload, your own bill, and your own return on investment.

TPC-DS is a genuine standard, which is exactly why it deserves careful reading. The Transaction Processing Performance Council defines the schema, the query set, the scale factors, the metric formula, and the disclosure requirements for an official result. Those rules are precise, and the precision cuts both ways: a compliant result is comparable to another compliant result at the same scale factor, and almost nothing else follows from it automatically.

Each section below stands alone. If you are triaging a vendor claim today, start with the disclosure checklist. If you are building a budget case, start with the cost model and the worked ROI example. If you are deciding what to demand in an RFP, start with the scale factor section. Every section applies the same discipline: separate what the benchmark measured from what you need to know, then price the difference.

What TPC-DS Actually Measures — and What It Leaves Out

TPC-DS models the decision support functions of a retail product supplier. The schema aggregates multiple snowflake schemas across three sales channels — store, catalog, and internet — with seven fact tables and seventeen dimension tables, averaging eighteen columns per table. The workload contains 99 distinct SQL queries with randomized substitution parameters, spanning ad-hoc, reporting, iterative OLAP, and data mining query classes, plus a data maintenance phase that inserts, updates, and deletes records to simulate refresh from operational systems.

The benchmark times three phases. The load test measures how fast the system ingests generated flat files, builds constraints and auxiliary structures, and analyzes the loaded data. The power test runs the query set sequentially to capture the experience of a single user. The throughput test runs multiple concurrent query streams, each simulating one user, with a minimum stream count tied to the scale factor — three streams at SF100 rising to fifteen at SF100000. A data maintenance run executes between the two query runs.

The primary metric, QphDS@SF, combines these phases into a single query-throughput figure. The numerator counts the queries executed across both runs and all streams; the denominator sums the two query-run times, the data maintenance time, and a fraction of the load time. Price/performance, expressed as $/QphDS@SF, divides the priced configuration cost by that throughput figure. The specification also defines secondary metrics — load time, power test elapsed time, throughput test elapsed times, and optionally energy per performance — each of which must be reported with its scale factor attached.

What the benchmark does not measure is stated in the specification itself, and buyers should treat these exclusions as load-bearing:

  • Extraction is not modeled. The data maintenance process begins from generated flat files assumed to be the output of an external extraction step. The E in ETL is outside the benchmark boundary.

  • The queries are examples, not your queries. The specification states plainly that the queries chosen are "merely examples of the types of queries that might be used in such an environment, not necessarily actual user queries."

  • No benchmark replicates a specific environment. The specification acknowledges that no benchmark can exactly mimic a particular environment and remain broadly applicable, and warns that relative performance derived from the benchmark does not necessarily hold for other workloads.

  • Concurrency is simulated, not observed. Streams model concurrent users, but the arrival patterns, think times, and query mixes of your actual user population are not reproduced.

The practical consequence: a TPC-DS result tells you how a configured system performed on a standardized, disclosed, reproducible decision-support workload at a stated data volume. It does not tell you how that system will perform on your schema, your query shapes, your concurrency pattern, or your data freshness requirements. Those are separate questions, and the rest of this guide treats them as such.

How to Tell an Official TPC Result from a Vendor-Run Benchmark

The distinction between an official result and a vendor-run benchmark is not cosmetic. It determines whether the number is auditable, whether the configuration is fully disclosed, and whether you can compare it to anything else.

An official TPC-DS result carries a full disclosure report submitted to the TPC and made publicly available. The report must include the executive summary, the numerical quantity summary, diagrams of both the measured and priced configurations, the database load process diagram, DDL scripts, the physical organization of tables and indexes, and an auditor's attestation letter with the auditing agency's contact details. Pricing must cover all hardware and software licenses, warranty coverage, and maintenance costs under the TPC pricing specification. The specification requires that any public reference to a TPC-DS result include four components: the test database size, the QphDS@Size performance metric, the $/QphDS@Size price/performance metric, and the availability date of the complete configuration.

That four-part requirement is your fastest triage tool. A claim that quotes a throughput multiple without a scale factor, a price/performance figure, and an availability date is not presenting a compliant TPC-DS reference, whatever the underlying test may have been.

Vendor-run benchmarks take several recognizable forms:

  • Specification-derived but non-compliant. A vendor runs the TPC-DS query set and dataset but omits phases, alters the metric, or skips auditing, and says so. Alibaba Cloud's Hologres documentation is an honest example: it states that its TPC-DS implementation "does not comply with all TPC-DS requirements" and that "the results cannot be compared with published TPC-DS benchmark results." That disclosure is exactly what a careful buyer wants to see — it tells you the number is an internal performance reference, not a comparable standard result.

  • Partial-metric exports. Some tooling exports a value under a TPC-DS metric name that is not the specification metric. The BenchBox documentation, for instance, discloses that earlier releases exported a geometric mean of power and throughput under the QphDS name, omitting load and data maintenance times, and that such results are not comparable with published TPC-DS results. A metric name is not a metric.

  • Marketing multiples. A ratio against a competitor, presented without the underlying configuration, scale factor, or pricing basis, is a claim about a test you cannot inspect. Treat it as a hypothesis to validate, not a finding.

One further trap deserves emphasis. The TPC's Dynamic Pricing Model, introduced with TPC-DS V4, changes how the price component is calculated — the pricing period is now defined as the load time plus the performance test duration, and costs reflect consumption-based resource allocation rather than fixed configuration pricing. The TPC states explicitly that V4 results are not comparable with any previous versions of the benchmark. If a vendor compares its V4-era number to a competitor's older result, the comparison is invalid on its face.

Which Scale Factor to Demand for Your Data Volume

TPC-DS permits a defined series of scale factors: 100, 300, 1000, 3000, 10000, 30000, and 100000, corresponding to raw data sizes from 100GB to 100TB. No other scale factors may be used for a TPC-DS result. The specification is equally direct about comparability: results at different scale factors are not comparable, due to the substantially different computational challenges found at different data volumes.

Minimum query streams by TPC-DS scale factor Horizontal bar chart of the minimum number of query streams required at each permitted scale factor, from 3 streams at SF100 (100 GB) to 15 streams at SF100000 (100 TB). Larger scale factors require both more data and more concurrent simulated users. Minimum query streams by TPC-DS scale factor. horizontal bar data: SF100 (100 GB) 3; SF300 (300 GB) 5; SF1000 (1 TB) 7; SF3000 (3 TB) 9; SF10000 (10 TB) 11; SF30000 (30 TB) 13; SF100000 (100 TB) 15.Source: The Making of TPC-DS (Figure 12: Minimum Required Query Streams) 2006. Minimum query streams byTPC-DS scale factor SF100 (100 GB) 3 SF300 (300 GB) 5 SF1000 (1 TB) 7 SF3000 (3 TB) 9 SF10000 (10 TB) 11 SF30000 (30 TB) 13 SF100000 (100TB) 15 Source: The Making of TPC-DS (Figure 12: MinimumRequired Query Streams) (2006)

Minimum query streams required at each permitted TPC-DS scale factor, with approximate raw data size. These are minimums, not forecasts of your concurrency; the stream-count series is version-dependent and should be re-checked against the current specification. Source: Nambiar & Poess, VLDB 2006 (Figure 12) and 'The Making of TPC-DS' presentation (Figure 12).

That rule is not bureaucratic caution. The benchmark's data design makes larger scale factors qualitatively different, not just bigger. Fact tables scale linearly while dimension tables scale sub-linearly, so the ratio of fact to dimension data shifts as volume grows. The minimum stream count rises with scale factor, meaning larger systems must not only query more data but serve more concurrent users. Query behavior changes too: the same query template at a larger scale factor may select a different fraction of a skewed dimension, producing a different execution profile.

Choosing the scale factor to demand is therefore a reasoning exercise, not a lookup. Work through these steps:

  1. Establish your raw data volume. Sum the uncompressed size of the fact and dimension tables your analytical workload actually queries. This is your anchor, because TPC-DS scale factors refer to raw data size, and compression changes the internal representation without changing the scale factor.

  2. Round to the nearest permitted scale factor at or above your volume. If you hold 800GB of analytical data, SF1000 is the nearest permitted factor that does not understate your volume. A result at SF100 understates the computational challenge your system will face.

  3. Check the concurrency assumption. The benchmark's minimum stream count at your chosen scale factor is a floor, not a forecast. If your peak concurrent analytical user count exceeds the benchmark's stream count at that factor, the benchmark is testing a lighter concurrency regime than yours, and you should say so when you interpret the result.

  4. Demand the same factor from every vendor you compare. Because cross-factor comparison is invalid, a comparison table that mixes SF1000 and SF10000 results is not a comparison. Require each vendor to report at your chosen factor, or treat the results as separate data points that cannot be ranked against each other.

  5. Ask what the load time implies for your refresh window. Load time is a secondary metric reported at the scale factor. If your operational reality includes a nightly window, the load time at your scale factor is a directly relevant input — and it is reported separately precisely because it matters independently of throughput.

A vendor that cannot or will not run at your scale factor is telling you something useful. Either the result does not exist at that volume, or it exists and is not being shown. Both are reasons to ask for a workload-specific proof of concept before signing.

Why a Benchmark Win May Not Transfer to Your Production Workload

The benchmark literature is unusually candid about the limits of benchmark transfer, and buyers should borrow that candor.

The TPC-DS designers built the benchmark partly in response to tuning exploits in earlier benchmarks. "The Making of TPC-DS" notes that TPC-H has relatively few distinct queries known before execution, allowing engineers to tune optimizers and execution paths to artificially increase performance, and that real warehouses define indexes and summary tables that benchmark constraints may not permit. TPC-DS widened the query set and randomized substitution parameters — the RNGSEED is set from the load end timestamp so parameter values are not known in advance — specifically to blunt that advantage. But the same paper concedes the deeper problem: a DBA with prior knowledge of a benchmark's queries and dataset can tune the system to increase benchmark performance, whereas in real life knowledge of the dataset is limited to a point in time and knowledge of the query set is limited to special types of queries.

That asymmetry is the core transferability risk. Your vendor's engineers have run the benchmark many times. They know which queries are expensive, which auxiliary structures help, and how the data is distributed. You will not have that knowledge of your own workload on day one, and your workload will change as the business changes.

Four specific mismatches deserve a question in every evaluation:

  • Query-shape mismatch. TPC-DS queries are standardized templates. Your workload includes queries shaped by your schema, your business logic, and your BI tool's generated SQL. Ask the vendor to run a sample of your actual queries — including your worst-performing ones — on a representative dataset, and compare the relative performance pattern, not just the aggregate.

  • The extraction boundary. The benchmark starts from generated flat files. Your pipelines start from operational systems, with extraction latency, schema drift, and source-system contention that the benchmark does not model. If your ROI case depends on end-to-end freshness, the benchmark's load and maintenance times are inputs, not answers.

  • Concurrency pattern. Benchmark streams are a controlled simulation. Your users arrive in bursts, run dashboards that re-execute the same heavy query, and compete with scheduled ETL for the same compute. Ask how the platform isolates or prioritizes workloads, and test it under a realistic concurrency mix.

  • Data skew and cardinality. TPC-DS deliberately includes skew — seasonal sales, frequent names, skewed state distributions — but your skew is different. A platform that handles the benchmark's skew well may still stumble on a dimension with a distribution your business actually has.

None of this makes TPC-DS useless. It makes it a screening instrument. A strong result at your scale factor establishes that the platform can execute a demanding, standardized analytical workload at your data volume with disclosed configuration and pricing. That is a meaningful filter. It is not a prediction, and a business case that treats it as one is building on a foundation the benchmark never claimed to provide.

A Disclosure Checklist for Reading a Vendor Benchmark Claim

Use this checklist on every benchmark claim that reaches your desk. Each item is answerable — from the vendor's published materials, from the TPC's results pages, or by asking the vendor directly. A claim that fails an early item does not need the later ones.

  1. Is the result official? Look for a full disclosure report submitted to the TPC, an auditor's attestation, and the four required reporting components: database size, QphDS@Size, $/QphDS@Size, and availability date. If any is missing, the claim is a vendor-run benchmark and should be labeled as such in your evaluation notes.

  2. Which specification version? Confirm whether the result is under V4 or an earlier version. V4 results are not comparable with earlier results, so a mixed-version comparison is invalid.

  3. What scale factor, and is it yours? Confirm the scale factor matches the one you demanded. Cross-factor comparison is not permitted by the specification.

  4. What is the priced configuration? For an official result, the priced configuration should be reconstructable from the disclosure report. For a vendor-run benchmark, ask for the compute size, storage, region, and pricing basis. A throughput number without a configuration is not a price/performance claim.

  5. Which phases were run? Confirm whether load, power, throughput, and data maintenance were all executed. A result that omits data maintenance or load is not the specification metric, even if it carries the metric's name.

  6. What was the tuning state? Ask whether the result used default settings or tuned configurations, and whether the same tuning effort was applied to every system compared. Asymmetric tuning invalidates a comparison even when both numbers are real.

  7. What is the comparison basis? If the claim is a multiple against a competitor, ask for the competitor's configuration, scale factor, and pricing basis. A ratio without a disclosed denominator is a marketing statement.

  8. What does the vendor say it does not measure? A vendor that volunteers the limits of its benchmark — as the Hologres documentation does — is giving you information you can use. A vendor that presents a derived benchmark as if it were official is telling you something about its disclosure practices.

  9. What is the availability date? The specification requires it. A result for a configuration that is not yet orderable is a forward-looking claim, and your procurement timeline should reflect that.

  10. What is the path from this result to your workload? Ask for a proof of concept on your data and a sample of your queries. The benchmark result justifies running the proof of concept; it does not replace it.

Keep the completed checklist with your evaluation record. When a vendor's claim changes between meetings — a new multiple, a new competitor comparison — the checklist tells you which inputs changed and whether the new claim is comparable to the old one.

How to Convert Benchmark Price/Performance into Your Own Cost Model

A published $/QphDS@SF figure is a ratio, and a ratio is only as useful as the denominator you can substitute into it. The conversion exercise has three parts: mapping benchmark quantities to your cost drivers, adjusting for costs the benchmark price treats differently than your bill does, and stress-testing the result.

Map the quantities. The benchmark gives you throughput (QphDS@SF), load time, power test elapsed time, throughput test elapsed times, and a priced configuration cost. Your cost model needs the same quantities expressed in your units:

  • Throughput → capacity. Your peak analytical query demand, expressed in queries per hour at peak, divided by the benchmark's QphDS@SF at your scale factor, gives a first-order estimate of how many benchmark-equivalent systems you need. Treat this as a floor, because your query mix is not the benchmark's.

  • Load time → refresh window cost. If your refresh window is fixed, the load time at your scale factor tells you whether the platform fits the window. If it does not fit, the cost of extending the window — in delayed reporting, in overtime, in workarounds — belongs in your model.

  • Priced configuration → your bill. The benchmark's priced configuration is a specific set of hardware and software at list pricing under the TPC pricing specification. Your bill will differ: committed-use discounts, reserved capacity, spot pricing, egress, storage growth, and support tiers all move the number. Use the benchmark price as a starting ratio, then replace each component with your contracted rate.

Adjust for what the benchmark price treats differently. The V4 dynamic pricing model is the most important recent change here. Under a consumption-based model, the price component reflects resources used during the pricing period — load time plus performance test duration — rather than a fixed configuration cost. If your platform bills by consumption, the benchmark's V4 price/performance is closer to your billing structure than a fixed-configuration figure would be. If your platform bills by reserved capacity, the benchmark's consumption-based price may understate your cost during idle periods, because you pay for capacity whether or not queries are running. The TPC frames the dynamic pricing model as eliminating the "idle tax" — the cost of maintaining peak-load hardware during low-activity periods — which is precisely the cost a reserved-capacity buyer continues to carry.

Stress-test the result. Build three scenarios: the benchmark ratio applied directly, a scenario with your query mix 50% less favorable, and a scenario with your concurrency 2x the benchmark's stream count at your scale factor. If the business case survives all three, the benchmark result is doing real work in your decision. If it only survives the first, the case is resting on the benchmark's assumptions rather than yours.

A Worked ROI Model with Replaceable Inputs

The model below is an illustrative inference, not observed data and not a vendor claim. Every input is a placeholder you should replace with your own measured or contracted values. The arithmetic is shown so you can audit it and substitute freely.

Illustrative inputs:

Input

Illustrative value

Replace with

Analytical data volume

1TB raw

Your measured raw volume

Peak analytical query demand

2,000 queries/hour

Your measured peak

Benchmark throughput at your scale factor

10,000 QphDS

The result you validated at your scale factor

Current platform annual cost

$600,000

Your contracted annual spend

Candidate platform annual cost

$400,000

Your quoted annual spend

Analyst hours per week on slow-query workarounds

40 hours

Your measured or estimated hours

Fully loaded analyst cost per hour

$75

Your HR figure

Migration and retraining cost, one-time

$150,000

Your project estimate

Planning horizon

3 years

Your planning horizon

Step 1 — Capacity check. Peak demand of 2,000 queries/hour against 10,000 QphDS means one benchmark-equivalent system covers peak demand with a 5x headroom factor. If your query mix is heavier than the benchmark's, reduce the headroom factor; if lighter, increase it. The capacity check tells you whether the platform's throughput is sufficient, not whether it is optimal.

Step 2 — Run-rate savings. Current annual cost of $600,000 minus candidate annual cost of $400,000 gives $200,000 per year in platform savings. This is the most defensible component because both numbers are contracted figures, not benchmark-derived estimates.

Step 3 — Productivity recovery. Forty analyst hours per week at $75 per hour is $3,000 per week, or $156,000 per year at 52 weeks. Apply a realization factor — not all slow-query time is recovered — of 50%, giving $78,000 per year. The realization factor is a judgment call; state it explicitly and defend it with a pilot measurement if you can.

Step 4 — Total annual benefit. $200,000 plus $78,000 equals $278,000 per year.

Step 5 — Net present value over three years. Undiscounted, three years of benefit is $834,000 against a one-time migration cost of $150,000, for a net of $684,000. At a 10% discount rate, the present value of the annual benefit stream is approximately $691,000, and the net present value after migration cost is approximately $541,000. Payback period on the one-time cost is roughly 6.5 months of benefit.

Step 6 — Sensitivity. Re-run with the candidate platform cost 25% higher ($500,000), the productivity realization factor at 25%, and the migration cost doubled ($300,000). Annual benefit falls to $139,000, three-year undiscounted net falls to $117,000, and payback extends to roughly 26 months. If the case still clears your hurdle rate under these assumptions, the decision is robust to the inputs most likely to be wrong.

The benchmark's role in this model is confined to Step 1 and to the credibility of the candidate platform cost. It does not appear in the savings arithmetic, and it should not. A benchmark result that improves by 50% does not improve your ROI by 50%; it improves your confidence that the capacity check passes. Keeping that distinction explicit is what separates a defensible business case from a vendor slide.

Where Singdata Fits in This Evaluation

Singdata publishes TPC-DS-derived performance comparisons at 1TB and 10TB scale factors against Snowflake and Databricks, alongside SSB-FLAT and TPC-H results. Those are vendor-run benchmarks in the sense this guide has defined: they are Singdata's own tests, and they should be read with the disclosure checklist above. The company reports, for example, a 1TB TPC-DS comparison at identical 128-vCore compute with default settings on both platforms, and a 10TB comparison against Databricks. Treat these as Singdata's claims about its own product, to be validated in your environment, not as independent findings.

What the guide's criteria suggest you test with Singdata specifically:

  • Scale factor fit. Ask for results at the scale factor matching your data volume, and ask what the load time and data maintenance times were at that factor, not just the query throughput.

  • Workload transfer. Singdata's architecture centers on a single engine covering batch, streaming, and incremental compute, with a proprietary incremental compute engine. The relevant question for your ROI model is whether incremental compute reduces the work your recurring pipelines actually repeat — a question answerable in a proof of concept on your data, not from a benchmark chart. The incremental processing guide describes how that mechanism is meant to work.

  • Cost structure. Singdata positions its pricing against consumption-based competitors. Map its billing structure onto the cost model above, particularly the idle-period behavior that the dynamic pricing discussion highlighted; the pricing comparison framework sets out the components to compare.

  • Disclosure posture. Ask Singdata the same ten checklist questions you would ask any vendor, and compare the quality of the answers.

The point of this section is not to conclude that Singdata wins. It is to show that the evaluation criteria the guide has built apply to Singdata as readily as to any competitor, and that the company's own published comparisons are inputs to that evaluation rather than substitutes for it.

FAQ

Is a TPC-DS result comparable to a vendor's internal benchmark?

No. The specification permits comparison only between TPC-DS results at the same scale factor, and only official results carry the disclosure and auditing that make comparison meaningful. A vendor's internal benchmark may use the TPC-DS query set and dataset, but unless it follows the specification's phases, metric formula, and disclosure requirements, its results are not comparable to published TPC-DS results — and vendors that run such tests often say so explicitly.

Can I compare a 1TB result to a 10TB result?

No. The specification states that results at different scale factors are not comparable because the computational challenges differ substantially across data volumes. Fact tables scale linearly while dimensions scale sub-linearly, and the minimum stream count rises with scale factor, so a larger-scale result tests a different workload, not just a bigger one.

Does a higher QphDS@SF always mean a better platform for me?

Not necessarily. QphDS@SF measures query throughput on the standardized workload at a stated scale factor. Your workload's query shapes, concurrency pattern, data skew, and freshness requirements may favor a platform that scores lower on the benchmark. Use the benchmark as a screening filter and a proof of concept on your own data as the deciding test.

What changed in TPC-DS V4, and why does it matter for my comparison?

V4 introduced the Dynamic Pricing Model: the pricing period is defined as load time plus performance test duration, and costs reflect consumption-based resource allocation rather than fixed configuration pricing. The TPC states that V4 results are not comparable with any previous version, so a V4 number set against an older result is invalid regardless of how the figures look.

How do I handle a vendor that will not disclose its benchmark configuration?

Treat the claim as unverified and weight it accordingly. The specification's disclosure requirements exist so a result can be independently reconstructed and audited; a vendor withholding configuration, scale factor, pricing basis and tuning state is asking you to accept a number on trust. Ask instead for a proof of concept on your data, which produces evidence you can inspect directly.

Is the benchmark's price/performance figure my actual cost?

No. The priced configuration is a specific set of hardware and software at list pricing under the TPC pricing specification. Your bill reflects contracted rates, discounts, storage growth, egress and support tiers. Use the benchmark figure as a starting ratio, then replace each component with your own contracted values as the cost model section describes.

What should I demand in an RFP?

Demand a result at your scale factor, with all four required reporting components if it is official, or a clear statement of configuration and phases if it is vendor-run. Demand the same scale factor from every vendor you compare, and demand a proof of concept on your data with a sample of your actual queries, because no benchmark result substitutes for that test.

Previous posts

https://www.singdata.com/

https://www.singdata.com/company

https://singdata.com/trending/powerbi-date-difference-dax-calculations/