Long Context Is a Substitute for an Index, and the Invoice Arrives Every Turn
A cost model for choosing between long context and retrieval: per-turn billing arithmetic, a worked comparison on published list prices, three architectures across five axes, and acceptance criteria to set before you evaluate.
A million-token context window is not an architecture. It is a billing arrangement. When a team decides to paste the corpus into every prompt instead of building an index, it has not eliminated the retrieval layer — it has moved that layer's cost from a fixed build-and-serve line item into a variable charge that recurs on every turn of every conversation. The corpus is paid for once when you index it. It is paid for again, and again, and again when you resend it.
That distinction matters most to the people who sign the invoice and the people who own the latency budget: CTOs and VPs of engineering choosing a platform direction, data architects deciding where governance lives, and CFOs who see the monthly line move without a corresponding change in headcount or volume. The rest of this piece supplies the arithmetic to make that decision explicitly: a cost formula you can populate with your own numbers, a worked comparison built on published per-token prices, a comparison of three architectures across five operational axes, acceptance criteria to write down before you evaluate, and a short FAQ scoped to the evidence.
The Window Grew; the Meter Did Not Shrink
The current narrative runs roughly like this: context windows now exceed a million tokens, so retrieval is legacy infrastructure. The inference is a category error. A context window is a capacity limit — how much input a model will accept in one request. A cost structure is a different object entirely, and the two are set independently.
The published pricing pages make the separation concrete. On OpenAI's API, GPT-6 Astra is listed with a 1,050,000-token context window and a standard input rate of $10.00 per million tokens. The same page states that prompts with more than 272K input tokens are priced at 2x input and cache rates and 1.5x output for the full request. That is not a surcharge on the excess tokens. It is a multiplier applied to the whole request once the threshold is crossed. A 300K-token prompt does not pay the long-context rate on the 28K tokens above the line; it pays it on all 300K.
Caching changes the rate, not the structure. OpenAI's prompt caching guide describes cached input as discounted up to 95%, and the Astra pricing table lists cached input at $1.00 per million tokens against $10.00 uncached. But the guide also states plainly that cached input tokens still count toward tokens-per-minute limits, and that cache hits require an exact prefix match. Anthropic's prompt caching documentation makes the same mechanism explicit: cache hits require 100 percent identical prompt segments up to and including the block marked with cache control, and modifying tool definitions invalidates the entire cache.
So the accurate statement is narrower than the narrative. A large window lets you send a large prompt. It does not change the fact that you are billed for what you send, on every request, and that the discount for reuse depends on the prompt prefix staying byte-identical. Those are properties of a meter, not of an index.
The Cost Shape Is the Argument
The economic claim is simple enough to state in one line and specific enough to check: context-only cost scales with turns multiplied by tokens per turn, while index cost scales roughly with corpus size plus a small per-query retrieval charge.
Illustrative arithmetic on cited GPT-6 Astra list prices ($10.00 per million input tokens, $1.00 cached, 1.25x 30-minute cache write), not observed billing. Assumes a fixed 200,000-token working set — below the 272K long-context threshold, so no multiplier applies — one byte-identical prefix every turn, no eviction, and no output tokens. Cache line: $2.50 write on turn 1, then $0.20 per read.
The evidence is the published rate card. Input tokens are billed per request at a per-million rate, with a documented multiplier above a stated threshold and a discounted rate for cached prefixes. The warrant follows directly: a conversation that resends the corpus on every turn pays the corpus rate on every turn, because each request is a new billing event. An index pays to embed and store the corpus once, then pays to retrieve a small slice per query and to send only that slice to the model.
The implication is that the two curves cross, and the crossing point is computable before you commit. Write the comparison as monthly cost:
Context-only:
Monthly cost = T × 30 × (C × r in + Q × r out)
where T is turns per day, C is corpus tokens sent per turn, Q is output tokens per turn, and r in and r out are the per-token input and output rates. If C exceeds the provider's long-context threshold, replace r in with the multiplied rate for the full request.
Index-based:
Monthly cost = B + S + T × 30 × ((R + H) × r in + Q × r out)
where B is the one-time or periodic index build cost, S is the fixed serving cost of the retrieval endpoint, R is retrieved tokens per query, and H is conversation history tokens carried per turn.
The structural difference is in what multiplies. In the first formula, C — the whole corpus — sits inside the per-turn term. In the second, only R + H does, and R is typically one to two orders of magnitude smaller than C. That is the entire argument, and it does not depend on any particular vendor's numbers. It depends on which quantity is inside the loop.
A Worked Comparison You Can Recompute
The following is an illustrative inference built on published list prices. It is not observed spend, not a benchmark, and not a prediction of any vendor's invoice. Substitute your own inputs before using it.
Illustrative monthly inference from the article's stated workload — 2,000 turns per day, 400,000-token corpus, 500 output tokens per turn, 2,000 history tokens, 8,000 retrieved tokens, 30-day month — using GPT-6 Astra list prices ($10.00 input, $1.00 cached input, $50.00 output per million tokens) and the documented 2x input / 1.5x output multiplier above 272K input tokens. Index-based figure adds an assumed $600 monthly serving cost and $1,000 monthly rebuild allowance. Not observed spend, not a benchmark, and not a prediction of any vendor's invoice.
Stated workload. 2,000 turns per day across all users. A 400,000-token corpus. Average output of 500 tokens per turn. Conversation history carried at 2,000 tokens per turn. Retrieval returns 8,000 tokens per query. Model priced at $10.00 per million input tokens and $50.00 per million output tokens, with the documented 2x input multiplier above 272K tokens. Thirty-day month.
Context-only, no cache. Every turn sends 400,000 corpus tokens plus 2,000 history tokens. Because 402,000 exceeds the 272K threshold, the full request is billed at 2x input.
Input per turn: 402,000 × 10.00 / 1,000,000 × 2 =8.04
Output per turn: 500 × 50.00 / 1,000,000 × 1.5 =0.0375
Per turn: $8.0775
Monthly: 2,000 × 30 × 8.0775 ≈ 484,650
Context-only, with a warm cache on the corpus prefix. The corpus is cached at $1.00 per million tokens, still subject to the 2x long-context multiplier on cache rates. History and output are billed normally.
Cached corpus per turn: 400,000 × 1.00 / 1,000,000 × 2 =0.80
Uncached history per turn: 2,000 × 10.00 / 1,000,000 × 2 =0.04
Output per turn: $0.0375
Per turn: $0.8775
Monthly: 2,000 × 30 × 0.8775 ≈ 52,650
Index-based. Retrieval endpoint priced from a published example: Databricks lists a Standard AI Search configuration at 3 units for 5M vectors, 720 hours, US East, at $605 per month. Assume a comparable serving cost of roughly $600 per month for a corpus of this size, plus a periodic rebuild allowance of $1,000 per month for embedding and reindexing.
Fixed: $600 + $1,000 = $1,600
Input per turn: (8,000 retrieved + 2,000 history) × $10.00 / 1,000,000 = $0.10
Output per turn: $0.0375
Per turn: $0.1375
Monthly: 1,600 + 2,000 × 30 ×0.1375 ≈ $9,850
The gap between the uncached and cached context-only figures is the point most often missed in this debate. Caching is a real and large lever — roughly a 9x reduction in this workload. But the cached context-only figure is still more than five times the index-based figure, because the corpus remains inside the per-turn loop even at a discounted rate.
Sensitivity. At 200 turns per day, the uncached context-only cost falls to about $48,000 per month and the cached figure to about $5,300, while the index-based figure falls to about $2,400. The index still wins on cost, but the absolute gap is small enough that operational simplicity may reasonably decide the question. At 20,000 turns per day, the uncached figure reaches roughly $4.8 million per month and the index-based figure roughly $84,000 — a gap that no amount of caching closes. The most dangerous case is a poor cache hit rate: if conversational turns rewrite the prefix or tool definitions change mid-session, the cached figure collapses back toward the uncached one, and the team discovers this on the invoice rather than in the design review.
Three Architectures, Five Axes
The three architectures are not points on a quality scale. They are different mechanisms with different cost shapes and different failure modes. The table states mechanism facts where the documentation supports them and reasoned inferences where it does not.
Axis | Context-only | Index-only | Hybrid (routed) |
|---|---|---|---|
Cost shape | Linear in turns × corpus tokens per turn; long-context multiplier applies to the full request above the provider's threshold | Fixed in corpus size (build + serving) plus small per-query retrieval tokens | Mostly index cost, plus context cost on the minority of queries routed to full context |
Latency | Grows with prompt length; prefill work scales with input tokens | Retrieval adds a round trip; generation is short. Published vector search examples show roughly 20–50 ms on in-memory endpoints and about 300–500 ms on storage-optimized endpoints at scale (Databricks engineering) | Retrieval latency on the common path; full-context latency on the routed minority |
Freshness | Whatever is in the prompt is current by construction | Bounded by sync mode: triggered sync is cheaper, continuous sync costs more for near-real-time updates (Databricks cost management) | Inherits the index's sync mode for the common path |
Provenance | The model can cite position in the prompt, but there is no document identity unless you add one | Retrieved chunks carry document IDs and scores, which can be surfaced to the user | Provenance available on the routed path; must be preserved when falling back to full context |
Access control | Enforced only by what you choose to include in the prompt; the model sees everything sent | Enforced at retrieval time by filtering the candidate set before it reaches the model | Access control must be applied consistently on both paths, which is the main engineering risk |
The hybrid row is the one that gets misread as a compromise. It is not. It is a routing decision, and the routing rule is the design. The EMNLP 2024 comparison by Li et al. proposes exactly this: a self-reflection step that sends a query to retrieval when the retrieved chunks suffice and escalates to full context only when they do not. In their results, 82% of queries for one model were resolved at the retrieval step, and the routed approach reduced cost by 65% for Gemini-1.5-Pro and 39% for GPT-4o while keeping performance comparable to full long context (Li et al., EMNLP 2024).
The access-control row deserves emphasis because it is where context-only architectures fail quietly. A retrieval filter that excludes documents a user is not entitled to see is a control point. A prompt that contains the whole corpus has no such control point unless the corpus was filtered before assembly — at which point you have built a retrieval layer and called it something else.
What the Evidence Actually Says About Accuracy
The strongest objection is not about cost. It is that long context is more accurate, and accuracy is worth paying for. That objection deserves a direct answer, and the answer is that the evidence is more conditional than either camp usually admits.
The most cited direct comparison is Li et al., published in the EMNLP 2024 Industry Track. Its abstract states that when resourced sufficiently, long context consistently outperforms RAG in terms of average performance, and that RAG's significantly lower cost remains a distinct advantage. The paper also reports that predictions from the two approaches were identical for over 60% of queries — and notes that identical predictions are not necessarily correct, since the two methods tend to make similar errors as well as similar correct answers. That last detail is the one that matters for a routing decision: if the majority of queries produce the same answer either way, paying full-context prices on all of them buys nothing on the majority.
The scope of that study is specific. It benchmarks three models across a subset of LongBench and ∞Bench datasets, filtered to English, real, query-based tasks. It is not a claim about every model, every domain, or every corpus.
The position-dependence finding comes from a different line of work. Liu et al.'s "Lost in the Middle" study found that performance is often highest when relevant information appears at the beginning or end of the input context and degrades significantly when the model must access information in the middle, even for explicitly long-context models (Liu et al., 2023). That study measured multi-document question answering and key-value retrieval; it did not measure cost, and it did not compare against a retrieval pipeline.
The effective-context finding comes from a third. Leng et al. varied total context from 2,000 to 128,000 tokens across three domain-specific datasets and reported that only a handful of the most recent state-of-the-art models maintain consistent accuracy above 64k tokens, with most open-source models handling effective RAG tasks up to roughly 16k–32k tokens (Leng et al., 2024). Their datasets were Natural Questions, DocsQA, and FinanceBench; the failure patterns they observed may not generalize.
These studies do not rank against each other, and treating them as a scoreboard is a mistake. They measure different things under different conditions. What they jointly support is narrower and more useful: accuracy is an empirical property of your workload, your model, and your document layout. It is not a settled property of either architecture, and it cannot be assumed from a context window's advertised size. The only way to know whether full context beats retrieval on your queries is to run both on a held-out set and measure.
Acceptance Criteria Before You Choose
The purpose of writing criteria down before the evaluation is to prevent the result from being rationalized after the invoice arrives. A team that decides what "acceptable" means only after seeing the numbers has not made a decision; it has made a justification.
Measurable acceptance criteria. Each of these should have a numeric threshold and an owner before testing begins.
Cost per resolved query at projected volume. Not cost per token, and not cost per request. Divide total monthly spend by the number of queries that produced a correct, accepted answer. This is the only cost figure that survives contact with a real workload.
p95 latency, end to end. Measured from user submit to first complete answer, at production concurrency, not at single-user load.
Freshness SLA. The maximum acceptable age of the newest document the system can answer from. This determines whether triggered sync is sufficient or continuous sync is required, and the two carry different costs.
Provenance requirement. Whether every answer must cite a specific document and passage, or whether a correct answer without citation is acceptable. This is a product decision with architectural consequences.
Access-control granularity. Whether filtering happens at the document, section, or row level, and whether the same rule applies on every path the query can take.
Quality threshold on a held-out set. A minimum accuracy figure on queries the team did not use during development, evaluated identically for each architecture.
Pre-commitment items. Write these down and date them.
The workload assumptions: turns per day, corpus size in tokens, average output length, concurrency.
The crossover turn count at which the cheaper architecture flips, computed from the formula in the previous section.
The cache-hit assumption, stated as a percentage, with a note on what breaks the prefix — tool definition changes, mid-conversation system messages, and compaction are all documented invalidators.
The condition under which the team will revisit the decision: a volume threshold, a price change, a model change, or a quality regression below the held-out threshold.
A checklist that cannot be filled in with numbers is not a checklist. If the team cannot state its projected turns per day, that is the first thing to measure, and it is measurable before any architecture is chosen.
The Counterargument Worth Taking Seriously
The opposing case is not weak, and it should be stated at full strength. Token prices have fallen substantially over the past two years. Prompt caching discounts input by up to 95% on some models. Context windows have grown past a million tokens. And operating an index is itself a system with costs and failure modes: stale indexes, sync lag between source and index, embedding drift when the model behind the embeddings changes, and the operational burden of running a retrieval service with its own on-call rotation.
The response is that caching changes the multiplier, not the shape. A cached token is cheaper, but it is still billed, still counts toward rate limits per OpenAI's documentation, and still requires an exact prefix match per Anthropic's. Conversational workloads break exact prefix matches routinely: a tool definition changes, a system message is inserted mid-conversation, or a compaction step rewrites earlier turns. Each of those events is documented as a cache invalidator, and each one returns the workload to the uncached rate for the affected portion of the prompt.
The index-side costs are real and should be counted. A retrieval endpoint that receives no traffic still incurs serving costs, which is why Databricks' own cost guidance recommends identifying unused endpoints and combining low-QPS workloads onto a single endpoint. Continuous sync costs more than triggered sync. Embedding a large corpus is not free. The honest comparison includes all of it — which is why the formula in the earlier section has a B and an S term rather than pretending the index is free.
And there are cases where context-only genuinely wins. Small corpora that fit comfortably under the long-context threshold without triggering the multiplier. Short sessions where the corpus is sent once or twice rather than fifty times. Low concurrency, where the absolute monthly figure is small enough that operational simplicity dominates. And workloads where the corpus is the conversation — a coding assistant working on a single repository, a document review task on one contract — so freshness is trivially satisfied because there is no separate corpus to keep current. In those cases the index is overhead, and the right answer is to skip it.
Where This Leaves the Decision
Choose the architecture whose cost curve matches your workload's shape, and write the crossover point down before you build.
The artifacts to carry into the next architecture review are the two formulas and the pre-commitment checklist. Populate the formulas with your own turns per day, corpus size, and cache-hit assumption. Compute the crossover turn count. Then compare it against your projected volume and see which side of the line you are on. If you are far from the crossover in either direction, the decision is easy and the debate is over. If you are near it, the decision is close enough that the operational factors — freshness SLA, provenance, access control — should decide, and the checklist is where those live.
The one thing not to do is treat the context window's size as an answer. It is a capacity figure. The invoice is the cost figure, and it arrives every turn.
FAQ
Does prompt caching make long context free?
No. Cached input is discounted, not zero. OpenAI's pricing lists cached input at $1.00 per million tokens against $10.00 uncached for GPT-6 Astra, and its caching guide states that cached tokens still count toward tokens-per-minute limits. Anthropic's documentation states that cache hits require 100 percent identical prompt segments and that modifying tool definitions invalidates the entire cache. A discount on a recurring charge is still a recurring charge.
Is RAG obsolete?
No. Li et al. (EMNLP 2024) found that when resourced sufficiently, long context consistently outperformed RAG on average, while RAG's significantly lower cost remained a distinct advantage. The same study found the two approaches produced identical predictions on over 60% of queries. The practical reading is that retrieval remains a viable default for the majority of queries, with full context reserved for the cases where it measurably helps.
Does a larger context window mean I can skip the index?
It means you can send more tokens per request. Whether you should depends on how many requests you send and how large the corpus is. Compute the monthly cost both ways at your projected volume before deciding; the crossover is arithmetic, not opinion.
What breaks a prompt cache?
Per Anthropic's documentation, changes to tool definitions invalidate the entire cache, and cache hits require exact matching of all content up to the cache breakpoint. Per OpenAI's guide, compaction replaces earlier conversation content and can prevent reuse from the first changed token onward. Any of these events returns the affected portion of the prompt to the uncached rate.
Previous posts
1. Medallion Architecture: What Changes Between Bronze, Silver, and Gold? — https://singdata.com/trending/medallion-bronze-silver-gold
2. How Much Data Preparation Is Hidden in Your BI Reports? — https://singdata.com/trending/hidden-data-prep-in-bi-reports
3. How to Migrate from ETL to ELT Without Breaking Reporting or Agentic AI Freshness — https://singdata.com/trending/2026-11-04-elt-file-first-ingestion