Five Thousand Agents on One Repository Force a Storage Rewrite: A Databricks Alternative Buyer's Guide
A buyer's guide to evaluating a Databricks alternative for agent-scale lakehouse workloads: how to characterize concurrent agent writes, test conflict handling and maintenance boundaries, and probe pricing transparency before signing.
GitHub's engineering team has published a detailed account of rebuilding its Git storage layer because agents changed the shape of its write traffic, not merely its size. The numbers are striking: 7.38 billion commits in September alone, more than five times the year-earlier figure, and pushes up 4.9x year over year from 0.69 billion to 3.35 billion per month (GitHub Blog). That case is useful to any team running a lakehouse, because the same pressure — thousands of concurrent writers converging on a small ordered state — is what agent workloads put on a table format's commit path.
This guide treats that pressure as an evaluation problem. It does not argue that any particular platform wins. It sets out how to characterize your own agent workload, which criteria to test against a Databricks alternative, which warning signs deserve investigation, and how to probe pricing transparency before you sign. The final checklist is the gate; everything before it exists to populate that gate with evidence from your environment rather than a vendor's.
Why Agent-Scale Concurrency Stresses Storage Differently
The GitHub case is worth reading closely because it separates two things buyers tend to conflate: how much data is written, and how writes are ordered. Agent workloads change the second far more than the first.
Vendor-run benchmark (LanceDB) on Amazon S3, metadata operations only. Values are active latest-state metadata after 10,000 sequential appends, excluding retained history; no independent replication. Iceberg figures reflect two catalog paths (Postgres via iceberg-rust SqlCatalog; Apache Polaris backed by Postgres). Treat as one vendor's measurements of open-source formats, not a general ranking.
The write pattern changes, not just the volume
A BI or human analyst workload is bursty on the read side and periodic on the write side. Dashboards refresh, scheduled jobs land batches, and the write path sees relatively few, relatively large commits. An agent workload inverts this. GitHub describes an agent in a tight loop as committing or checkpointing after nearly every action, so its speed is bounded by how fast a single push completes — latency a human would never notice becomes the limiting factor. Thousands of agents working on their own branches in one repository produce a sustained write rate that converges on a single point in the architecture (GitHub Engineering).
Map that onto a lakehouse and the convergence point is the table's commit log. Every writer that appends, merges, or checkpoints has to publish a new table version through the same metadata path. The volume may be modest — an agent checkpoint is often small — but the number of ordering events per hour is high and continuous rather than batched.
Retries are the hidden multiplier
Delta Lake documents its concurrency model as optimistic: a write stages its changes, validates them against anything committed since its snapshot, and either commits a new versioned snapshot or fails with a concurrent modification exception rather than corrupting the table (Delta Lake docs). That is the correct behavior, and it is also the mechanism that turns contention into extra load. A failed write is not free; it consumed reads, staged files, and a commit attempt, and the client will typically try again.
The LanceDB metadata benchmark makes the consequence concrete. In its concurrent-writer sweep on S3, at 200 writers Lance and Delta Lake both reported 5.5 successful commits per second, but Lance did so with a 16% failed-attempt rate while Delta Lake had an 88% failed-attempt rate (LanceDB blog). The benchmark's own reading is that as concurrency rises, throughput becomes bounded by object storage request latency and rate limits, and conflict traffic rather than raw write capability becomes the binding constraint. Treat the specific figures as one vendor's benchmark of open-source formats on one storage service; the structural point — that retries amplify load on the same resource everyone is contending for — is what you should test for.
Object count and metadata grow faster than data
Small, frequent writes produce small files. Databricks publishes a table relating table size to target file size and the resulting file count: a 10 GB table at a 256 MB target is roughly 40 files, a 1 TB table roughly 4,096, and a 100 TB table at a 1 GB target roughly 114,437 (Databricks docs). Read that as a shape, not a target. If agent checkpoints arrive at a cadence that outpaces compaction, the file count climbs faster than the byte count, and every reader pays for it in listing and planning work.
Metadata scales with file and version count too. The same benchmark reports an observed slope of roughly 4.5 KiB of active metadata per active data file for Iceberg and roughly 82 bytes per fragment for Lance, which at a million files means about 4.3 GiB versus about 82 MB of active metadata. The numbers are the benchmark author's measurements of specific formats; the transferable lesson is that metadata cost is a function of object count, so an agent workload's small writes stress metadata even when storage capacity looks comfortable.
What a Storage Rewrite Concretely Changes — and What It Does Not Fix
A storage rewrite is a real architectural change, and it is worth being precise about what it buys. GitHub's account is unusually explicit about the division of labor.
Levers a rewrite can move
Four categories of change show up in GitHub's design, and each has a lakehouse analogue you can ask a vendor about.
Decoupling storage from compute. GitHub moved authoritative repository data into Azure Blob Storage so that losing a compute worker is closer to a cache miss than a durability event; a replacement worker starts serving and fills its cache from durable storage. In a lakehouse, the equivalent question is whether a failed writer or a restarted cluster is a recovery event or a retry.
Ordering at a finer granularity than a global lock. GitHub describes holding a repository-wide lock for the entire reference update as turning the quorum into a single unordered queue with one write in flight, where each write waits longer than the one before. Its replacement orders each reference independently. For a table format, the analogue is how narrowly the commit path scopes conflicts — whether two writers touching different partitions or files contend at all.
Coalescing concurrent writes. GitHub's virtual write replica coalesces transactions on disjoint references into a single set of network I/Os, so a busy repository pays one round trip across many concurrent writes instead of one per write. On a lakehouse, look for whether the platform batches or merges concurrent small commits before they reach object storage.
Moving maintenance onto separate capacity. GitHub scales compaction by adding compaction nodes, with a lease so only one node compacts a shard at a time. Databricks offers a comparable separation: predictive optimization runs maintenance asynchronously on serverless compute, while auto compaction runs on the cluster performing the write (Databricks docs).
What stays your problem
None of these levers removes maintenance. Databricks states directly that auto compaction and optimized writes each reduce small-file problems but are not a full replacement for OPTIMIZE, and recommends scheduling OPTIMIZE for tables larger than 1 TB. OPTIMIZE compacts files but does not remove the old ones; VACUUM does that, and its default retention threshold is 7 days, after which you lose the ability to time travel to older versions (VACUUM reference). Microsoft's guidance for Fabric tables adds that VACUUM reduces retained storage but does not improve the active file layout (Microsoft Learn).
Clustering-key design is also yours. Databricks recommends liquid clustering for new tables and warns that choosing clustering columns very different from the original partition columns triggers a large reclustering operation on the first OPTIMIZE run (Databricks docs). And conflict-aware write patterns — narrowing a MERGE's match condition so it touches fewer partitions — remain a design decision the platform cannot make for you.
The migration sequencing lesson
GitHub's bridge post states the principle plainly: changing the storage architecture and the hosting platform at the same time, under that growth, carried too much risk, so the team moved the data first and evolved the architecture once it arrived. If you are migrating off or alongside Databricks while also introducing agent workloads, sequence the moves. Changing storage format, compute platform, and write patterns in one step leaves you unable to attribute a regression to any of them.
Needs Assessment: Define Your Agent Workload Before You Shop
Before comparing vendors, measure what you actually have. The point of this section is to produce numbers you can put in front of a vendor and later reconcile against a pilot.
Quantify concurrency and write rate
Record the number of concurrent writers at peak, the commits or checkpoints each agent produces per hour, and the peak write rate. GitHub's growth multiples are a useful reference for how quickly these can move: total Git events roughly doubled in a year while pushes grew 4.9x. If your agent fleet is planned to grow, model the write rate at the planned size, not today's.
Characterize your table and file profile
Inventory table sizes, files per partition, and metadata growth over time. Use the Databricks file-count table as a benchmark shape — a healthy large table sits in the tens of thousands of files, not the millions — and note where your agent-written tables sit relative to it. Also record your retention and time-travel requirements, because they set the floor on VACUUM retention and therefore on how much history you must keep readable.
Separate agent traffic from analyst traffic
Agent loops are latency-sensitive and write-heavy; BI queries are throughput-oriented and read-heavy. They impose different demands on the same storage layer, and a platform that serves one well may not serve the other. Keep separate measurements for each so you can tell a vendor which workload a given test is meant to represent.
Evaluation Criteria Explained
Four dimensions cover most of what distinguishes a Databricks alternative for this workload. Each should be tied to something you can observe in a test rather than accept from a datasheet.
Concurrency and conflict handling
Ask how the platform orders concurrent commits, what happens on conflict, whether retries are bounded or backed off, and how write coalescing works. Delta Lake's documented behavior — fail with a concurrent modification exception rather than corrupt the table — is a reasonable baseline to compare against. What you want to know is the conflict scope: do two writers touching different files contend, and what does the client do when it loses?
Maintenance automation and its boundaries
Verify what is automated (compaction, vacuum, clustering) and, more importantly, where the vendor documents that automation stops. Databricks is explicit that auto compaction is not a replacement for scheduled OPTIMIZE above 1 TB and that VACUUM retention governs time travel. A vendor that describes automation without stating its limits has given you less than one that does.
Storage and compute separation
Ask what happens when a compute node fails mid-write, whether local state is cache or source of truth, and how recovery scales. GitHub's framing — a lost worker as a cache miss rather than a durability event — is the property to test for.
Pricing transparency
The billing dimensions you need before committing are the unit of measure, the metering granularity, what is included versus excluded, region variation, and how commitments and discounts draw down. Published models differ in how far they go. IBM watsonx.data publishes a resource unit at a list price of USD 1 per RU, metered per second with a one-minute minimum, and states that pricing includes compute costs only and excludes core support services at 3.00 RUs/hr (IBM). Snowflake bills storage on average monthly usage after compression and separates on-demand from pre-paid capacity (Snowflake). Databricks publishes a price list of SKUs with list prices and per-second pay-as-you-go billing, with committed-use contracts for discounts (Databricks), and Azure publishes a DBU-hour table by workload type plus a DSU for storage (Azure).
What these have in common is that the unit and the granularity are stated. What they share as a limitation is that the actual rate often is not: Azure's table shows $-/DBU-hour placeholders, and Databricks directs buyers to a quote for committed pricing. Transparency, for evaluation purposes, means you can build a model from published dimensions — not that every number is on the page.
Red Flags and Deal-Breakers
Keep these two categories separate. A red flag is a reason to investigate further; a deal-breaker is a condition that disqualifies a vendor for this workload regardless of how the investigation goes.
Red flags to investigate
A pricing page that omits the unit of measure or the metering granularity.
Performance claims stated without the workload conditions — data size, concurrency, file layout — under which they were measured.
Maintenance automation described without documented limits or failure modes.
Concurrency handling described only in marketing terms, with no statement of conflict behavior or retry policy.
Deal-breakers for agent-scale workloads
A repository- or table-wide write lock with no finer ordering, which caps write throughput at one write per round trip.
No documented conflict or retry behavior, leaving you unable to predict what happens under contention.
Retention or time-travel limits incompatible with your recovery requirements.
Pricing that cannot be modeled from published dimensions before signature.
How to Test Pricing Transparency Before You Commit
This is a procedure for testing whether a vendor's pricing can be modeled, not a judgment about any vendor's honesty.
Build a unit-economics model from published dimensions
Start from the vendor's published unit, metering granularity, and inclusions and exclusions. Build a cost estimate for your measured workload — write rate, table sizes, retention — and list every dimension that is quoted only on request. That list is your agenda for the pilot.
Run a bounded pilot with metered billing
Design a pilot that exercises agent-scale write concurrency: many concurrent writers, small commits, your real retention settings. Run it long enough to cross at least one compaction and vacuum cycle. Then compare the metered bill against your model and isolate which dimensions diverged. A divergence in one dimension is a modeling gap; divergence across several suggests the published dimensions do not describe your workload.
Ask the questions the pricing page does not answer
How does price vary by region, and is the published rate for your region?
What are the egress charges for cross-region or cross-cloud access?
Is storage billed on raw or compressed size, and how is compression measured?
How does a commitment draw down across SKUs or workloads, and what happens to unused commitment?
Which support, networking, and managed-service costs sit outside the compute price?
Pre-Commitment Acceptance Checklist
Use this as the gate before signing. Each item should be answerable with evidence from your environment or the vendor's documentation, not from a sales conversation alone.
Workload characterized: concurrent writers, commits per agent per hour, peak write rate, and planned growth.
Table and file profile inventoried, including files per partition and metadata growth.
Agent and analyst traffic measured separately.
Concurrency behavior verified under test: conflict scope, retry policy, and write coalescing observed.
Maintenance boundaries confirmed in writing: what is automated, what is not, and at what table size.
Storage and compute separation tested: a failed writer is a retry, not a recovery event.
Pricing model built from published dimensions and reconciled against a metered pilot bill.
Every dimension quoted only on request documented and priced.
Migration sequencing agreed: storage, platform, and write-pattern changes staged rather than simultaneous.
Retention and time-travel requirements mapped to the platform's documented limits.
Conclusion
Agent-scale concurrency is a storage-architecture problem before it is a compute problem. The GitHub case shows that when thousands of agents write continuously, the binding constraint becomes the ordered state they all converge on — the reference update, or in a lakehouse, the commit log — and the retries that contention generates. A Databricks alternative earns consideration by demonstrating, under your workload and your billing, that it orders writes at a fine enough granularity, documents where its maintenance automation stops, and prices from dimensions you can model before you sign. The checklist above is how you turn that demonstration into a decision.
FAQ
Why does agent concurrency differ from BI concurrency? BI workloads are read-heavy and write in periodic batches. Agent workloads write continuously, checkpoint after nearly every action, and converge many writers on one ordered state, so the number of ordering events per hour rises even when data volume does not. See Why Agent-Scale Concurrency Stresses Storage Differently.
What does a storage rewrite fix, and what does it not? It can decouple storage from compute, order writes at a finer granularity, coalesce concurrent writes, and move compaction onto separate capacity. It does not remove the need for compaction, retention policy, clustering-key design, or conflict-aware write patterns. See What a Storage Rewrite Concretely Changes.
How do I test pricing transparency? Build a cost model from the vendor's published unit, metering granularity, and inclusions, list every dimension quoted only on request, then run a bounded pilot with metered billing and reconcile the bill against the model. See How to Test Pricing Transparency Before You Commit.
What belongs on the acceptance checklist? Workload characterization, concurrency behavior verified under test, maintenance boundaries confirmed in writing, storage and compute separation tested, pricing reconciled against a pilot bill, and migration sequencing agreed. See Pre-Commitment Acceptance Checklist.
Previous posts
1. https://singdata.com/trending/natural-language-analytics-tools-permission-controls/
2. https://singdata.com/trending/vector-index-row-level-security-layout