3× Query Performance Boost | How Mediatrack Upgraded to a Unified Data Platform with Singdata Lakehouse
📌 Introduction:
Mediatrack (分秒帧) is a cloud-based production collaboration platform for audio and video creators, offering an all-in-one cloud collaboration service for the next generation of cloud-era audio/video production. It helps creators build workflows in the cloud — including online review, sharing & delivery, workflow management, asset management, and remote monitoring. It now serves over 1 million creators and 3,000+ enterprises across verticals including advertising, media, MCN, education, gaming, animation, e-commerce, and film & television.
Mediatrack replaced its original three-pipeline data platform with a unified "Single-Engine" data platform built on Singdata Lakehouse. Through lake-house integration, Mediatrack achieved compute-storage separation, eliminated data redundancy, and reduced storage and compute costs by over 60%. By leveraging incremental computation to enable real-time processing across all domains, the platform achieved end-to-end second-level data freshness, simplified the data processing pipeline, and reduced operating costs by over 50%.
Author: Ji Huabin, Technical Manager, Mediatrack Product R&D Department
Mediatrack's Data Platform Evolution and Pain Points
Mediatrack's tech team needed to support BI reporting for multiple internal departments — including business operations, marketing, sales, and customer success. The team started with Spark for offline reporting, then progressively added Spark Streaming, Kafka, and ClickHouse to support OLAP analysis and real-time BI dashboards, eventually forming a Lambda architecture-based data platform.
Architecture before the upgrade:
As the platform continuously ingested data from multiple cross-platform sources and integrated a growing number of components, the architecture became increasingly complex, with high operational overhead and mounting query performance bottlenecks. The core problems fell into three areas:
1. Diverse data types made unification difficult. Website traffic data, CRM customer relationship data, Sensors Data behavioral logs, ad platform performance data, and more were scattered across data lakes and warehouses — containing large volumes of semi-structured and unstructured text, audio, and video data. Business definitions were ambiguous, metrics were duplicated, and integration costs were high. Without unified user profiling, closing the loop between upstream and downstream business processes was impossible.
2. Coexistence of offline and real-time analytics caused data redundancy and governance challenges. The platform had to support traditional offline analysis, real-time analytics, and complex scenarios like user behavior analysis and membership retention/growth. To support real-time QoE (Quality of Experience) video playback analysis, QoE data was redundantly stored across multiple pipelines including EMR and ClickHouse — creating over 50% data redundancy and making governance extremely difficult.
3. Complex ingestion and ETL architecture drove up development and ops costs. Multiple data pipelines meant multiple copies of data, with high downstream development and maintenance costs. The original platform used ClickHouse and EMR in a classic Lambda architecture — combining batch ETL jobs, Spark Streaming for real-time scenarios, and several other pipelines. The Lambda architecture essentially kept "stitching" components together, and the resulting storage redundancy was the primary pain point.
Cost breakdown across the full data pipeline:
After auditing cost drivers across the full pipeline — from ingestion to operations — the team identified cost issues not just in compute, but across architecture, data flow, and governance. They concluded: while open-source solutions can optimize individual points, systemic cost reduction requires addressing the entire pipeline holistically.
To resolve these issues, Mediatrack set the following upgrade goals:
- Architecture upgrade: Merge real-time and offline pipelines into a fully real-time, unified pipeline; reduce data redundancy from complex multi-chain architectures.
- Storage upgrade: Move from data warehouse to lake-house integration, enabling support for semi-structured/unstructured data and future AI capabilities.
- Overall cost reduction: Target a 30% reduction in ops costs, 20% reduction in storage/compute costs.
- From self-managed to fully managed: Company strategy shifted away from self-building a data platform, seeking a managed product to minimize maintenance overhead.
Why Singdata
In 2022, Mediatrack began evaluating data platform upgrades and ultimately chose Singdata. The "Single-Engine" unified architecture philosophy aligned with the team's goal of simplifying complexity and reducing development and ops costs. Singdata Lakehouse's open lake-house storage enabled unified metadata management, supported unstructured data, and accommodated future AI/ML use cases. Its fully managed, zero-ops model eliminated routine system maintenance.
The most compelling factor was the potential for significant reductions in resource and operational costs.
Performance benchmark comparison:
According to public benchmark data, Singdata Lakehouse outperforms Spark by 9× in large-scale batch processing (TPC-DS 10TB), by 7× in ad-hoc analytics (TPC-H 100GB), and by 40× in real-time analytics (SSB-FLAT 100GB) — all without any system switching between offline and real-time workloads, and 20% faster than ClickHouse on real-time analytics.
Upgraded Architecture
The new architecture resolved the limitations of both ClickHouse and EMR:
- ClickHouse excelled at single large-table queries but degraded sharply on multi-table joins.
- EMR required 30+ minutes for complex ETL jobs; Hive SQL had performance bottlenecks and couldn't satisfy latency-sensitive scenarios.
Singdata Lakehouse handles multi-table interactive analytics (replacing ClickHouse) and batch processing workloads (replacing EMR) — within a single engine, with no separate pipelines required. This resolved data redundancy and dramatically simplified governance.
Architecture after the upgrade:
Results After Upgrading to Singdata Lakehouse
1. Unified Data Assets — Unlocking Business and Operational Value
The lake-house integration platform makes it easier to consolidate data from all domains and maximize business value. For example, using Singdata's OneID capability, Mediatrack integrated semi-structured behavioral logs from Sensors Data with website, product, and CRM data — building a 360° user profile and a scoring model to identify potential users and new traffic opportunities, enabling targeted operational strategies for different audience segments.
2. Low-Cost Real-Time and Offline Unification
Using Singdata Lakehouse's incremental computation technology, Mediatrack replaced high-cost standalone EMR and Spark Streaming with a single unified pipeline. One engine, one dataset, one codebase — with flexible scheduling frequency to meet different data freshness requirements at significantly lower cost.
3. Simplified Architecture — Reduced Platform Complexity
The unified lake-house platform replaced the original multi-component stack:
- Kafka + Spark Streaming + ClickHouse
- MySQL + binlog + Canal + ClickHouse
One Singdata Lakehouse replaced 5 components, dramatically reducing platform complexity and operational overhead.
4. Storage and Compute Costs Down 60%+; Query Performance Up 3×
Singdata Lakehouse's compute-storage separation architecture, combined with efficient storage formats and compression algorithms, delivered:
- Storage costs: reduced by 60%+
- Compute costs: reduced by 60%+ through elastic scaling and pay-as-you-go billing
- Development & ops costs: zero ops investment as a fully managed SaaS platform with unified SQL-based development
The platform dynamically scales compute resources based on actual workload demand:
Business Value Delivered
1. Unified data assets. All data from diverse sources and formats was consolidated into a central repository — enabling unified user profiling, automated lead management, and a complete closed-loop from marketing to product to customer success. Data insights became more precise, enabling targeted operations for different audience segments.
2. Tiered data freshness. Mediatrack now sets differentiated refresh schedules for different use cases. Business dashboards can upgrade from T+1 to H+1 or M+5, and upstream applications can perform real-time or near-real-time tag computation and metric aggregation.
3. More flexible business support. Developers write a single unified SQL codebase for data development, and the system automatically refreshes pipelines via MV-on-MV incremental computation — making development simpler and cross-team collaboration more efficient.
Summary and Outlook
By adopting Singdata Lakehouse, Mediatrack significantly simplified its technical architecture — transitioning from a Lambda architecture to a Kappa architecture — achieving the goals of cost reduction and efficiency improvement across integration, development, and operations. With operations delegated to Singdata, the data team was freed from maintenance work and could focus on data-driven business innovation and decision support.
Mediatrack also experienced firsthand the benefits of a fully managed product: continuous, visible product iteration. Feature requests submitted to the Singdata product team received responses, with the following enhancements already on the roadmap:
- Expanded incremental computation scenarios: MV real-time task operations support; backfill and schema change support for real-time tasks.
- Custom function support: Extended support for UDAF and UDTF in Java and Python.
This is Mediatrack's experience with Singdata Lakehouse, shared as a reference for data architecture teams evaluating similar upgrades.
