How NeoCRM Built Real-Time Analytics on Singdata Lakehouse

Author: Luo Yi, Vice President of Technology at NeoCRM

📌 Introduction:

A Lakehouse combines the flexibility and scalability of a data lake with the query performance and governance capabilities of a data warehouse. NeoCRM, a SaaS CRM provider, uses this architecture to power real-time analytics across its product portfolio and deliver a faster, more responsive data experience to customers. In this case study, NeoCRM shares how it put the Lakehouse model into practice to support data-intensive applications at scale. Topics include:

• NeoCRM’s data landscape
• Data intelligence applications
• Challenges and bottlenecks
• Putting the Lakehouse into practice
• Q&A

---

NeoCRM’s Data Landscape

The Challenge: Supporting a Multi-Tenant Data Platform

NeoCRM provides a metadata-driven, multi-tenant SaaS platform built around its CRM product suite. Supporting that platform at scale presents several distinct challenges:

• A large and growing tenant base

  Over 13 years in business, NeoCRM has grown to serve more than 200,000 tenants. Its customer base ranges from several thousand core enterprise accounts—including Fortune Global 500 companies—to a long tail of free users. Both the total number of tenants and the share of large enterprise customers continue to rise, driving sustained growth in data volumes. At the same time, CRM has evolved beyond internal sales management to support direct engagement between businesses and their customers, creating more customer-facing workloads with high concurrency.

• Diverse business requirements

  Sales processes vary widely by industry, and modern CRM now extends well beyond sales into marketing, service, and end-to-end customer operations. Each industry and use case brings different patterns for CRUD operations—create, read, update, and delete—along with its own search and analytics requirements.

• Dynamic schemas

  Applications do not operate directly on the database or expose DDL (Data Definition Language). Instead, schemas are defined through metadata. This enables NeoCRM to roll out seamless, transparent upgrades without interrupting customers’ operations.

• Consistent performance and reliability

  Performance and reliability are non-negotiable for a SaaS platform. Most mid-market and long-tail customers share database instances, while selected large enterprise customers run on dedicated instances. The platform must deliver consistent performance across both models.

• Complex access controls

  Customer data is one of an enterprise’s most valuable assets, making access control both critical and highly complex. Large corporate groups may require intricate permissions at the business and data layers. In China, the growing use of social channels for customer engagement adds another layer of complexity, requiring fine-grained controls across new interaction models.

NeoCRM multi-tenant data platform

NeoCRM addressed these challenges by building a multi-tenant data platform around shared resources. By default, the platform is designed to allocate resources fairly across tenants at both the application and data layers. As the number of large enterprise customers grew, NeoCRM also introduced safeguards for their business-critical workloads. Large customers can run on dedicated database instances while still benefiting from a shared-storage architecture.

The platform is built around five core capabilities:

• Entity metadata

  NeoCRM represents business-facing database structures as business objects or entities and manages them through a unified metadata layer. Standard CRM entities include accounts, contacts, opportunities, and orders, while tenants can add custom entities and fields to match their own processes. An automotive company, for example, might extend the standard customer entity to capture vehicle-owner information.

• Multiple storage models

  The platform combines several storage technologies. Core data resides in SQL relational databases, caching reduces pressure on those databases, and Elasticsearch indexes support search. For the past six or seven years, Greenplum served as the analytical data warehouse.

• Multi-tenant entity tables

  Each customer works with what appears to be its own database. Behind the scenes, multiple tenants share physical database instances and tables, with a mapping layer connecting each tenant’s virtual database to the underlying storage.

• Database and table sharding

  Database and table sharding, together with data partitioning, allows the platform to handle large tenant counts, massive datasets, and high-concurrency workloads while preventing any individual tenant’s workload from destabilizing the system.

• Query planning and optimization

  Indexing is particularly challenging in a metadata-driven storage model. Wide tables may contain too many columns to index indiscriminately, and every tenant has different access patterns. NeoCRM therefore migrated its operational database from MySQL to PostgreSQL. PostgreSQL removes previous column-count constraints and supports dynamic partitioning and partition-level indexes. NeoCRM can now create tenant-specific partitions and targeted indexes based on actual business needs, all managed through its query planning and optimization layer.

Together, these capabilities give NeoCRM the scale to handle demanding workloads without sacrificing the flexibility individual tenants need.

Data Architecture

The diagram below shows NeoCRM’s data architecture.

NeoCRM data architecture

NeoCRM offers products for marketing, sales, service, and other business scenarios. Rather than interacting directly with the underlying wide tables, these applications access data through a unified data service. That service manages and resolves entity metadata, reads and writes table data, validates business objects and entities, performs calculations, and supports logging and open integration. It also exposes SQL-like query APIs and search APIs to business applications, alongside a broader set of data management capabilities.

At the storage layer, most operational data now resides in PostgreSQL, although some project metadata remains in MySQL. NeoCRM uses Redis for caching and Elasticsearch for search. Its multi-cloud architecture spans Tencent Cloud, AWS, and Huawei Cloud, with data stored in S3 or OSS depending on the environment. Most private-cloud customers use Alibaba Cloud OSS. Supporting middleware includes MyCat for database and table sharding, as well as messaging middleware.

Virtual Database: Entity Interface + Shared Storage + Tenant Isolation

NeoCRM virtual database architecture

This architecture enables metadata-driven data access in a multi-tenant environment.

From a tenant’s perspective, business entities such as leads, opportunities, accounts, orders, contracts, and payments live in a virtual database and are accessed through the unified data service. Underneath, the data is stored in a relational database. The physical model includes auxiliary index tables, metadata tables that describe each tenant’s entities and data structures, and wide entity master tables that hold the actual records.

Virtual Database: Entity Metadata

NeoCRM entity metadata model

The metadata layer supports a wide range of configurations. NeoCRM can define field formats and business types—for example, individual and corporate customer types—with different fields and page layouts for each. It also supports relationships such as master-detail, one-to-many, many-to-one, and polymorphic associations. Fields can be checked for duplicates, and custom calculated fields can populate values automatically using formulas.

Virtual Database: Multi-Tenant Entity Table Structure

NeoCRM multi-tenant entity table structure

The multi-tenant table structure consists of metadata tables, index tables, and wide entity master tables. The index tables manage tenant-specific indexes. The metadata tables record exactly where each business field is stored in the underlying master tables. When an application works with a business entity, the platform automatically maps the request to the correct physical table and column.

This separation makes upgrades invisible to tenants. Metadata definitions are decoupled from changes to the underlying entity data, so NeoCRM can evolve the platform without rewriting business-object logic or relationships for every upgrade. This architectural pattern is now common among leading SaaS platforms.

Data Intelligence Applications

NeoCRM’s most widely used data intelligence application is its in-house, integrated BI platform. It is designed around three principles:

• Real-time data: Operational data flows into the warehouse continuously, making it available for real-time analysis.
• Self-service analytics: Business users can create reports and analyses around the metrics that matter to them without relying on technical teams.
• Built-in governance: BI permissions are fully integrated with the business platform, so users see only the data they are authorized to access. Exceptions can be configured when needed, but the secure model is the default.

Overview of NeoCRM’s Data Intelligence Portfolio

NeoCRM data intelligence portfolio

NeoCRM synchronizes its operational platform with the data warehouse in real time and makes governed BI available across the organization. Rather than asking users to work directly with wide physical tables, the platform lets them define data sources and datasets through business metadata. This makes analytics easier to understand and far more accessible to non-technical users.

Embedded BI components can also be placed directly within CRM pages and workflows while retaining the same centralized permission model. Over the past two years, NeoCRM has expanded the portfolio with AI capabilities, including machine-learning models that score leads, opportunities, and accounts, as well as next-best-action recommendations for sales teams. More recently, it has begun turning large language model capabilities into commercial products; a customer service chatbot is one early example.

BI Architecture

NeoCRM BI architecture

The BI architecture includes a data synchronization layer. NeoCRM initially relied on an ETL product from a Chinese vendor called Data Pipeline, but later replaced it with an in-house tool tailored to its environment. The custom solution makes synchronization easier to manage while improving data freshness. It can typically move data from the operational database into the warehouse within 15 minutes and, in many cases, with millisecond-level latency. The analytical warehouse and BI capabilities were built on Greenplum.

A key component is the query parser. Because datasets, BI views, and analyses are defined through entity metadata, every warehouse query must be translated into the appropriate query against the underlying wide tables. The parser also injects the relevant permission predicates into each query, ensuring that users can view and analyze only the data they are entitled to access.

Challenges and Bottlenecks

NeoCRM data platform challenges and bottlenecks

Over six years of operating its BI platform, NeoCRM began to encounter limitations in the original architecture. Rather than building a separate analytical model, it had replicated the operational database schema directly into the warehouse. That approach left the warehouse exposed to frequent schema changes. The environment had grown to roughly 8,000 tables, with highly complex access controls, while the business increasingly expected sub-second interactive queries.

Greenplum had been the right choice when NeoCRM selected it six years earlier. As the company began serving more large enterprises—including Fortune Global 500 customers—expectations for BI performance and analytical capabilities rose sharply. Scaling Greenplum to meet those expectations became increasingly expensive.

The data itself was also changing. New customer engagement scenarios introduced more semi-structured and document-based data alongside traditional structured records. At the same time, NeoCRM was bringing AI and large language model capabilities to market, creating a new requirement: data needed to be easier for AI applications to access and use. Together, these pressures made a data platform upgrade necessary.

Putting the Lakehouse into Practice

Requirements for a Next-Generation Data Platform

Requirements for NeoCRM’s next-generation data platform

NeoCRM identified four priorities for its next-generation platform:

• Open, low-friction data integration

  The goal was to integrate CRM operational data quickly and cost-effectively. Because NeoCRM offers an integrated product suite, ETL—extract, transform, and load—is central to the architecture. The new platform needed straightforward tools for ingesting traditional CRM records, behavioral data, and unstructured content.

• Real-time processing

  NeoCRM’s in-house synchronization tool already delivered millisecond-level latency in many scenarios. The new platform needed to preserve that freshness and support real-time processing and analytics.

• Unified batch, streaming, and interactive workloads

  Large customers often require both external bulk data synchronization and interactive analysis. Bringing batch processing, stream processing, and interactive queries onto one platform would reduce end-to-end complexity.

• AI readiness

  The platform also needed to support NeoCRM’s long-term AI roadmap and make enterprise data easier for AI applications to consume.

After an extensive evaluation, NeoCRM selected the cloud-native Singdata Lakehouse as the foundation for its next generation of data and AI applications. Singdata Lakehouse offered standards-based SQL compatibility while addressing the elasticity and scaling limitations NeoCRM had encountered with Greenplum, making it a strong fit for enterprise workloads.

From Greenplum to Lakehouse: Validation

NeoCRM Lakehouse validation results

NeoCRM began with a proof of concept using anonymized, production-like data that mirrored the scale and diversity of its live environment as closely as possible. Interactive query performance was strong, and the platform scaled elastically as workloads changed. Each service endpoint maps to its own cluster, with clusters deployed across Tencent Cloud, AWS, and Huawei Cloud for different customers. In BI workloads, a single cluster sustained 150 QPS, while the longest response time for an individual SQL query remained under four seconds. Large datasets still benefit from deliberate data modeling, but the platform met these targets in the vast majority of scenarios.

The team then turned to data integration. It simulated all 8,000 tables in the production environment and tested how quickly schema changes and the resulting data updates could be synchronized. Under normal operating conditions, maximum latency stayed below five minutes, while most data arrived in under one second. Continuous checks using Singdata Lakehouse’s online data comparison tool confirmed 100% data consistency.

From Greenplum to Lakehouse: Cutover

NeoCRM cutover from Greenplum to Lakehouse

Once validation was complete, NeoCRM began the production cutover. One important architectural decision was to keep customer data in NeoCRM-controlled storage. On AWS, for example, the company uses its own S3 account. This avoids the security and compliance risk of storing customer data in an undisclosed or unauthorized location—an especially important consideration for SaaS providers serving large enterprises.

Before moving applications, the team validated data integration, corrected syntax incompatibilities, and ran stress tests. The process began with dual writes, followed by a tenant-by-tenant cutover and an extended observation period to ensure a seamless transition.

With support from Singdata, NeoCRM has completed most of the cutover for its Tencent Cloud clusters in Beijing and Guangzhou, while migration of its AWS clusters in Beijing is under way. Nearly all customers in China can already use the analytics platform powered by Singdata Lakehouse.

Value Delivered by the Lakehouse

Value delivered by Singdata Lakehouse

Singdata Lakehouse has delivered value in three key areas:

• Fresher data: NeoCRM rebuilt its in-house synchronization tool around Singdata’s native real-time ingestion capabilities. The standard synchronization window fell from 15 minutes to five minutes, while most data now arrives in under one second.
• Faster real-time compute: Real-time compute performance improved by more than 30%. NeoCRM can now ease previous restrictions on BI metrics, data exports, and other resource-intensive features. Elastic compute also gives the team greater confidence in supporting complex workloads for mid-sized and large enterprises.
• A stronger foundation for AI: NeoCRM can bring structured CRM records, semi-structured behavioral data, and document-based content under a single governance framework. This creates a broader, better-managed data foundation for new analytical and AI applications.

Q&A

Q1: Is tenant management implemented by adding a tenant ID to each wide table?

Yes. Every wide table includes a tenant ID because each tenant has its own source data. The raw-data layer contains standard CRM entities such as opportunities and accounts, as well as tenant-specific entities—for example, vehicle owners for an automotive customer. Tenant IDs are therefore also included in the entity metadata tables.

Q2: How do dynamic schemas work?

The mechanism is relatively straightforward. A metadata table defines each business entity and maps it to the underlying wide table where the data is physically stored. Applications operate on metadata-defined entities rather than physical tables. At the database layer, the platform translates those operations into queries against the appropriate tables and columns.

Q3: Does data synchronization run as a streaming job?

Yes. Synchronization is based on real-time streams, with database changes captured through CDC. Singdata Lakehouse provides native streaming capabilities; under the hood, it may use a framework similar to Flink to process CDC events.

Q4: Is there a better way to support dimensional drill-down and roll-up across hundreds of millions of records?

If the current view performs well, drilling down into the next dimension will usually perform well too. For more complex analytical workloads, however, a cube-based approach may be a better fit.

Q5: How compatible are Greenplum and Lakehouse, and how difficult is the migration?

There are two aspects to compatibility. The first is data migration. Because the operational database remains the system of record, NeoCRM can load data into Lakehouse through a combination of full and incremental synchronization. That makes the data migration itself relatively straightforward.

The second is SQL compatibility for queries and analytics. In NeoCRM’s work with Singdata, the vast majority of workloads migrated without changes. Only a small number of cases—such as certain date and time functions—required syntax adjustments.