Managed Iceberg Tables

A Managed Iceberg Table is a native Apache Iceberg table whose underlying data files and metadata are fully managed by Singdata Lakehouse. Built on the open Apache Iceberg table format, it combines Lakehouse SQL read/write capabilities, unified governance, automated storage maintenance, and incremental computation with open multi-engine read/write interoperability via the built-in Iceberg REST Catalog (for engines such as Apache Spark and DuckDB).

Comparing Table Types

DimensionManaged Iceberg TableRegular Table (Lakehouse Native)Iceberg External Table
Table formatOpen Apache IcebergLakehouse native columnar format (Iceberg-compatible read view available under certain conditions)External Apache Iceberg
Storage & metadata managementFully managed by Singdata LakehouseFully managed by Singdata LakehouseManaged by external storage and catalog
Write capabilitiesFull SQL DML (INSERT, UPDATE, DELETE, MERGE) + external REST engine writesFull SQL DML + IGS (Ingestion Service) real-time/bulk ingestion (no external engine writes)Primarily federated read queries
Storage maintenanceBackground automatic compaction, manual OPTIMIZE, snapshot expiration, and orphan file cleanupAutomatic & manual compaction, recycling, and lifecycle managementHandled by external systems
Multi-engine interoperabilityBuilt-in Iceberg REST Catalog supporting external reads and writes (Spark, DuckDB, etc.)Supports read-only Iceberg-compatible external access via Iceberg REST Catalog under certain conditionsShared via external catalog
Lakehouse-specific accelerationContinuously aligning with Regular Tables; hash bucketing, secondary indexes, and materialized views are not yet coveredHash bucketing, Bloomfilter / inverted / vector indexes, materialized viewsLakehouse-native indexes not applicable
Best forOpen lakehouse architectures, cross-engine read/write sharing, avoiding format lock-inMaximum internal acceleration, CDC primary-key deduplication, multimodal index search (with optional external reads)Federated queries over existing external Iceberg lakes

How It Works

Managed Iceberg Tables store Parquet data files alongside standard Iceberg metadata files (metadata.json, manifest lists, and manifest files) in Lakehouse-managed storage, with all commits coordinated by the Lakehouse catalog:

  • Unified commit coordination: Whether data is written via Lakehouse SQL or an external engine through the Iceberg REST Catalog, commits atomically publish the latest metadata pointer through the Lakehouse catalog to maintain snapshot consistency.
  • Format version storage: The Iceberg format version is recorded in the top-level format-version field of metadata.json rather than stored as a regular table property. Singdata Lakehouse currently supports up to Iceberg v2 table format specification, with newer versions planned on the roadmap.
  • Standard Iceberg semantics: Hidden partition transforms, column field id resolution, and write.*.mode row-level DML strategies follow the Apache Iceberg specification for consistent cross-engine behavior.

Key Capabilities

  • ACID transactions, row-level DML & Time Travel: Supports INSERT INTO, INSERT OVERWRITE, UPDATE, DELETE, MERGE INTO, and TRUNCATE TABLE, as well as historical snapshot queries via VERSION AS OF and TIMESTAMP AS OF. Row-level modifications default to Merge-on-Read (MoR, Position Deletes) and can be configured per operation to Copy-on-Write (CoW) using standard Iceberg table properties.
  • Hidden partitioning & schema evolution: Supports identity partitions and standard transforms (years, months, days, hours, bucket, truncate). Supports adding, dropping, renaming, reordering columns, widening allowed types (such as INT -> BIGINT, FLOAT -> DOUBLE, and DECIMAL precision widening), nested field evolution, and updating comments without rewriting historical data files.
  • Incremental computation integration: Works as both an upstream source (including capturing external engine writes) and a target for Dynamic Tables with incremental refresh, and supports change data capture via Table Streams directly from Iceberg snapshots.
  • Automated storage maintenance: Includes background automatic compaction and manual OPTIMIZE compaction, plus scheduled snapshot expiration and orphan file cleanup to minimize small-file bloat and storage overhead.
  • Open engine interoperability: Exposes an Apache Iceberg REST Catalog endpoint with Vended Credentials so external engines such as Apache Spark and DuckDB can securely read and write managed tables.

Supported Data Types

CategorySupported Data Types
Boolean & NumericBOOLEAN, INT, BIGINT, FLOAT, DOUBLE, DECIMAL(p, s)
String & BinarySTRING, BINARY
Date & TimeDATE, TIMESTAMP (TIMESTAMP_NTZ, TIMESTAMP_LTZ)
Complex / NestedARRAY, MAP, STRUCT

Current Limitations

Singdata Lakehouse is continuously aligning Managed Iceberg Tables with Regular Tables in both feature coverage and performance while preserving open Iceberg compatibility. At the current stage, keep the following scope notes in mind:

  • Format version: Currently supports up to Iceberg v2 table format specification. Unsetting or downgrading format-version via ALTER TABLE is not supported.
  • Lakehouse-native table features: CLUSTERED BY hash bucketing, secondary indexes (CREATE INDEX), Materialized Views (use CREATE DYNAMIC TABLE ... USING ICEBERG for incremental computation), CLONE, RESTORE TABLE, and column masking policies are not yet covered at this stage. Change capture works directly through Iceberg snapshots, so the change_tracking table property is not needed.
  • Evolution & metadata tables: Partition evolution, sort order evolution, and Spark-style Iceberg Metadata Tables (such as <table>.snapshots or <table>.files) are on the roadmap; use DESC HISTORY, SHOW FILES FOR, and SHOW PARTITIONS for equivalent operational inspection.
  • Ingestion channels: SQL DML, COPY INTO TABLE, and Pipes are supported (check_commit_conflict=false is not yet available for COPY INTO). IGS (Ingestion Service) real-time/bulk ingestion currently targets Regular Tables and is not yet available for Managed Iceberg Tables.
  • REST Catalog scope: External engines support catalog discovery, metadata loading, data reads, and row-level DML writes via the REST Catalog. Creating tables (CREATE TABLE), schema/partition DDL updates, View/UDF catalogs, and server-side scan planning (/plan) are not yet available via REST; perform those DDL operations in Singdata Lakehouse.