April 16, 2024 Lakehouse Platform Release Notes
In this release (Release 2024.04.16), we have introduced a series of new features, enhancements, and fixes. Additionally, we have made some updates to the product's behavior. Please note that these updates will be gradually rolled out to the following regions:
- Alibaba Cloud Shanghai Region
- Tencent Cloud Shanghai Region
- Alibaba Cloud Singapore Region
- Tencent Cloud Beijing Region
Import and Export Updates
COPY INTO supports exporting table data to VOLUME
The COPY INTO command now supports exporting tables or query results as files to a specified directory in the data lake storage Volume. You can configure the export file storage format, compression type, and naming rules.
For more information, please refer to the COPY INTO Export Files documentation.
COPY INTO supports importing BSON format files
The COPY INTO command now supports importing BSON format files. For example, you can upload MongoDB data files in .bson format exported by mongodump to object storage and use the COPY INTO command to import them in bulk.
For more information, please refer to the COPY INTO Import Files documentation.
Data lake query supports using SQL to query BSON format files
Data lake query has extended support for BSON format files in addition to CSV, TEXT, PARQUET, and ORC file formats. You can use SQL to directly query BSON format files in the Volume.
For more information, please refer to the Data Lake Query documentation.
Real-time Incremental Computing Updates
Dynamic tables support automatic generation of intermediate tables to improve incremental optimization for complex logic SQL
In actual production environments, SQL ETL tasks often use temporary views (such as CTE) for joins to meet complex processing logic. These temporary views often can only determine the scope of the join after the computation is complete, making incremental processing difficult. For such scenarios, dynamic tables support automatically decomposing SQL tasks and dynamically creating intermediate temporary tables based on the processing logic to achieve incremental processing of complex logic.
Dynamic table processing performance optimization
Dynamic tables have added Dynamic Filter pushdown optimization to improve processing performance.