January 5, 2024 Lakehouse Studio Release Notes

This update (Release 2024.01.05) aims to provide users with a more comprehensive and convenient data management and analysis experience. We have introduced a series of new features, optimizations, and bug fixes. The update will be gradually rolled out to the following regions:

Alibaba Cloud Shanghai Region
Tencent Cloud Shanghai Region
Alibaba Cloud Singapore Region
Tencent Cloud Beijing Region

Note: The update will be completed within one to two weeks from the release date, depending on your region.

Incompatible Changes

No incompatible changes were introduced in this update.

New Features and Enhancements

Data Integration Enhancements

Full Incremental Real-time Synchronization: A new feature supports real-time mirroring of entire MySQL and PostgreSQL databases to Lakehouse, as well as multi-table merge synchronization. For example, you can synchronize the orders table and customers table from the sales database in real-time, ensuring the timeliness and accuracy of data analysis.

Task Development Improvements

Full Download: You can now download query results in CSV format to your local machine for further analysis and data exploration. For example, you can download user behavior data and further process it using Excel or other analysis tools.
Enhanced Data Tree: The data tree now supports name keyword search and provides features such as data preview, data upload, and adding table names/fields to the editor.
External Parameters: This new feature allows you to quickly view the parameters used in a task and assign values to run the task. For example, in a data synchronization task, you can set and adjust concurrency parameters to optimize task execution efficiency.

Data Asset Management

Data Asset Map: Integrates the original data catalog and database functions, providing a unified entry point for viewing data asset information. It offers unified data retrieval and query capabilities, data search, visual schema creation, and visual data object creation.

Operations Center Optimization

Enhanced DAG Functionality: The DAG in periodic task details and instance details now supports grouping and multi-level expansion, helping you better understand the task execution process.
Batch Data Completion: A new feature that improves operational efficiency and simplifies the data completion process.

Monitoring and Alerts Expansion

New Monitoring and Alert Types: Added various monitoring events and metrics to help you better track task execution and data quality.

Data Quality Management

Metric Value Change Verification: A new feature supports comparing data fluctuations to ensure data quality meets expectations.
Lakehouse View and MV Quality Monitoring: Added support for configuring quality monitoring rules for Lakehouse views and materialized views.

Security Enhancements

Multi-Factor Authentication (MFA): Added support for MFA to enhance account security.

Optimizations and Improvements

Data Integration: Improved the definition and usage of parameters in task configuration, making the configuration more intuitive and understandable.
Task Development: Optimized the auto-completion of SQL single-table queries, added a LIMIT restriction to reduce resource consumption.

Bug Fixes

Data Integration: Fixed an issue where running a postgresql-lh integration task with the splitpk field filled in would result in an error.
Data Integration: Fixed a timezone deviation issue when converting MySQL timestamp type to Lakehouse string and timestamp_ltz.
Data Upload: Fixed an issue where data import failed due to delimiter handling problems.
Data Quality: Fixed an issue where the time parameter value was incorrectly replaced when triggering validation rules for periodic tasks.
Operations Center: Fixed an inconsistency between the cluster information displayed in the list and the cluster information in task configuration.