November 9, 2023 Lakehouse Studio Release Notes

Overview

This update (version number: 2023.11.09) aims to provide you with a better user experience by adding a series of practical features, enhancing existing functions, and fixing some issues. The update will be gradually rolled out in stages to the following regions:

• Tencent Cloud Shanghai Region
• Alibaba Cloud Shanghai Region
• Alibaba Cloud Singapore Region

Note: The update will be completed within one to two weeks from the release date, depending on your region.

Update Contents

Incompatible Changes

1. Global Update: To improve system stability and security, we have revamped multi-region domain names and gateways, now using centralized control. Please note that login domain names and URLs have changed.
2. Account Security: A new account login security lock mechanism has been added. If the number of consecutive login failures reaches 10, the account will be temporarily locked for 1 minute.

New Features

1. Product Homepage Upgrade: A brand-new user guide and key metrics display area to help you quickly understand core functions and directly access commonly used workspaces.

   • New Homepage

2. Data Integration Preview: Now supports real-time full database mirroring synchronization for PostgreSQL and MySQL databases, including task creation, operation, and basic monitoring and maintenance.

   • Data Integration Sync

3. Enhanced Data Integration: Added support for multi-concurrent split keys for String type fields, improving data processing flexibility.

4. Data Management Expansion: Increased the maximum upload size for a single file to 2GB, facilitating the handling of larger-scale data.

   • Data Upload Expansion

5. Workspace Encryption: Added storage encryption feature, which can be configured when creating a workspace. Once enabled, data in the workspace will be encrypted using the AES-256 algorithm by default, and this setting cannot be changed.

   • Storage Encryption Configuration

6. Operations Center Optimization: Optimized the periodic task details and instance details pages, providing a more intuitive information display. Instance detail graphs support grouping and aggregation by various conditions.

7. Granular Data Replenishment: Added a data replenishment feature, allowing users to finely control data replenishment for the current node and downstream nodes.

8. Monitoring and Alert Linkage: Alert event status will be linked with task instance status. When the instance recovers to a successful status, the alert event will automatically close.

9. Data Quality Rule List: Added a quality rule list page to facilitate viewing and managing all configured quality rules, supporting trial runs, editing, and other operations.

   • Quality Rule List

10. Quality Rule Scheduled Triggers: Added support for more trigger frequencies, such as multiple times a day, once a week, etc.

Optimizations and Improvements

1. Task Development Search: Added search filter functionality to the task tree, allowing quick location of required tasks through task name keywords.

   • Task Search Filter

2. SQL Script Task Optimization: Automatically adds a limit constraint to SQL script task query statements, displayed on the run button, with a maximum limit of 10,000.

3. Scheduling Stability Enhancement: Optimized scheduling logic, including instance generation and execution queues, to improve task scheduling stability.

4. Operations Center List Optimization: Optimized the display information of task lists and instance lists, supporting user customization.

5. Compute Cluster Page Optimization: Optimized the compute cluster list and details pages, providing usage consumption metrics and supporting custom columns.

6. Data Management Performance Improvement: Optimized the performance of directory tree loading and display, improving the efficiency of handling massive nodes.

7. Custom SQL Optimization: Increased the code length limit to 5000 lines to meet the needs of more complex SQL writing.

8. Security Center Permission Simplification: Supports using the all keyword when granting permissions, simplifying the permission granting process.

9. Workspace Role Simplification: The workspace administrator role (workspace_admin) now has all permissions within the space, eliminating the need to grant additional roles.

Bug Fixes

1. Data Integration: Fixed the issue with mapping constant fields in partition table partition fields.
2. Data Integration: Resolved the issue where synchronization failed when the OSS file suffix was not .txt.
3. Task Development: Fixed the issue of occasional memory leaks causing page crashes after the page was open for a long time.
4. Task Scheduling: Fixed the issue where the task status was still displayed as failed on the page after successful data supplementation.
5. Monitoring and Alerts: Fixed the occasional alert loss issue in the Feishu alert scenario.