August 7, 2023 Lakehouse Studio Release Notes

New Features Introduction

Comprehensive Product Interface Upgrade

To enhance your user experience, we have comprehensively upgraded the visual branding of the product and optimized the information architecture, making navigation more intuitive and understandable. Additionally, we have introduced a workspace feature to help you manage tasks more efficiently.

We have added a user login state control feature to ensure that active users can stay logged in, while users who have been inactive for a set period will be automatically logged out to ensure account security.

Data Integration Optimization

We have expanded the data integration capabilities, now supporting offline synchronization, and added read and write support for ES data sources (version 7.x), as well as read support for Tencent Cloud COS and Alibaba Cloud SLS data sources. Additionally, we have added support for various data sources in bulk, including Oracle, DB2, Greenplum, ADB MySQL, ADB PG, Hologres, PolarDB, SelectDB, StarRocks, MariaDB, TiDB, etc., to meet your diverse data integration needs.

Enhanced Data Development Features

To make data development more convenient, we have added a feature that automatically displays a comparison between the submitted content and the production environment content when submitting files. Additionally, it supports comparing and rolling back the version of the submitted file or the saved version directly with the current local content. The data display component of the running results has also been optimized to support sorting and filtering functions. Python and Shell task runs now support configurable resource sizes to accommodate different task requirements.

Operations Center Feature Upgrade

We have upgraded the real-time synchronization task startup feature to support configuration of startup parameters. Additionally, the maximum concurrency configuration feature for Virtual Cluster analysis has been enhanced.

Data Management and Monitoring Alert Optimization

The data management feature now supports importing local file data and allows setting file configurations such as headers and delimiters, with the maximum size limit relaxed to 100Mb. The monitoring alert feature now supports the enterprise WeChat alert channel, making it easier for you to receive important notifications in a timely manner.

Data Quality Monitoring

We have introduced a data quality module that supports configuring quality rule monitoring for Lakehouse and sends alerts when issues are detected, helping you better ensure data quality.

Optimization Improvements

Interface and Operation Optimization

We have added a quick copy button in the code compilation box to facilitate quick copying of code. The data source list filter now supports option search functionality, helping you find the required data source more quickly. We have also added the CMD+D shortcut key, allowing you to copy the line where the cursor is located or the selected content and insert it into the next line of the cursor position.

Task Scheduling Function Improvement

When selecting upstream dependency nodes, we have optimized the display of the full path of the nodes to help you identify nodes with the same name. The intelligent recommendation feature for scheduling dependencies and output tables in the scheduling configuration has also been optimized. We have also optimized the Schema parsing logic of the output table, taking the definition in the scheduling configuration as the standard, and supporting self-dependency across cycles (such as cross-day) to depend on the instance of the previous cycle.

Operations Center Experience Enhancement

We have optimized the default display time range of the instance list and improved the page loading speed. The batch operation limit has been expanded from a maximum of 20 to 50. A breadcrumb navigation feature has been added, and the issue of too many open tabs in the Operations Center has been optimized. We have also optimized the display method of the task DAG diagram and the logic of the data supplementation operation, supporting the direct selection of the natural date of the task schedule instead of the business date. Additionally, we have revamped the tables in the Operations Center, including instance details - operation logs, periodic task details - operation logs, periodic task lists, etc.

Monitoring Alerts and Job History Optimization

We have optimized the display of suppression status in alert events, revealing suppression time and status click functionality. In terms of job history, we have optimized the display information and performance of the Job Profile page.

Defect Fixes

Data Integration and Task Development Fixes

We have fixed the issue where tinyint in MySQL data sources was displayed as bit, and the issue where the task stop and run status were inconsistent.

Task Scheduling Issue Fixes

We have fixed the issue where downstream nodes were still scheduled normally when upstream nodes were paused, and the issue where self-dependency relationships might be lost when generating task instances on a schedule.

Job History Issue Fixes

We have fixed the issue of abnormal time display in the job execution plan and resolved the issue where opening job history would fail when the job plan was too large.