Create Compute Cluster

Description

This feature allows users to create a compute cluster in the SQL workspace based on the specified name and configuration. The compute cluster (Virtual-Cluster, abbreviated as: VCluster) is a compute resource cluster service provided by Singdata Lakehouse, offering resources such as CPU, memory, and temporary storage required for executing query analysis. Users can use the compute cluster to perform various ETL, streaming analysis, ad-hoc queries, and data integration tasks. When executing SQL Select queries or various DML operations (such as DELETE, INSERT, UPDATE, etc.) that require computation in Lakehouse, the compute cluster will be used.

The compute cluster includes three types: General Purpose Virtual Cluster (abbreviated as: GP type), Analytics Purpose Virtual Cluster (abbreviated as: AP type), and Integration Virtual Cluster.

In the General Purpose compute cluster, jobs submitted to the compute cluster share the cluster's computing resources, suitable for handling offline jobs; the Analytics Purpose cluster has features such as multiple compute instances and automatic scaling, suitable for handling online, high-concurrency jobs. The Integration cluster is specifically used for data integration tasks.

Syntax

-- 创建计算集群
CREATE VCLUSTER [IF NOT EXISTS] <name>
objectProperties
[COMMENT '']

--参数说明 
--创建分析型计算集群（ANALYTICS PURPOSE VIRTUAL CLUSTER）适用属性           
objectProperties ::=
            VCLUSTER_SIZE = num --integer from 1 to 256
            VCLUSTER_TYPE = ANALYTICS
            MIN_REPLICAS = num
            MAX_REPLICAS = num
            AUTO_SUSPEND_IN_SECOND = num
            AUTO_RESUME = TRUE| FALSE
            MAX_CONCURRENCY = num
            QUERY_RUNTIME_LIMIT_IN_SECOND = num
            PRELOAD_TABLES = "<schema_name>.<table_name>[,<schema_name>.<table_name>,...]"
            AUTO_PRELOAD_IN_SECOND = num
            
--创建通用型计算集群（GENERAL PURPOSE VIRTUAL CLUSTER）适用属性             
objectProperties ::=
            [VCLUSTER_SIZE = num | MIN_VCLUSTER_SIZE=num  MAX_VCLUSTER_SIZE=num] --integer from 1 to 256
            VCLUSTER_TYPE = GENERAL 
            AUTO_SUSPEND_IN_SECOND = num
            AUTO_RESUME = TRUE| FALSE
            QUERY_RUNTIME_LIMIT_IN_SECOND = num
            QUERY_RESOURCE_LIMIT_RATIO=num;

--创建同步型计算集群（INTEGRATION VIRTUAL CLUSTER）适用属性             
objectProperties ::=
            [VCLUSTER_SIZE = num | MIN_VCLUSTER_SIZE=num  MAX_VCLUSTER_SIZE=num] --integer from 1 to 256
            VCLUSTER_TYPE = INTEGRATION 
            AUTO_SUSPEND_IN_SECOND = num
            AUTO_RESUME = TRUE| FALSE
            QUERY_RUNTIME_LIMIT_IN_SECOND = num;

1 .name: The name of the compute cluster. It must be unique within the workspace and cannot be changed once created. Naming rules: 3 to 28 characters, only letters, underscores, and decimal numbers (0-9) are supported, and spaces are not allowed.

2. objectProperties: The properties that can be specified when creating a compute cluster, along with their specific meanings and values, are as follows:

3. Specify the maximum and minimum values for GP type VC during creation

CREATE VCLUSTER [IF NOT EXISTS] <name> 
VCLUSTER_TYPE=GENERAL 
MIN_VCLUSTER_SIZE=num 
MAX_VCLUSTER_SIZE=num;

• VCLUSTER_SIZE, MIN_VCLUSTER_SIZE, and MAX_VCLUSTER_SIZE cannot be set simultaneously.

4. comment Specify the description information of the computing cluster, supporting up to 1024 characters.

Usage Example

1. Create a computing cluster using default properties:

   CREATE VCLUSTER sample_vc;
   

2. Specify the creation of a general-purpose computing cluster, XSMALL specification, auto-start, auto-stop time of 60 seconds, maximum job execution time of 600 seconds:

   CREATE VCLUSTER demo_gp_vcluster 
   VCLUSTER_SIZE = 1 
   VCLUSTER_TYPE = GENERAL 
   AUTO_SUSPEND_IN_SECOND = 60 
   AUTO_RESUME = TRUE 
   QUERY_RUNTIME_LIMIT_IN_SECOND = 600;
   

Specify the creation of an analytical computing cluster, XSMALL specification, auto-start, auto-stop time of 1 minute, minimum instance count of 1, maximum instance count of 2, maximum concurrency per instance of 16, maximum job execution time of 600 seconds, pre-read data from public.demo and billing.payment tables, pull table data cache every 600 seconds:

CREATE VCLUSTER demo_ap_vcluster 
VCLUSTER_SIZE = 1
VCLUSTER_TYPE = ANALYTICS
MIN_REPLICAS = 1
MAX_REPLICAS = 2
MAX_CONCURRENCY = 16
AUTO_SUSPEND_IN_SECOND = 60 
AUTO_RESUME = TRUE 
QUERY_RUNTIME_LIMIT_IN_SECOND = 600
PRELOAD_TABLES = 'public.demo,billing.payment';