借你慧眼，把 StarRocks Compaction 看得明明白白
作者：Ding_Kai
- 2024-08-13 北京
本文字数：16085 字
阅读完需：约 53 分钟
说明
StarRocks 中 Compaction 是为了将不同版本的数据文件进行整合，合并成大文件的动作，减少系统中小文件数量，进而提升查询效率。相比于存算一体表，StarRocks 存算分离实现了新的 Compaction 调度机制，表现为：
Compaction 调度由 FE 发起，BE 执行。FE 按照 Partition 为单位来发起 Compaction 任务
Compaction 会生成一个新版本，也走导入的写数据、commit、publish version 这套完整流程
本文提供 Compaction 相关的参数调优说明，供实践参考。关于 StarRocks 存算分离的 Compaction 原理解读，可移步这里：
丁凯：StarRocks 存算分离 Compaction 原理33 赞同 · 2 评论文章
查看分区的 Compaction score
注意：以下命令需要连接 Leader FE 节点执行
StarRocks 在内部为每个分区（Partition) 维护了一个 Compaction Score 值，它反映了分区当前数据文件合并情况，Compaction score 越高，代表了数据文件合并程度越低。
StarRocks 提供了命令可查看 Partition 当前的 Compaction Score，FE 会以此作为发起 Compaction 任务的参考，用户也可以此作为判断当前 Partition 是否存在版本数过多的依据：
方法 1：
MySQL [(none)]> show proc '/dbs/load_benchmark/store_sales/partitions';+-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+| PartitionId | PartitionName | CompactVersion | VisibleVersion | NextVersion | State | PartitionKey | Range | DistributionKey | Buckets | DataSize | RowCount | CacheTTL | AsyncWrite | AvgCS | P50CS | MaxCS | +-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+| 38028 | store_sales | 913 | 921 | 923 | NORMAL | | | ss_item_sk, ss_ticket_number | 64 | 15.6GB | 273857126 | 2592000 | false | 10.00 | 10.00 | 10.00 |+-------------+---------------+----------------+----------------+-------------+--------+--------------+-------+------------------------------+---------+----------+-----------+----------+------------+-------+-------+-------+1 row in set (0.20 sec)复制代码
方法 2：自从新版本 3.1.9 & 3.2.4 ，我们在系统表中增加了 partitions_meta 表，方便用户通过各种复杂 SQL 来查看系统所有 Partition 信息：
mysql> select * from information_schema.partitions_meta order by Max_CS;+--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+--------------------------------------------------------------------------------------------------------+| DB_NAME | TABLE_NAME | PARTITION_NAME | PARTITION_ID | COMPACT_VERSION | VISIBLE_VERSION | VISIBLE_VERSION_TIME | NEXT_VERSION | PARTITION_KEY | PARTITION_VALUE | DISTRIBUTION_KEY | BUCKETS | REPLICATION_NUM | STORAGE_MEDIUM | COOLDOWN_TIME | LAST_CONSISTENCY_CHECK_TIME | IS_IN_MEMORY | IS_TEMP | DATA_SIZE | ROW_COUNT | ENABLE_DATACACHE | AVG_CS | P50_CS | MAX_CS | STORAGE_PATH |+--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+--------------------------------------------------------------------------------------------------------+| tpcds_1t | call_center | call_center | 11905 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | cc_call_center_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 12.3KB | 42 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11906/11905 || tpcds_1t | web_returns | web_returns | 12030 | 3 | 3 | 2024-03-17 08:40:48 | 4 | | | wr_item_sk, wr_order_number | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 3.5GB | 71997522 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/12031/12030 || tpcds_1t | warehouse | warehouse | 11847 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | w_warehouse_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 4.2KB | 20 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11848/11847 || tpcds_1t | ship_mode | ship_mode | 11851 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | sm_ship_mode_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.7KB | 20 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11852/11851 || tpcds_1t | customer_address | customer_address | 11790 | 0 | 2 | 2024-03-17 08:32:19 | 3 | | | ca_address_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 120.9MB | 6000000 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11791/11790 || tpcds_1t | time_dim | time_dim | 11855 | 0 | 2 | 2024-03-17 08:30:48 | 3 | | | t_time_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 864.7KB | 86400 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11856/11855 || tpcds_1t | web_sales | web_sales | 12049 | 3 | 3 | 2024-03-17 10:14:20 | 4 | | | ws_item_sk, ws_order_number | 128 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 47.7GB | 720000376 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/12050/12049 || tpcds_1t | store | store | 11901 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | s_store_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 95.6KB | 1002 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11902/11901 || tpcds_1t | web_site | web_site | 11928 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | web_site_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 13.4KB | 54 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11929/11928 || tpcds_1t | household_demographics | household_demographics | 11932 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | hd_demo_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 2.1KB | 7200 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11933/11932 || tpcds_1t | web_page | web_page | 11936 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | wp_web_page_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 43.5KB | 3000 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11937/11936 || tpcds_1t | customer_demographics | customer_demographics | 11809 | 0 | 2 | 2024-03-17 08:30:49 | 3 | | | cd_demo_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 2.7MB | 1920800 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11810/11809 || tpcds_1t | reason | reason | 11874 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | r_reason_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.9KB | 65 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11875/11874 | | tpcds_1t | promotion | promotion | 11940 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | p_promo_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 69.6KB | 1500 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11941/11940 || tpcds_1t | income_band | income_band | 11878 | 0 | 2 | 2024-03-17 08:30:48 | 3 | | | ib_income_band_sk | 1 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 727B | 20 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11879/11878 || tpcds_1t | catalog_page | catalog_page | 11944 | 0 | 2 | 2024-03-17 08:30:52 | 3 | | | cp_catalog_page_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.8MB | 30000 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11945/11944 || tpcds_1t | item | item | 11882 | 0 | 2 | 2024-03-17 08:30:51 | 3 | | | i_item_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 37.1MB | 300000 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11883/11882 || tpcds_1t | store_returns | store_returns | 11755 | 3 | 3 | 2024-03-17 09:02:48 | 4 | | | sr_item_sk, sr_ticket_number | 32 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 11.3GB | 287999764 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11756/11755 || tpcds_1t | date_dim | date_dim | 11828 | 0 | 2 | 2024-03-17 08:30:47 | 3 | | | d_date_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.5MB | 73049 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11829/11828 || tpcds_1t | catalog_sales | catalog_sales | 12215 | 3 | 3 | 2024-03-17 11:44:37 | 4 | | | cs_item_sk, cs_order_number | 256 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 94.7GB | 1439982416 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/12216/12215 || tpcds_1t | store_sales | store_sales | 12474 | 3 | 3 | 2024-03-17 13:33:04 | 4 | | | ss_item_sk, ss_ticket_number | 256 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 133.4GB | 2879987999 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/12475/12474 || _statistics_ | histogram_statistics | histogram_statistics | 11729 | 0 | 1 | 2024-03-17 08:29:45 | 2 | | | table_id, column_name | 10 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 0B | 0 | 1 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10004/11730/11729 || _statistics_ | external_column_statistics | external_column_statistics | 11742 | 0 | 1 | 2024-03-17 08:29:45 | 2 | | | table_uuid, partition_name, column_name | 10 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 0B | 0 | 1 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10004/11743/11742 || tpcds_1t | catalog_returns | catalog_returns | 12180 | 3 | 3 | 2024-03-17 08:51:32 | 4 | | | cr_item_sk, cr_order_number | 32 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 7.4GB | 143996756 | 0 | 0 | 0 | 0 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/12181/12180 || _statistics_ | table_statistic_v1 | table_statistic_v1 | 11703 | 0 | 4 | 2024-03-17 10:24:32 | 5 | | | table_id, column_name, db_id | 10 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 55KB | 77 | 1 | 2.8 | 3 | 3 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10004/11704/11703 || tpcds_1t | inventory | inventory | 11963 | 3 | 3 | 2024-03-17 08:52:40 | 4 | | | inv_item_sk | 64 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.4GB | 783000000 | 0 | 1.046875 | 0 | 4 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11964/11963 || tpcds_1t | customer | customer | 11909 | 0 | 2 | 2024-03-17 08:33:36 | 3 | | | c_customer_sk | 16 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 640.6MB | 12000000 | 0 | 3.0625 | 3 | 5 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10328/11910/11909 || _statistics_ | column_statistics | column_statistics | 11716 | 14 | 23 | 2024-03-17 10:19:27 | 24 | | | table_id, partition_id, column_name | 10 | 1 | HDD | 9999-12-31 23:59:59 | NULL | 0 | 0 | 1.4MB | 348 | 1 | 7.7 | 8 | 9 | s3://starrocks-cloud-data-zhangjiakou/dingkai/536a3c77-52c3-485a-8217-781734a970b1/db10004/11717/11716 |+--------------+----------------------------+----------------------------+--------------+-----------------+-----------------+----------------------+--------------+---------------+-----------------+-----------------------------------------+---------+-----------------+----------------+---------------------+-----------------------------+--------------+---------+-----------+------------+------------------+----------+--------+--------+--------------------------------------------------------------------------------------------------------+28 rows in set (0.04 sec)复制代码
需要关注以下参数：
AvgCS：当前 Partition 上所有 Tablet 平均 Compaction Score
MaxCS: 当前 Partition 上所有 Tablet 最大 Compaction Score
查看 Compaction 任务
随着导入任务的执行，系统内部也在不断地调度执行 Compaction 任务，这些任务会被发往计算节点 CN 执行，系统也提供了一系列命令可以查看当前 Compaction 任务执行情况。
首先，用户可以通过如下命令来观察系统当前所有 Compaction 任务的整体情况：
MySQL [(none)]> show proc '/compactions';+----------------------------------------------------+--------+---------------------+------------+---------------------+---------------------------------------------------------------------------------+| Partition | TxnID | StartTime | CommitTime | FinishTime | Error |+----------------------------------------------------+--------+---------------------+------------+---------------------+---------------------------------------------------------------------------------+| load_benchmark.store_sales.store_sales | 197562 | 2023-05-24 15:50:33 | 2023-05-24 15:51:00 | 2023-05-24 15:51:02 | NULL |+----------------------------------------------------+--------+---------------------+------------+---------------------+---------------------------------------------------------------------------------+1 rows in set (0.21 sec)复制代码
上面显示了当前有一个 Compaction 任务正在进行，其各字段含义如下：
Partition：当前正在进行的 Compaction 任务针对的 Partition
TxnID：FE 为当前 Compaction 任务分配的事务 id
StartTime：该 Compaction 任务的开始时间
CommitTime：该 Compaction 任务的 Commit 时间
FinishTime：该 Compaction 任务的结束时间
Error：该 Compaction 任务出错信息，如果一切正常，值为 NULL
上面的命令展示了每个 Partition 上对应的 Compaction 任务总体情况。而每个 Compaction 任务底层会被按照 Tablet 分为多个子任务，系统也提供了如下命令来观察每个 Compaction 子任务的详细进展情况：
MySQL [(none)]> select * from information_schema.be_cloud_native_compactions where TXN_ID = 197562;+-------+--------+-----------+---------+---------+------+---------------------+---------------------+----------+--------+| BE_ID | TXN_ID | TABLET_ID | VERSION | SKIPPED | RUNS | START_TIME | FINISH_TIME | PROGRESS | STATUS |+-------+--------+-----------+---------+---------+------+---------------------+---------------------+----------+--------+| 36027 | 197562 | 38033 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 36027 | 197562 | 38038 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 36027 | 197562 | 38039 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:39 | 100 | OK || 36027 | 197562 | 38040 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:39 | 100 | OK || 36027 | 197562 | 38044 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:42 | 100 | OK || 36027 | 197562 | 38047 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:43 | 100 | OK || 36027 | 197562 | 38055 | 365 | 0 | 1 | 2023-05-24 15:50:39 | 2023-05-24 15:50:43 | 100 | OK || 36027 | 197562 | 38056 | 365 | 0 | 1 | 2023-05-24 15:50:39 | 2023-05-24 15:50:43 | 100 | OK || 36027 | 197562 | 38058 | 365 | 0 | 1 | 2023-05-24 15:50:42 | 2023-05-24 15:50:47 | 100 | OK || 36027 | 197562 | 38060 | 365 | 0 | 1 | 2023-05-24 15:50:43 | 2023-05-24 15:50:47 | 100 | OK || 36027 | 197562 | 38063 | 365 | 0 | 1 | 2023-05-24 15:50:43 | 2023-05-24 15:50:48 | 100 | OK || 36027 | 197562 | 38066 | 365 | 0 | 1 | 2023-05-24 15:50:43 | 2023-05-24 15:50:48 | 100 | OK || 36027 | 197562 | 38070 | 365 | 0 | 1 | 2023-05-24 15:50:47 | NULL | 85 | OK || 36027 | 197562 | 38071 | 365 | 0 | 1 | 2023-05-24 15:50:47 | NULL | 61 | OK || 36027 | 197562 | 38080 | 365 | 0 | 1 | 2023-05-24 15:50:48 | NULL | 33 | OK || 36027 | 197562 | 38083 | 365 | 0 | 1 | 2023-05-24 15:50:48 | NULL | 29 | OK || 36027 | 197562 | 38085 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36027 | 197562 | 38086 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36027 | 197562 | 38090 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36027 | 197562 | 38091 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36027 | 197562 | 38094 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36026 | 197562 | 38031 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:39 | 100 | OK || 36026 | 197562 | 38037 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 36026 | 197562 | 38042 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 36026 | 197562 | 38043 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 36026 | 197562 | 38045 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:42 | 100 | OK || 36026 | 197562 | 38048 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:43 | 100 | OK || 36026 | 197562 | 38049 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:43 | 100 | OK || 36026 | 197562 | 38051 | 365 | 0 | 1 | 2023-05-24 15:50:39 | 2023-05-24 15:50:43 | 100 | OK || 36026 | 197562 | 38054 | 365 | 0 | 1 | 2023-05-24 15:50:42 | 2023-05-24 15:50:46 | 100 | OK || 36026 | 197562 | 38057 | 365 | 0 | 1 | 2023-05-24 15:50:43 | 2023-05-24 15:50:47 | 100 | OK || 36026 | 197562 | 38062 | 365 | 0 | 1 | 2023-05-24 15:50:43 | 2023-05-24 15:50:47 | 100 | OK || 36026 | 197562 | 38069 | 365 | 0 | 1 | 2023-05-24 15:50:43 | 2023-05-24 15:50:48 | 100 | OK || 36026 | 197562 | 38073 | 365 | 0 | 1 | 2023-05-24 15:50:46 | NULL | 98 | OK || 36026 | 197562 | 38074 | 365 | 0 | 1 | 2023-05-24 15:50:47 | NULL | 77 | OK || 36026 | 197562 | 38075 | 365 | 0 | 1 | 2023-05-24 15:50:47 | NULL | 51 | OK || 36026 | 197562 | 38077 | 365 | 0 | 1 | 2023-05-24 15:50:48 | NULL | 33 | OK || 36026 | 197562 | 38078 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36026 | 197562 | 38079 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36026 | 197562 | 38081 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36026 | 197562 | 38082 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36026 | 197562 | 38084 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 36026 | 197562 | 38089 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 10004 | 197562 | 38032 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 10004 | 197562 | 38034 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 10004 | 197562 | 38035 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 10004 | 197562 | 38036 | 365 | 0 | 1 | 2023-05-24 15:50:34 | 2023-05-24 15:50:38 | 100 | OK || 10004 | 197562 | 38041 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:42 | 100 | OK || 10004 | 197562 | 38046 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:42 | 100 | OK || 10004 | 197562 | 38050 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:42 | 100 | OK || 10004 | 197562 | 38052 | 365 | 0 | 1 | 2023-05-24 15:50:38 | 2023-05-24 15:50:42 | 100 | OK || 10004 | 197562 | 38053 | 365 | 0 | 1 | 2023-05-24 15:50:42 | 2023-05-24 15:50:46 | 100 | OK || 10004 | 197562 | 38059 | 365 | 0 | 1 | 2023-05-24 15:50:42 | 2023-05-24 15:50:47 | 100 | OK || 10004 | 197562 | 38061 | 365 | 0 | 1 | 2023-05-24 15:50:42 | 2023-05-24 15:50:46 | 100 | OK || 10004 | 197562 | 38064 | 365 | 0 | 1 | 2023-05-24 15:50:42 | 2023-05-24 15:50:47 | 100 | OK || 10004 | 197562 | 38065 | 365 | 0 | 1 | 2023-05-24 15:50:46 | NULL | 99 | OK || 10004 | 197562 | 38067 | 365 | 0 | 1 | 2023-05-24 15:50:46 | NULL | 92 | OK || 10004 | 197562 | 38068 | 365 | 0 | 1 | 2023-05-24 15:50:47 | NULL | 87 | OK || 10004 | 197562 | 38072 | 365 | 0 | 1 | 2023-05-24 15:50:47 | NULL | 89 | OK || 10004 | 197562 | 38076 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 10004 | 197562 | 38087 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 10004 | 197562 | 38088 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 10004 | 197562 | 38092 | 365 | 0 | 0 | NULL | NULL | 0 | OK || 10004 | 197562 | 38093 | 365 | 0 | 0 | NULL | NULL | 0 | OK |+-------+--------+-----------+---------+---------+------+---------------------+---------------------+----------+--------+64 rows in set (0.22 sec)复制代码
由于该 Partition 有 64 个 Tablet，因而，这里逐一展示了 64 个 Tablet 的 Compaction 完成情况，关注两个字段：
PROGRESS：代表 Tablet 当前 Compaction 进展情况，为百分比
STATUS：代表任务状态，如果有错误，这里会展示错误详细信息
取消 Compaction 任务
用户可以通过下面的命令来取消特定的 Compaction 任务。需要注意的是，该命令需要连接 Leader FE 节点执行
CANCEL COMPACTION WHERE TXN_ID = 123;复制代码
参数调优
StarRocks 中有如下参数来控制存算分离下的 Compaction 行为。
FE 参数
# 最小的Compaction score，低于该值的 Partition 不会发起Compaction任务lake_compaction_score_selector_min_score = 10.0;
# FE 上可同时发起的 Compaction Task 数量# 默认值为-1，即FE会根据系统中 BE 数量自动计算# 如果设置为0，则 FE 不会发起任何 Compaction 任务lake_compaction_max_tasks = -1;
# 控制show proc '/compactions' 显示的结果数量，默认为12lake_compaction_history_size = 12;lake_compaction_fail_history_size = 12;复制代码
FE 所有上述 Compaction 相关参数均可通过 SQL 命令动态修改，例如：
admin set frontend config ("lake_compaction_max_tasks" = "0");复制代码
BE / CN 参数
# 控制 BE/CN 上同时执行 Compaction 任务的线程数，默认值为4# 也即 BE 上可同时为多少个 Tablet进行 Compactioncompact_threads = 4
# BE 上 Compaction任务队列大小，控制可接收来自FE的最大Compaction 任务数# 默认值为100compact_thread_pool_queue_size = 100
# 单次 Compaction 任务最多合并的数据文件数量，默认为1000# 在实践中我们建议将该值调整为100，这样，每个 Compaction Task 可以更快速地结束# 且消耗更少的资源max_cumulative_compaction_num_singleton_deltas=100复制代码
BE 所有上述 Compaction 相关参数在新版本中已经支持动态修改，可以通过如下方式修改：
mysql> update information_schema.be_configs set value = 8 where name = "compact_threads";Query OK, 0 rows affected (0.01 sec)复制代码
最佳实践
因为 Compaction 对于查询性能的影响至关重要，因此，我们建议用户时刻关注系统中表与分区的后台数据合并情况，在这里我们给用户提供几点最佳实践的建议和指导。
关注 Compaction Score，根据指标配置告警。StarRocks 自带的 Grafana 监控模板已经包含了该指标
关注 Compaction 消耗的资源情况，尤其是内存。StarRocks 自带的 Grafana 监控模板已经包含了该指标
在资源较空闲时可增加计算节点上并行 Compaction 工作线程数，以加快 Compaction 任务的执行
划线
评论
复制
发布于: 刚刚阅读数: 2
Ding_Kai
关注
还未添加个人签名 2022-09-27 加入
还未添加个人简介
评论