pd-ctl 选项 --jq 格式化语法使用案例详解
作者: OnTheRoad 原文来源:https://tidb.net/blog/d71e3609
1. json 格式化工具 jq
在阅读关于 PD Control 工具使用的文档时,发现文档未对 –jq 格式化选项的语法作太多详细介绍。于是,诞生了本文档,希望对如我这种新手小白有一定的帮助
1.1. jq 简介
在介绍 pd-ctl 的命令行选项 –jq 之前,首先了解一下 Linux 中的命令行工具 jq。jq 是一个命令行的 JSON 文本格式化工具。jq 将给定的 Filter(过滤器)应用于其输入的 JSON 文本中,并将执行结果以 JSON 格式返回到标准输出中。
官方文档链接:https://stedolan.github.io/jq/manual/
1.2. jq 安装
1.3. jq 使用
1.3.1. 语法格式
1.3.2. jq 常用示例
1.3.2.1. 准备测试文本
首先,准备 2 份测试文件。文件 1(jsonstr.txt
)内容为一段未格式化的 JSON 字符串;文件 2(jsonfmt.txt
)内容为一段格式化后的 JSON 文本。
【注意】该测试文本为一个 TiDB 集群数据库中的 store 信息。可通过如下方式获取类似格式的 JSON 文本。
> ~]$ tiup ctl:v6.1.0 pd -u http://192.168.3.221:2379 store > jsonfmt.txt > ``` > > 其中 `http://192.168.3.221:2379` 为 PD 实例的地址,获取的json信息通过`>`重定向输入到`jsonfmt.txt`。 #### 1.3.2.2. JSON 字符串格式化 1. 直接通过 jq 命令,格式化文本
~]$ jq –tab -S ‘.’ jsonstr.txt{ “Skills”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ], “Age”: 38, “Birthday”: “19th Jan”, “Name”: “Jack”, “Email”: “Jack@outlook.com”, “Education”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professonal”: “Computer Science & Technology”, “Year”: “2007” }}
这里的过滤器`'.'`,表示只对输入的内容做 JSON 格式化,而不修改其内容,效果与下面的过滤器 'values' 一致。
~]$ cat jsonstr.txt | jq ‘values’{ “Skills”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ], “Age”: 38, “Birthday”: “19th Jan”, “Name”: “Jack”, “Email”: “Jack@outlook.com”, “Education”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” }}
2. 通过管道符调用 jq,格式化文本
~]$ cat jsonstr.txt | jq –tab -S ‘.’{ “Age”: 38, “Birthday”: “19th Jan”, “Education”: { “College”: “Electronics & Information Engineering”, “Professonal”: “Computer Science & Technology”, “University”: “LNTU”, “Year”: “2007” }, “Email”: “Jack@outlook.com”, “Name”: “Jack”, “Skills”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ]}
默认的缩进为 2 个空格,`--tab` 表示将缩进替换为 tab 制表符;`-S` 表示对格式化后的 JSON 文本按 `KEY` 排序。 > **【注意】**为简化描述,避免文档过于冗长,后续示例均以管道方式来描述 jq 的使用。 #### 1.3.2.3. 以列表形式获取 Key(只能获取 1 级的 Key 信息)
~]$ cat jsonstr.txt | jq ‘keys’[ “Age”, “Birthday”, “Education”, “Email”, “Name”, “Skills”]
这里的 `keys` 为关键字 #### 1.3.2.4. 提取指定 Key 的 Value(Key Filter)
jq ‘.Key1 名称, .Key2 名称, …’
1. 获取某个 Key 的 Value 当 JSON 文本过长,只想提取其中某个 Key 的 Value。可通过如下语法完成。
~]$ jq ‘.Education’ jsonstr.txt{ “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professonal”: “Computer Science & Technology”, “Year”: “2007”}
~]$ cat jsonstr.txt |jq ‘.Education’{ “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professonal”: “Computer Science & Technology”, “Year”: “2007”}
~]$ cat jsonstr.txt | jq ‘.Education, .Skills’{ “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007”}[ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex”]
jq ‘{Key1 名称}, {Key2 名称}, …’
~]$ cat jsonstr.txt | jq ‘{Education}’{ “Education”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” }}
~]$ cat jsonstr.txt | jq ‘{Education}, {Skills}’{ “Education”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” }}{ “Skills”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ]}
~]$ cat jsonstr.txt | jq ‘.[]’[ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex”]38“19th Jan”“Jack”“Jack@outlook.com”{ “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007”}
~]$ cat jsonstr.txt | jq ‘.Education[]’“LNTU”“Electronics & Information Engineering”“Computer Science & Technology”“2007”
通过 .stores[] 递归取出 stores 中每个元素(元素中包含 store 和 status 两部分内容);
再通过 .store 过滤出每个元素中的 store 部分的内容。
~]$ cat jsonfmt.txt | jq ‘.stores[].store’{ “id”: 1, “address”: “192.168.3.225:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.225:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078152, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660966497198077000, “state_name”: “Up”}{ “id”: 4, “address”: “192.168.3.224:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.224:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078147, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660966491404376800, “state_name”: “Up”}{ “id”: 5, “address”: “192.168.3.226:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.226:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078162, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660966497108902000, “state_name”: “Up”}
通过 .stores[].store 过滤出多个 store
对多个 store 应用 select 条件(id ==5)查询,筛选出包含 id == 5 的 store。
~]$ cat jsonfmt.txt | jq ‘.stores[].store | select(.id == 5)’{ “id”: 5, “address”: “192.168.3.226:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.226:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078162, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660966497108902000, “state_name”: “Up”}
jq ‘.Key 名称[元素下标, 元素下标,,]’
~]$ cat jsonstr.txt | jq ‘.Skills[]’“1.Oracle”“2.Python”“3.MySQL”“4.Latex”
~] cat jsonstr.txt | jq ‘.Skills[1]’“2.Python”~] cat jsonstr.txt | jq ‘.Skills[1,3]’“2.Python”“4.Latex”
jq ‘.Key 名称[起始下标: 结束下标(不含)]’
~] cat jsonstr.txt | jq ‘.Skills[0:1]’[ “1.Oracle”]
~]$ cat jsonstr.txt | jq ‘.Skills[0:2]’[ “1.Oracle”, “2.Python”]
~]$ cat jsonstr.txt | jq ‘.Skills[1:10]’[ “2.Python”, “3.MySQL”, “4.Latex”]
~]$ cat jsonstr.txt | jq ‘.Skills[-2,3]’“3.MySQL”“4.Latex”
~]$ cat jsonstr.txt | jq ‘.Skills[-2:1]’[]
~]$ cat jsonstr.txt | jq ‘.Education’{ “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007”}
~]$ cat jsonstr.txt | jq ‘.Education | length’4
~]$ cat jsonstr.txt | jq ‘.Education | .College’“Electronics & Information Engineering”
~]$ cat jsonstr.txt | jq ‘..’{ “Skills”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ], “Age”: 38, “Birthday”: “19th Jan”, “Name”: “Jack”, “Email”: “Jack@outlook.com”, “Education”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” }}[ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex”]“1.Oracle”“2.Python”“3.MySQL”“4.Latex”38“19th Jan”“Jack”“Jack@outlook.com”{ “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007”}“LNTU”“Electronics & Information Engineering”“Computer Science & Technology”“2007”
~]$ cat jsonstr.txt | jq ‘map(. | length)’[ 4, 38, 8, 4, 16, 4]
~]$ cat jsonstr.txt | jq ‘.Education, .Skills’{ “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007”}[ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex”]
~]$ cat jsonstr.txt | jq ‘[.Education, .Skills]’[ { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” }, [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ]]
~]$ cat jsonstr.txt | jq ‘{Education, Skills}’{ “Education”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” }, “Skills”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ]}
~]$ cat jsonstr.txt | jq ‘{EducationDetail: .Education, SkillInfo: .Skills}’{ “EducationDetail”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” }, “SkillInfo”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ]}
~]$ cat jsonstr.txt | jq ‘{EducationDetail: {Education}, SkillInfo: {Skills}}’{ “EducationDetail”: { “Education”: { “University”: “LNTU”, “College”: “Electronics & Information Engineering”, “Professnal”: “Computer Science & Technology”, “Year”: “2007” } }, “SkillInfo”: { “Skills”: [ “1.Oracle”, “2.Python”, “3.MySQL”, “4.Latex” ] }}
~]$ export PD_ADDR=http://192.168.3.222:2379~]$ tiup ctl:v6.1.0 pd config showStarting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd config show{ “replication”: { “enable-placement-rules”: “true”, “enable-placement-rules-cache”: “false”, “isolation-level”: “”, “location-labels”: “”, “max-replicas”: 3, “strictly-match-label”: “false” }, “schedule”: { “enable-cross-table-merge”: “true”, “enable-joint-consensus”: “true”, “high-space-ratio”: 0.7, “hot-region-cache-hits-threshold”: 3, “hot-region-schedule-limit”: 4, “hot-regions-reserved-days”: 0, “hot-regions-write-interval”: “10m0s”, “leader-schedule-limit”: 4, “leader-schedule-policy”: “count”, “low-space-ratio”: 0.8, “max-merge-region-keys”: 200000, “max-merge-region-size”: 20, “max-pending-peer-count”: 64, “max-snapshot-count”: 64, “max-store-down-time”: “30m0s”, “max-store-preparing-time”: “48h0m0s”, “merge-schedule-limit”: 8, “patrol-region-interval”: “10ms”, “region-schedule-limit”: 2048, “region-score-formula-version”: “v2”, “replica-schedule-limit”: 64, “split-merge-interval”: “1h0m0s”, “tolerant-size-ratio”: 0 }}
~]$ tiup ctl:v6.1.0 pd storeStarting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd store{ “count”: 3, “stores”: [ { “store”: { “id”: 1, “address”: “192.168.3.225:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.225:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078152, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660978688453748544, “state_name”: “Up” }, “status”: { “capacity”: “19.56GiB”, “available”: “15.7GiB”, “used_size”: “1.011GiB”, “leader_count”: 6, “leader_weight”: 1, “leader_score”: 6, “leader_size”: 417, “region_count”: 22, “region_weight”: 1, “region_score”: 6860013051.732055, “region_size”: 1811, “slow_score”: 1, “start_ts”: “2022-07-29T15:02:32+08:00”, “last_heartbeat_ts”: “2022-08-20T14:58:08.453748544+08:00”, “uptime”: “527h55m36.453748544s” } }, { “store”: { “id”: 4, “address”: “192.168.3.224:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.224:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078147, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660978692660915975, “state_name”: “Up” }, “status”: { “capacity”: “19.56GiB”, “available”: “15.69GiB”, “used_size”: “1.011GiB”, “leader_count”: 11, “leader_weight”: 1, “leader_score”: 11, “leader_size”: 932, “region_count”: 22, “region_weight”: 1, “region_score”: 6861560216.947071, “region_size”: 1811, “slow_score”: 1, “start_ts”: “2022-07-29T15:02:27+08:00”, “last_heartbeat_ts”: “2022-08-20T14:58:12.660915975+08:00”, “uptime”: “527h55m45.660915975s” } }, { “store”: { “id”: 5, “address”: “192.168.3.226:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.226:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078162, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660978688364585582, “state_name”: “Up” }, “status”: { “capacity”: “19.56GiB”, “available”: “15.69GiB”, “used_size”: “1.031GiB”, “leader_count”: 5, “leader_weight”: 1, “leader_score”: 5, “leader_size”: 462, “region_count”: 22, “region_weight”: 1, “region_score”: 6862846131.837539, “region_size”: 1811, “slow_score”: 1, “start_ts”: “2022-07-29T15:02:42+08:00”, “last_heartbeat_ts”: “2022-08-20T14:58:08.364585582+08:00”, “uptime”: “527h55m26.364585582s” } } ]}
~]$ tiup ctl:v6.1.0 pd region | jq ‘.’Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd region{ “count”: 22, “regions”: [ { “id”: 172, “start_key”: “7480000000000000FF5C00000000000000F8”, “end_key”: “7480000000000000FF5E00000000000000F8”, “epoch”: { “conf_ver”: 47, “version”: 68 }, “peers”: [ { “id”: 173, “store_id”: 1, “role_name”: “Voter” }, { “id”: 4467, “store_id”: 4, “role_name”: “Voter” }, { “id”: 4497, “store_id”: 5, “role_name”: “Voter” } ], “leader”: { “id”: 4467, “store_id”: 4, “role_name”: “Voter” }, “written_bytes”: 0, “read_bytes”: 0, “written_keys”: 0, “read_keys”: 0, “approximate_size”: 85, “approximate_keys”: 1079434 }, ……
]}
~]$ tiup ctl:v6.1.0 pd store –jq=“.stores[]” |jq ‘{store}, {status}’Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd store –jq=.stores[]{ “store”: { “id”: 4, “address”: “192.168.3.224:20160”, “version”: “6.1.0”, “status_address”: “192.168.3.224:20180”, “git_hash”: “080d086832ae5ce2495352dccaf8df5d40f30687”, “start_timestamp”: 1659078147, “deploy_path”: “/tidb-deploy/tikv-20160/bin”, “last_heartbeat”: 1660980592856686300, “state_name”: “Up” }}{ “status”: { “capacity”: “19.56GiB”, “available”: “15.69GiB”, “used_size”: “1.012GiB”, “leader_count”: 11, “leader_weight”: 1, “leader_score”: 11, “leader_size”: 932, “region_count”: 22, “region_weight”: 1, “region_score”: 6861587385.861746, “region_size”: 1811, “slow_score”: 1, “start_ts”: “2022-07-29T15:02:27+08:00”, “last_heartbeat_ts”: “2022-08-20T15:29:52.856686305+08:00”, “uptime”: “528h27m25.856686305s” }}……
~]$ tiup ctl:v6.1.0 pd store –jq=“.stores[]” |jq ‘{store_id: .store.id, addr: .store.address, state: .store.state_name, freespace: .status.available}’Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd store –jq=.stores[]{ “store_id”: 1, “addr”: “192.168.3.225:20160”, “state”: “Up”, “freespace”: “15.7GiB”}{ “store_id”: 4, “addr”: “192.168.3.224:20160”, “state”: “Up”, “freespace”: “15.69GiB”}{ “store_id”: 5, “addr”: “192.168.3.226:20160”, “state”: “Up”, “freespace”: “15.69GiB”}
~]$ tiup ctl:v6.1.0 pd store –jq=“.stores[]” |jq ‘{store_id: .store.id, addr: .store.address, state: .store.state_name, freespace: .status.available}’ | jq ‘. | select(.state=“Up”)’
~]$ tiup ctl:v6.1.0 pd store –jq=“.stores[]” |jq ‘{store_id: .store.id, addr: .store.address, state: .store.state_name, path: .store.deploy_path, freespace: .status.available}’ |jq ‘. |select(.path == “/tidb-deploy/tikv-20160/bin”)’
Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd store –jq=.stores[]{ “store_id”: 1, “addr”: “192.168.3.225:20160”, “state”: “Up”, “path”: “/tidb-deploy/tikv-20160/bin”, “freespace”: “15.7GiB”}{ “store_id”: 4, “addr”: “192.168.3.224:20160”, “state”: “Up”, “path”: “/tidb-deploy/tikv-20160/bin”, “freespace”: “15.69GiB”}{ “store_id”: 5, “addr”: “192.168.3.226:20160”, “state”: “Up”, “path”: “/tidb-deploy/tikv-20160/bin”, “freespace”: “15.69GiB”}
~]$ tiup ctl:v6.1.0 pd region | jq ‘.regions[] | {region_id: .id, peer_stores: [.peers[].store_id], leader_store: .leader.store_id}’Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd region{ “region_id”: 192, “peer_stores”: [ 1, 5, 4 ], “leader_store”: 4}{ “region_id”: 244, “peer_stores”: [ 5, 1, 4 ], “leader_store”: 5}……
~]$ tiup ctl:v6.1.0 pd region | jq ‘.regions[] | {region_id: .id, key: [.start_key,.end_key], peer_stores: [.peers[].store_id], leader_store: .leader.store_id}’
Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd region{ “region_id”: 196, “key”: [ “7480000000000000FF645F728000000000FF04F8C70000000000FA”, “7480000000000000FF645F728000000000FF0630A00000000000FA” ], “peer_stores”: [ 1, 4, 5 ], “leader_store”: 4}{ “region_id”: 204, “key”: [ “7480000000000000FF645F728000000000FF09CDB50000000000FA”, “7480000000000000FF645F728000000000FF0D6E840000000000FA” ], “peer_stores”: [ 5, 4, 1 ], “leader_store”: 4}……
~]$ tiup ctl:v6.1.0 pd region –jq=“.regions[]” | jq ‘{region_id: .id, peer_stores: [.peers[].store_id] | select(length != 3)}’
Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd region –jq=.regions[]{ “region_id”: 252, “peer_stores”: [ 4, 5 ]}{ “region_id”: 216, “peer_stores”: [ 1, 4 ]}
~]$ tiup ctl:v6.1.0 pd region |jq ‘.regions[] | {region_id: .id, peer_stores: [.peers[].store_id] | select(any(.==4))}’
Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd region{ “region_id”: 252, “peer_stores”: [ 1, 4, 5 ]}{ “region_id”: 216, “peer_stores”: [ 5, 1, 4 ]}……
~]$ tiup ctl:v6.1.0 pd region |jq ‘.regions[] | {region_id: .id, peer_stores: [.peers[].store_id] | select(any(.==(1,4)))}’
~]$ tiup ctl:v6.1.0 pd region | jq ‘.regions[] | select(.leader.store_id == 1) | {region_id: .id, peer_stores: [.peers[].store_id]}’Starting component ctl
: /home/tidb/.tiup/components/ctl/v6.1.0/ctl pd region{ “region_id”: 200, “peer_stores”: [ 5, 4, 1 ]}{ “region_id”: 180, “peer_stores”: [ 5, 4, 1 ]}```
版权声明: 本文为 InfoQ 作者【TiDB 社区干货传送门】的原创文章。
原文链接:【http://xie.infoq.cn/article/008f9d8c5ee6ac1592ffe4a55】。文章转载请联系作者。
评论