[tidb@vm116 ~]$ [tidb@vm116 ~]$ tiup cluster display tidb-prdtiup is checking updates for component cluster ...Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster display tidb-prdCluster type:       tidbCluster name:       tidb-prdCluster version:    v5.4.3Deploy user:        tidbSSH type:           builtinTLS encryption:     enabledCA certificate:     /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crtClient private key: /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pemClient certificate: /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crtDashboard URL:      https://10.2.103.116:32379/dashboardGrafana URL:        http://10.2.103.116:5000ID                  Role          Host          Ports        OS/Arch       Status   Data Dir                            Deploy Dir--                  ----          ----          -----        -------       ------   --------                            ----------10.2.103.116:9793   alertmanager  10.2.103.116  9793/9794    linux/x86_64  Up       /data1/tidb-data/alertmanager-9793  /data1/tidb-deploy/alertmanager-979310.2.103.116:5000   grafana       10.2.103.116  5000         linux/x86_64  Up       -                                   /data1/tidb-deploy/grafana-500010.2.103.116:32379  pd            10.2.103.116  32379/3380   linux/x86_64  Up|L|UI  /data1/tidb-data/pd-32379           /data1/tidb-deploy/pd-3237910.2.103.116:9390   prometheus    10.2.103.116  9390/32020   linux/x86_64  Up       /data1/tidb-data/prometheus-9390    /data1/tidb-deploy/prometheus-939010.2.103.116:43000  tidb          10.2.103.116  43000/20080  linux/x86_64  Up       -                                   /data1/tidb-deploy/tidb-3400010.2.103.116:30160  tikv          10.2.103.116  30160/30180  linux/x86_64  Up       /data1/tidb-data/tikv-30160         /data1/tidb-deploy/tikv-3016010.2.103.116:30162  tikv          10.2.103.116  30162/30182  linux/x86_64  Up       /data1/tidb-data/tikv-30162         /data1/tidb-deploy/tikv-3016210.2.103.116:30163  tikv          10.2.103.116  30163/30183  linux/x86_64  Up       /data1/tidb-data/tikv-30163         /data1/tidb-deploy/tikv-30163Total nodes: 8

MySQL [(none)]> use test;Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -A
Database changedMySQL [test]> select count(*) from t3;+----------+| count(*) |+----------+|  3271488 |+----------+1 row in set (0.00 sec)

[tidb@vm116 ~]$ tiup cluster scale-in tidb-prd -N 10.2.103.116:30160,10.2.103.116:30162 --force tiup is checking updates for component cluster ...Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster scale-in tidb-prd -N 10.2.103.116:30160,10.2.103.116:30162 --force
  ██     ██  █████  ██████  ███    ██ ██ ███    ██  ██████  ██     ██ ██   ██ ██   ██ ████   ██ ██ ████   ██ ██  ██  █  ██ ███████ ██████  ██ ██  ██ ██ ██ ██  ██ ██   ███  ██ ███ ██ ██   ██ ██   ██ ██  ██ ██ ██ ██  ██ ██ ██    ██   ███ ███  ██   ██ ██   ██ ██   ████ ██ ██   ████  ██████
Forcing scale in is unsafe and may result in data loss for stateful components.DO NOT use `--force` if you have any component in Pending Offline status.The process is irreversible and could NOT be cancelled.Only use `--force` when some of the servers are already permanently offline.Are you sure to continue?(Type "Yes, I know my data might be lost." to continue): Yes, I know my data might be lost.This operation will delete the 10.2.103.116:30160,10.2.103.116:30162 nodes in `tidb-prd` and all their data.Do you want to continue? [y/N]:(default=N) yThe component `[tikv]` will become tombstone, maybe exists in several minutes or hours, after that you can use the prune command to clean itDo you want to continue? [y/N]:(default=N) yScale-in nodes...+ [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa.pub+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [ Serial ] - ClusterOperate: operation=DestroyOperation, options={Roles:[] Nodes:[10.2.103.116:30160 10.2.103.116:30162] Force:true SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:tidb SSHProxyIdentity:/home/tidb/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}Stopping component tikv        Stopping instance 10.2.103.116        Stop tikv 10.2.103.116:30160 successDestroying component tikv        Destroying instance 10.2.103.116Destroy 10.2.103.116 success- Destroy tikv paths: [/data1/tidb-data/tikv-30160 /data1/tidb-deploy/tikv-30160/log /data1/tidb-deploy/tikv-30160 /etc/systemd/system/tikv-30160.service]Stopping component tikv        Stopping instance 10.2.103.116        Stop tikv 10.2.103.116:30162 successDestroying component tikv        Destroying instance 10.2.103.116Destroy 10.2.103.116 success- Destroy tikv paths: [/data1/tidb-data/tikv-30162 /data1/tidb-deploy/tikv-30162/log /data1/tidb-deploy/tikv-30162 /etc/systemd/system/tikv-30162.service]+ [ Serial ] - UpdateMeta: cluster=tidb-prd, deleted=`'10.2.103.116:30160','10.2.103.116:30162'`+ [ Serial ] - UpdateTopology: cluster=tidb-prd+ Refresh instance configs  - Generate config pd -> 10.2.103.116:32379 ... Done  - Generate config tikv -> 10.2.103.116:30163 ... Done  - Generate config tidb -> 10.2.103.116:43000 ... Done  - Generate config prometheus -> 10.2.103.116:9390 ... Done  - Generate config grafana -> 10.2.103.116:5000 ... Done  - Generate config alertmanager -> 10.2.103.116:9793 ... Done+ Reload prometheus and grafana  - Reload prometheus -> 10.2.103.116:9390 ... Done  - Reload grafana -> 10.2.103.116:5000 ... DoneScaled cluster `tidb-prd` in successfully[tidb@vm116 ~]$ 

MySQL [test]> select count(*) from t3;ERROR 9005 (HY000): Region is unavailableMySQL [test]> 

[2023/03/24 11:08:45.587 +08:00] [INFO] [<unknown>] ["Subchannel 0x7f4177d8e640: Retry in 1000 milliseconds"][2023/03/24 11:08:45.587 +08:00] [ERROR] [raft_client.rs:504] ["connection aborted"] [addr=10.2.103.116:30160] [receiver_err="Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] }))"] [sink_error="Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] })))"] [store_id=1][2023/03/24 11:08:45.587 +08:00] [ERROR] [raft_client.rs:776] ["connection abort"] [addr=10.2.103.116:30160] [store_id=1][2023/03/24 11:08:46.180 +08:00] [INFO] [raft_client.rs:742] ["resolve store address ok"] [addr=10.2.103.116:30162] [store_id=5002][2023/03/24 11:08:46.180 +08:00] [INFO] [raft_client.rs:627] ["server: new connection with tikv endpoint"] [store_id=5002] [addr=10.2.103.116:30162][2023/03/24 11:08:46.181 +08:00] [INFO] [<unknown>] ["Connect failed: {\"created\":\"@1679627326.181128634\",\"description\":\"Failed to connect to remote host: Connection refused\",\"errno\":111,\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/iomgr/tcp_client_posix.cc\",\"file_line\":200,\"os_error\":\"Connection refused\",\"syscall\":\"connect\",\"target_address\":\"ipv4:10.2.103.116:30162\"}"][2023/03/24 11:08:46.181 +08:00] [INFO] [<unknown>] ["Subchannel 0x7f4177d8ed40: Retry in 1000 milliseconds"][2023/03/24 11:08:46.181 +08:00] [ERROR] [raft_client.rs:504] ["connection aborted"] [addr=10.2.103.116:30162] [receiver_err="Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] }))"] [sink_error="Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] })))"] [store_id=5002][2023/03/24 11:08:46.181 +08:00] [ERROR] [raft_client.rs:776] ["connection abort"] [addr=10.2.103.116:30162] [store_id=5002][2023/03/24 11:08:48.162 +08:00] [WARN] [endpoint.rs:606] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 7001, leader may None\" not_leader { region_id: 7001 }"][2023/03/24 11:08:48.174 +08:00] [WARN] [endpoint.rs:606] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 7001, leader may None\" not_leader { region_id: 7001 }"][2023/03/24 11:08:50.587 +08:00] [INFO] [raft_client.rs:742] ["resolve store address ok"] [addr=10.2.103.116:30160] [store_id=1][2023/03/24 11:08:50.587 +08:00] [INFO] [raft_client.rs:627] ["server: new connection with tikv endpoint"] [store_id=1] [addr=10.2.103.116:30160][2023/03/24 11:08:50.588 +08:00] [INFO] [<unknown>] ["Connect failed: {\"created\":\"@1679627330.588107444\",\"description\":\"Failed to connect to remote host: Connection refused\",\"errno\":111,\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/iomgr/tcp_client_posix.cc\",\"file_line\":200,\"os_error\":\"Connection refused\",\"syscall\":\"connect\",\"target_address\":\"ipv4:10.2.103.116:30160\"}"][2023/03/24 11:08:50.588 +08:00] [INFO] [<unknown>] ["Subchannel 0x7f4177d8f440: Retry in 1000 milliseconds"][2023/03/24 11:08:50.588 +08:00] [ERROR] [raft_client.rs:504] ["connection aborted"] [addr=10.2.103.116:30160] [receiver_err="Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] }))"] [sink_error="Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] })))"] [store_id=1][2023/03/24 11:08:50.588 +08:00] [INFO] [store.rs:2580] ["broadcasting unreachable"] [unreachable_store_id=1] [store_id=5001][2023/03/24 11:08:50.588 +08:00] [ERROR] [raft_client.rs:776] ["connection abort"] [addr=10.2.103.116:30160] [store_id=1][2023/03/24 11:08:51.181 +08:00] [INFO] [raft_client.rs:742] ["resolve store address ok"] [addr=10.2.103.116:30162] [store_id=5002][2023/03/24 11:08:51.181 +08:00] [INFO] [raft_client.rs:627] ["server: new connection with tikv endpoint"] [store_id=5002] [addr=10.2.103.116:30162][2023/03/24 11:08:51.182 +08:00] [INFO] [<unknown>] ["Connect failed: {\"created\":\"@1679627331.182361851\",\"description\":\"Failed to connect to remote host: Connection refused\",\"errno\":111,\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.9.1+1.38.0/grpc/src/core/lib/iomgr/tcp_client_posix.cc\",\"file_line\":200,\"os_error\":\"Connection refused\",\"syscall\":\"connect\",\"target_address\":\"ipv4:10.2.103.116:30162\"}"][2023/03/24 11:08:51.182 +08:00] [INFO] [<unknown>] ["Subchannel 0x7f4177d8fb40: Retry in 999 milliseconds"][2023/03/24 11:08:51.182 +08:00] [ERROR] [raft_client.rs:504] ["connection aborted"] [addr=10.2.103.116:30162] [receiver_err="Some(RpcFailure(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] }))"] [sink_error="Some(RpcFinished(Some(RpcStatus { code: 14-UNAVAILABLE, message: \"failed to connect to all addresses\", details: [] })))"] [store_id=5002][2023/03/24 11:08:51.182 +08:00] [INFO] [store.rs:2580] ["broadcasting unreachable"] [unreachable_store_id=5002] [store_id=5001][2023/03/24 11:08:51.182 +08:00] [ERROR] [raft_client.rs:776] ["connection abort"] [addr=10.2.103.116:30162] [store_id=5002]^C[tidb@vm116 log]$ 

[tidb@vm116 ~]$ tiup ctl:v5.4.3 pd -u "https://10.2.103.116:32379" --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -iStarting component `ctl`: /home/tidb/.tiup/components/ctl/v5.4.3/ctl pd -u https://10.2.103.116:32379 --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -i» config set region-schedule-limit 0Success!» config set replica-schedule-limit 0Success!» config set leader-schedule-limit 0Success!» config set merge-schedule-limit 0Success!» config set hot-region-schedule-limit 0Success!» 

»  region --jq='.regions[] | {id: .id, peer_stores: [.peers[].store_id] | select(length as $total | map(if .==(1,5002) then . else empty end) | length>=$total-length) }'{"id":2003,"peer_stores":[1,5002,5001]}{"id":7001,"peer_stores":[1,5002,5001]}{"id":7005,"peer_stores":[1,5002,5001]}{"id":7009,"peer_stores":[1,5002,5001]}
»  

[tidb@vm116 ~]$ tiup cluster display tidb-prdtiup is checking updates for component cluster ...Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster display tidb-prdCluster type:       tidbCluster name:       tidb-prdCluster version:    v5.4.3Deploy user:        tidbSSH type:           builtinTLS encryption:     enabledCA certificate:     /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crtClient private key: /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pemClient certificate: /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crtDashboard URL:      https://10.2.103.116:32379/dashboardGrafana URL:        http://10.2.103.116:5000ID                  Role          Host          Ports        OS/Arch       Status   Data Dir                            Deploy Dir--                  ----          ----          -----        -------       ------   --------                            ----------10.2.103.116:9793   alertmanager  10.2.103.116  9793/9794    linux/x86_64  Up       /data1/tidb-data/alertmanager-9793  /data1/tidb-deploy/alertmanager-979310.2.103.116:5000   grafana       10.2.103.116  5000         linux/x86_64  Up       -                                   /data1/tidb-deploy/grafana-500010.2.103.116:32379  pd            10.2.103.116  32379/3380   linux/x86_64  Up|L|UI  /data1/tidb-data/pd-32379           /data1/tidb-deploy/pd-3237910.2.103.116:9390   prometheus    10.2.103.116  9390/32020   linux/x86_64  Up       /data1/tidb-data/prometheus-9390    /data1/tidb-deploy/prometheus-939010.2.103.116:43000  tidb          10.2.103.116  43000/20080  linux/x86_64  Up       -                                   /data1/tidb-deploy/tidb-3400010.2.103.116:30163  tikv          10.2.103.116  30163/30183  linux/x86_64  Up       /data1/tidb-data/tikv-30163         /data1/tidb-deploy/tikv-30163Total nodes: 6[tidb@vm116 ~]$ tiup cluster stop tidb-prd -N 10.2.103.116:30163tiup is checking updates for component cluster ...Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster stop tidb-prd -N 10.2.103.116:30163Will stop the cluster tidb-prd with nodes: 10.2.103.116:30163, roles: .Do you want to continue? [y/N]:(default=N) y+ [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa.pub+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [ Serial ] - StopClusterStopping component tikv        Stopping instance 10.2.103.116        Stop tikv 10.2.103.116:30163 successStopping component node_exporterStopping component blackbox_exporterStopped cluster `tidb-prd` successfully

[tidb@vm116 v5.4.3]$  ./tikv-ctl  --data-dir  /data1/tidb-data/tikv-30163  unsafe-recover remove-fail-stores -s 1,5002 --all-regions[2023/03/24 11:25:47.978 +08:00] [WARN] [config.rs:612] ["compaction guard is disabled due to region info provider not available"][2023/03/24 11:25:47.978 +08:00] [WARN] [config.rs:715] ["compaction guard is disabled due to region info provider not available"]removing stores [1, 5002] from configurations...success

[tidb@vm116 v5.4.3]$ tiup cluster start  tidb-prd -N 10.2.103.116:30163tiup is checking updates for component cluster ...Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster start tidb-prd -N 10.2.103.116:30163Starting cluster tidb-prd...+ [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa.pub+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [ Serial ] - StartClusterStarting component tikv        Starting instance 10.2.103.116:30163        Start instance 10.2.103.116:30163 successStarting component node_exporter        Starting instance 10.2.103.116        Start 10.2.103.116 successStarting component blackbox_exporter        Starting instance 10.2.103.116        Start 10.2.103.116 success+ [ Serial ] - UpdateTopology: cluster=tidb-prdStarted cluster `tidb-prd` successfully

[tidb@vm116 v5.4.3]$ tiup ctl:v5.4.3 pd -u "https://10.2.103.116:32379" --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -iStarting component `ctl`: /home/tidb/.tiup/components/ctl/v5.4.3/ctl pd -u https://10.2.103.116:32379 --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -i» region --jq '.regions[]|select(has("leader")|not)|{id: .id, peer_stores: [.peers[].store_id]}'
»

MySQL [test]> select count(*) from t3;+----------+| count(*) |+----------+|  3271488 |+----------+1 row in set (0.55 sec)
MySQL [test]> MySQL [test]> admin check table t3;Query OK, 0 rows affected (0.00 sec)
MySQL [test]>

[tidb@vm116 ~]$ tiup ctl:v5.4.3 pd -u "https://10.2.103.116:32379" --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -iStarting component `ctl`: /home/tidb/.tiup/components/ctl/v5.4.3/ctl pd -u https://10.2.103.116:32379 --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -i» config set region-schedule-limit 2000Success!» config set replica-schedule-limit 32Success!» config set leader-schedule-limit 8Success!» config set merge-schedule-limit 16Success!» config set hot-region-schedule-limit 2Success!» 

./tikv-ctl  --data-dir  /data3/tidb/data unsafe-recover remove-fail-stores -s 1 -r 50377[INFO] [mod.rs:118] ["encryption: none of key dictionary and file dictionary are found."][INFO] [mod.rs:479] ["encryption is disabled."][WARN] [config.rs:587] ["compaction guard is disabled due to region info provider not available"][WARN] [config.rs:682] ["compaction guard is disabled due to region info provider not available"]removing stores [1] from configurations...Debugger::remove_fail_stores: "No such region 50377 on the store"

./tikv-ctl  --ca-path  /data3/tidb/deploy/tls/ca.crt  --key-path  /data3/tidb/deploy/tls/tikv.pem --cert-path /data3/tidb/deploy/tls/tikv.crt  --data-dir  /data3/tidb/data recreate-region -p https://10.2.103.116:32379  -r 50377[INFO] [mod.rs:118] ["encryption: none of key dictionary and file dictionary are found."][INFO] [mod.rs:479] ["encryption is disabled."]initing empty region  with peer_id ...success

[tidb@vm116 ~]$ tiup cluster display tidb-prd tiup is checking updates for component cluster ...Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster display tidb-prdCluster type:       tidbCluster name:       tidb-prdCluster version:    v6.1.5Deploy user:        tidbSSH type:           builtinTLS encryption:     enabledCA certificate:     /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crtClient private key: /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pemClient certificate: /home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crtDashboard URL:      https://10.2.103.116:32379/dashboardGrafana URL:        http://10.2.103.116:5000ID                  Role          Host          Ports        OS/Arch       Status   Data Dir                            Deploy Dir--                  ----          ----          -----        -------       ------   --------                            ----------10.2.103.116:9793   alertmanager  10.2.103.116  9793/9794    linux/x86_64  Up       /data1/tidb-data/alertmanager-9793  /data1/tidb-deploy/alertmanager-979310.2.103.116:5000   grafana       10.2.103.116  5000         linux/x86_64  Up       -                                   /data1/tidb-deploy/grafana-500010.2.103.116:32379  pd            10.2.103.116  32379/3380   linux/x86_64  Up|L|UI  /data1/tidb-data/pd-32379           /data1/tidb-deploy/pd-3237910.2.103.116:9390   prometheus    10.2.103.116  9390/32020   linux/x86_64  Up       /data1/tidb-data/prometheus-9390    /data1/tidb-deploy/prometheus-939010.2.103.116:43000  tidb          10.2.103.116  43000/20080  linux/x86_64  Up       -                                   /data1/tidb-deploy/tidb-3400010.2.103.116:30160  tikv          10.2.103.116  30160/30180  linux/x86_64  Up       /data1/tidb-data/tikv-30160         /data1/tidb-deploy/tikv-3016010.2.103.116:30162  tikv          10.2.103.116  30162/30182  linux/x86_64  Up       /data1/tidb-data/tikv-30162         /data1/tidb-deploy/tikv-3016210.2.103.116:30163  tikv          10.2.103.116  30163/30183  linux/x86_64  Up       /data1/tidb-data/tikv-30163         /data1/tidb-deploy/tikv-30163Total nodes: 8

[tidb@vm116 ~]$ tiup cluster scale-in tidb-prd -N 10.2.103.116:30160,10.2.103.116:30162 --force tiup is checking updates for component cluster ...Starting component `cluster`: /home/tidb/.tiup/components/cluster/v1.11.3/tiup-cluster scale-in tidb-prd -N 10.2.103.116:30160,10.2.103.116:30162 --force
  ██     ██  █████  ██████  ███    ██ ██ ███    ██  ██████  ██     ██ ██   ██ ██   ██ ████   ██ ██ ████   ██ ██  ██  █  ██ ███████ ██████  ██ ██  ██ ██ ██ ██  ██ ██   ███  ██ ███ ██ ██   ██ ██   ██ ██  ██ ██ ██ ██  ██ ██ ██    ██   ███ ███  ██   ██ ██   ██ ██   ████ ██ ██   ████  ██████
Forcing scale in is unsafe and may result in data loss for stateful components.DO NOT use `--force` if you have any component in Pending Offline status.The process is irreversible and could NOT be cancelled.Only use `--force` when some of the servers are already permanently offline.Are you sure to continue?(Type "Yes, I know my data might be lost." to continue): Yes, I know my data might be lost.This operation will delete the 10.2.103.116:30160,10.2.103.116:30162 nodes in `tidb-prd` and all their data.Do you want to continue? [y/N]:(default=N) yThe component `[tikv]` will become tombstone, maybe exists in several minutes or hours, after that you can use the prune command to clean itDo you want to continue? [y/N]:(default=N) yScale-in nodes...+ [ Serial ] - SSHKeySet: privateKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa, publicKey=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/ssh/id_rsa.pub+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [Parallel] - UserSSH: user=tidb, host=10.2.103.116+ [ Serial ] - ClusterOperate: operation=DestroyOperation, options={Roles:[] Nodes:[10.2.103.116:30160 10.2.103.116:30162] Force:true SSHTimeout:5 OptTimeout:120 APITimeout:600 IgnoreConfigCheck:false NativeSSH:false SSHType: Concurrency:5 SSHProxyHost: SSHProxyPort:22 SSHProxyUser:tidb SSHProxyIdentity:/home/tidb/.ssh/id_rsa SSHProxyUsePassword:false SSHProxyTimeout:5 CleanupData:false CleanupLog:false CleanupAuditLog:false RetainDataRoles:[] RetainDataNodes:[] DisplayMode:default Operation:StartOperation}failed to delete tikv: error requesting https://10.2.103.116:32379/pd/api/v1/store/7014, response: "[PD:core:ErrStoresNotEnough]can not remove store 7014 since the number of up stores would be 2 while need 3", code 400Stopping component tikv        Stopping instance 10.2.103.116        Stop tikv 10.2.103.116:30162 successDestroying component tikv        Destroying instance 10.2.103.116Destroy 10.2.103.116 success- Destroy tikv paths: [/data1/tidb-data/tikv-30162 /data1/tidb-deploy/tikv-30162/log /data1/tidb-deploy/tikv-30162 /etc/systemd/system/tikv-30162.service]failed to delete tikv: error requesting https://10.2.103.116:32379/pd/api/v1/store/7013, response: "[PD:core:ErrStoresNotEnough]can not remove store 7013 since the number of up stores would be 2 while need 3", code 400Stopping component tikv        Stopping instance 10.2.103.116        Stop tikv 10.2.103.116:30160 successDestroying component tikv        Destroying instance 10.2.103.116Destroy 10.2.103.116 success- Destroy tikv paths: [/etc/systemd/system/tikv-30160.service /data1/tidb-data/tikv-30160 /data1/tidb-deploy/tikv-30160/log /data1/tidb-deploy/tikv-30160]+ [ Serial ] - UpdateMeta: cluster=tidb-prd, deleted=`'10.2.103.116:30160','10.2.103.116:30162'`+ [ Serial ] - UpdateTopology: cluster=tidb-prd+ Refresh instance configs  - Generate config pd -> 10.2.103.116:32379 ... Done  - Generate config tikv -> 10.2.103.116:30163 ... Done  - Generate config tidb -> 10.2.103.116:43000 ... Done  - Generate config prometheus -> 10.2.103.116:9390 ... Done  - Generate config grafana -> 10.2.103.116:5000 ... Done  - Generate config alertmanager -> 10.2.103.116:9793 ... Done+ Reload prometheus and grafana  - Reload prometheus -> 10.2.103.116:9390 ... Done  - Reload grafana -> 10.2.103.116:5000 ... DoneScaled cluster `tidb-prd` in successfully

[tidb@vm116 ~]$ tiup ctl:v5.4.3 pd -u "https://10.2.103.116:32379" --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -iStarting component `ctl`: /home/tidb/.tiup/components/ctl/v5.4.3/ctl pd -u https://10.2.103.116:32379 --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -i» store{  "count": 3,  "stores": [    {      "store": {        "id": 7014,        "address": "10.2.103.116:30162",        "version": "6.1.5",        "peer_address": "10.2.103.116:30162",        "status_address": "10.2.103.116:30182",        "git_hash": "e554126f6e83a6ddc944ddc51746b6def303ec1a",        "start_timestamp": 1679629145,        "deploy_path": "/data1/tidb-deploy/tikv-30162/bin",        "last_heartbeat": 1679629565041246521,        "node_state": 1,        "state_name": "Disconnected"      },      "status": {        "capacity": "492GiB",        "available": "424GiB",        "used_size": "324.6MiB",        "leader_count": 2,        "leader_weight": 1,        "leader_score": 2,        "leader_size": 178,        "region_count": 7,        "region_weight": 1,        "region_score": 410.17607715843184,        "region_size": 251,        "slow_score": 1,        "start_ts": "2023-03-24T11:39:05+08:00",        "last_heartbeat_ts": "2023-03-24T11:46:05.041246521+08:00",        "uptime": "7m0.041246521s"      }    },    {      "store": {        "id": 5001,        "address": "10.2.103.116:30163",        "version": "6.1.5",        "peer_address": "10.2.103.116:30163",        "status_address": "10.2.103.116:30183",        "git_hash": "e554126f6e83a6ddc944ddc51746b6def303ec1a",        "start_timestamp": 1679629139,        "deploy_path": "/data1/tidb-deploy/tikv-30163/bin",        "last_heartbeat": 1679629599338071366,        "node_state": 1,        "state_name": "Up"      },      "status": {        "capacity": "492GiB",        "available": "435GiB",        "used_size": "342MiB",        "leader_count": 5,        "leader_weight": 1,        "leader_score": 5,        "leader_size": 73,        "region_count": 7,        "region_weight": 1,        "region_score": 409.8657305188558,        "region_size": 251,        "slow_score": 1,        "start_ts": "2023-03-24T11:38:59+08:00",        "last_heartbeat_ts": "2023-03-24T11:46:39.338071366+08:00",        "uptime": "7m40.338071366s"      }    },    {      "store": {        "id": 7013,        "address": "10.2.103.116:30160",        "version": "6.1.5",        "peer_address": "10.2.103.116:30160",        "status_address": "10.2.103.116:30180",        "git_hash": "e554126f6e83a6ddc944ddc51746b6def303ec1a",        "start_timestamp": 1679629155,        "deploy_path": "/data1/tidb-deploy/tikv-30160/bin",        "last_heartbeat": 1679629565148211763,        "node_state": 1,        "state_name": "Disconnected"      },      "status": {        "capacity": "492GiB",        "available": "424GiB",        "used_size": "324.6MiB",        "leader_count": 0,        "leader_weight": 1,        "leader_score": 0,        "leader_size": 0,        "region_count": 7,        "region_weight": 1,        "region_score": 410.17510527918483,        "region_size": 251,        "slow_score": 1,        "start_ts": "2023-03-24T11:39:15+08:00",        "last_heartbeat_ts": "2023-03-24T11:46:05.148211763+08:00",        "uptime": "6m50.148211763s"      }    }  ]}

MySQL [test]> select count(*) from t3;ERROR 9002 (HY000): TiKV server timeoutMySQL [test]>

[2023/03/24 11:48:47.549 +08:00] [ERROR] [raft_client.rs:824] ["connection abort"] [addr=10.2.103.116:30160] [store_id=7013][2023/03/24 11:48:47.813 +08:00] [INFO] [raft.rs:1550] ["starting a new election"] [term=9] [raft_id=7004] [region_id=7001][2023/03/24 11:48:47.813 +08:00] [INFO] [raft.rs:1170] ["became pre-candidate at term 9"] [term=9] [raft_id=7004] [region_id=7001][2023/03/24 11:48:47.813 +08:00] [INFO] [raft.rs:1299] ["broadcasting vote request"] [to="[250648, 250650]"] [log_index=1863] [log_term=9] [term=9] [type=MsgRequestPreVote] [raft_id=7004] [region_id=7001][2023/03/24 11:48:47.927 +08:00] [INFO] [<unknown>] ["subchannel 0x7fae72e89c00 {address=ipv4:10.2.103.116:30162, args=grpc.client_channel_factory=0x7faea78591a8, grpc.default_authority=10.2.103.116:30162, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x7fae00a22260, grpc.internal.security_connector=0x7fae72e0ad80, grpc.internal.subchannel_pool=0x7faea7832d60, grpc.primary_user_agent=grpc-rust/0.10.2, grpc.resource_quota=0x7faea799c630, grpc.server_uri=dns:///10.2.103.116:30162, random id=324}: connect failed: {\"created\":\"@1679629727.927814392\",\"description\":\"Failed to connect to remote host: Connection refused\",\"errno\":111,\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.10.1+1.44.0/grpc/src/core/lib/iomgr/tcp_client_posix.cc\",\"file_line\":200,\"os_error\":\"Connection refused\",\"syscall\":\"connect\",\"target_address\":\"ipv4:10.2.103.116:30162\"}"][2023/03/24 11:48:47.928 +08:00] [INFO] [<unknown>] ["subchannel 0x7fae72e89c00 {address=ipv4:10.2.103.116:30162, args=grpc.client_channel_factory=0x7faea78591a8, grpc.default_authority=10.2.103.116:30162, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x7fae00a22260, grpc.internal.security_connector=0x7fae72e0ad80, grpc.internal.subchannel_pool=0x7faea7832d60, grpc.primary_user_agent=grpc-rust/0.10.2, grpc.resource_quota=0x7faea799c630, grpc.server_uri=dns:///10.2.103.116:30162, random id=324}: Retry in 999 milliseconds"][2023/03/24 11:48:47.928 +08:00] [INFO] [advance.rs:296] ["check leader failed"] [to_store=7014] [error="\"[rpc failed] RpcFailure: 14-UNAVAILABLE failed to connect to all addresses\""][2023/03/24 11:48:47.928 +08:00] [INFO] [<unknown>] ["subchannel 0x7fae72eb5c00 {address=ipv4:10.2.103.116:30160, args=grpc.client_channel_factory=0x7faea78591a8, grpc.default_authority=10.2.103.116:30160, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x7fae00a22ee0, grpc.internal.security_connector=0x7fae72fbd640, grpc.internal.subchannel_pool=0x7faea7832d60, grpc.primary_user_agent=grpc-rust/0.10.2, grpc.resource_quota=0x7faea799c630, grpc.server_uri=dns:///10.2.103.116:30160, random id=325}: connect failed: {\"created\":\"@1679629727.928184608\",\"description\":\"Failed to connect to remote host: Connection refused\",\"errno\":111,\"file\":\"/rust/registry/src/github.com-1ecc6299db9ec823/grpcio-sys-0.10.1+1.44.0/grpc/src/core/lib/iomgr/tcp_client_posix.cc\",\"file_line\":200,\"os_error\":\"Connection refused\",\"syscall\":\"connect\",\"target_address\":\"ipv4:10.2.103.116:30160\"}"][2023/03/24 11:48:47.928 +08:00] [INFO] [<unknown>] ["subchannel 0x7fae72eb5c00 {address=ipv4:10.2.103.116:30160, args=grpc.client_channel_factory=0x7faea78591a8, grpc.default_authority=10.2.103.116:30160, grpc.http2_scheme=https, grpc.internal.channel_credentials=0x7fae00a22ee0, grpc.internal.security_connector=0x7fae72fbd640, grpc.internal.subchannel_pool=0x7faea7832d60, grpc.primary_user_agent=grpc-rust/0.10.2, grpc.resource_quota=0x7faea799c630, grpc.server_uri=dns:///10.2.103.116:30160, random id=325}: Retry in 1000 milliseconds"][2023/03/24 11:48:47.928 +08:00] [INFO] [advance.rs:296] ["check leader failed"] [to_store=7013] [error="\"[rpc failed] RpcFailure: 14-UNAVAILABLE failed to connect to all addresses\""][2023/03/24 11:48:48.037 +08:00] [WARN] [endpoint.rs:621] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 7009, leader may None\" not_leader { region_id: 7009 }"][2023/03/24 11:48:48.089 +08:00] [WARN] [endpoint.rs:621] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 7001, leader may None\" not_leader { region_id: 7001 }"][2023/03/24 11:48:48.409 +08:00] [WARN] [endpoint.rs:621] [error-response] [err="Region error (will back off and retry) message: \"peer is not leader for region 252001, leader may None\" not_leader { region_id: 252001 }"]^C[tidb@vm116 log]$

[tidb@vm116 ctl]$ tiup ctl:v6.1.5  pd -u "https://10.2.103.116:32379" --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -iStarting component `ctl`: /home/tidb/.tiup/components/ctl/v6.1.5/ctl pd -u https://10.2.103.116:32379 --cacert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/ca.crt --key=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.pem --cert=/home/tidb/.tiup/storage/cluster/clusters/tidb-prd/tls/client.crt -i» unsafe remove-failed-stores 7013,7014Success!

» unsafe remove-failed-stores show[  {    "info": "Unsafe recovery enters collect report stage: failed stores 7013, 7014",    "time": "2023-03-24 11:56:44.910"  },  {    "info": "Unsafe recovery enters force leader stage",    "time": "2023-03-24 11:56:49.390",    "actions": {      "store 5001": [        "force leader on regions: 7001, 7005, 7009, 252001, 252005, 252009, 2003"      ]    }  },  {    "info": "Collecting reports from alive stores(0/1)",    "time": "2023-03-24 11:57:02.286",    "details": [      "Stores that have not dispatched plan: ",      "Stores that have reported to PD: ",      "Stores that have not reported to PD: 5001"    ]  }]

» unsafe remove-failed-stores show[  {    "info": "Unsafe recovery enters collect report stage: failed stores 7013, 7014",    "time": "2023-03-24 11:56:44.910"  },  {    "info": "Unsafe recovery enters force leader stage",    "time": "2023-03-24 11:56:49.390",    "actions": {      "store 5001": [        "force leader on regions: 7001, 7005, 7009, 252001, 252005, 252009, 2003"      ]    }  },  {    "info": "Unsafe recovery enters demote failed voter stage",    "time": "2023-03-24 11:57:20.434",    "actions": {      "store 5001": [        "region 7001 demotes peers { id:250648 store_id:7014 }, { id:250650 store_id:7013 }",        "region 7005 demotes peers { id:250647 store_id:7013 }, { id:250649 store_id:7014 }",        "region 7009 demotes peers { id:250644 store_id:7014 }, { id:250646 store_id:7013 }",        "region 252001 demotes peers { id:252003 store_id:7013 }, { id:252004 store_id:7014 }",        "region 252005 demotes peers { id:252007 store_id:7013 }, { id:252008 store_id:7014 }",        "region 252009 demotes peers { id:252011 store_id:7013 }, { id:252012 store_id:7014 }",        "region 2003 demotes peers { id:250643 store_id:7013 }, { id:250645 store_id:7014 }"      ]    }  },  {    "info": "Unsafe recovery finished",    "time": "2023-03-24 11:57:22.443",    "details": [      "affected table ids: 73, 77, 68, 70"    ]  }]

MySQL [test]> select count(*) from t3;+----------+| count(*) |+----------+|  3271488 |+----------+1 row in set (0.50 sec)
MySQL [test]> MySQL [test]> admin check table t3;Query OK, 0 rows affected (0.00 sec)
MySQL [test]>