通过混沌工程 Chaod 打造更稳定 TiDB 数据库高可用 (三)

[root@k8s-master ~]# ldd --versionldd (GNU libc) 2.17Copyright (C) 2012 Free Software Foundation, Inc.This is free software; see the source for copying conditions.  There is NOwarranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.Written by Roland McGrath and Ulrich Drepper.

1、将要下载的Chaosd版本设置为环境变量，例如：v1.1.1export CHAOSD_VERSION=latest
如果要查看所有已发布的 Chaosd 版本，请参阅 releases(https://github.com/chaos-mesh/chaosd/releases) 如果要下载最新的非稳定版本，则使用 latest：
2、下载Chaosdcurl -fsSL -o chaosd-$CHAOSD_VERSION-linux-amd64.tar.gz https://mirrors.chaos-mesh.org/chaosd-$CHAOSD_VERSION-linux-amd64.tar.gz
3、解压 Chaosd 文件并转移到 /usr/local 目录下：tar zxvf chaosd-$CHAOSD_VERSION-linux-amd64.tar.gz && sudo mv chaosd-$CHAOSD_VERSION-linux-amd64 /usr/local/
4、将 Chaosd 目录加到环境变量 PATH 中：export PATH=/usr/local/chaosd-$CHAOSD_VERSION-linux-amd64:$PATH

1、创建实验[root@k8s-master ~]# chaosd attack process kill --process  sleep -s 15Attack process sleep successfully, uid: 6eaf29c0-dbf0-4612-bbcf-bcab3be1e7062、恢复实验[root@k8s-master ~]# chaosd recover 6eaf29c0-dbf0-4612-bbcf-bcab3be1e706[2022/04/07 11:25:15.995 +08:00] [WARN] [recover.go:75] ["error.core.non_recoverable_attack: only SIGSTOP process attack and process attack with the recover-cmd are supported to recover"] [uid=6eaf29c0-dbf0-4612-bbcf-bcab3be1e706] [kind=process]Recover 6eaf29c0-dbf0-4612-bbcf-bcab3be1e706 successfully

1、启动服务[root@k8s-master ~]# chaosd server --port 123452、创建实验[root@k8s-master ~]# curl -XPOST 127.0.0.1:12345/api/attack/stress -H "Content-Type:application/json" -d '{"action":"cpu", "load": 100, "wo{"status":200,"message":"attack successfully","uid":"86f18a92-7636-451e-9ab0-b3fc0c07a0ef"}3、删除实验[root@k8s-master ~]# curl -X DELETE "127.0.0.1:12345/api/attack/6eaf29c0-dbf0-4612-bbcf-bcab3be1e706"{"status":200,"message":"attack recover successfully","uid":"6eaf29c0-dbf0-4612-bbcf-bcab3be1e706"}

具体文档请参考https://chaos-mesh.org/zh/docs/simulate-process-chaos-in-physical-nodes/1、启动测试进程[root@k8s-master ~]# sleep 10002、创建kill实验[root@k8s-master ~]# chaosd attack process kill --process sleep -s 15Attack process sleep successfully, uid: c820427d-6035-4644-9672-06cfb28d7ef4###第一个创建的窗口[root@k8s-master ~]# sleep 1000Terminated3、恢复实验：[root@k8s-master ~]# chaosd recover c820427d-6035-4644-9672-06cfb28d7ef4[2022/04/07 11:52:40.087 +08:00] [WARN] [recover.go:75] ["error.core.non_recoverable_attack: only SIGSTOP process attack and process attack with the recover-cmd are supported to recover"] [uid=c820427d-6035-4644-9672-06cfb28d7ef4] [kind=process]Recover c820427d-6035-4644-9672-06cfb28d7ef4 successfully
4、创建stop实验，通过指定uid[root@k8s-master ~]# chaosd attack process stop --process sleep --uid 12345Attack process sleep successfully, uid: 12345 ##查看进程[root@k8s-master ~]# ps -aux|grep sleeproot     27457  0.0  0.0 108064   668 pts/2    T    12:02   0:00 sleep 1000root     27918  0.0  0.0 108064   736 pts/2    T    12:03   0:00 sleep 1000root     28622  0.0  0.0 112820  2128 pts/1    S+   12:04   0:00 grep --color=auto sleep####恢复实验[root@k8s-master ~]# chaosd recover 12345Recover 12345 successfully
###杀进程[root@k8s-master ~]# ps -ef|grep sleeproot     30841  4545  0 12:08 pts/2    00:00:00 sleep 1000root     31841  4545  0 12:10 pts/2    00:00:00 grep --color=auto sleep[root@k8s-master ~]# chaosd attack process  kill --process  30841Attack process 30841 successfully, uid: 3c63fcdd-7881-4623-b202-4727b0e1c4f4

1、搜索网络相关的任务chaosd search --kind network --status success
2、丢包演示chaosd attack network loss --correlation 10 --percent 50 -H baidu.com -d eth0或chaosd  attack network loss -d eth0 -i 172.16.5.146 --percent 60
#ping baidu.com[root@k8s-master ~]# ping baidu.comPING baidu.com (220.181.38.251) 56(84) bytes of data.64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=2 ttl=48 time=5.41 ms64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=3 ttl=48 time=6.04 ms64 bytes from 220.181.38.251 (220.181.38.251): 64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=37 ttl=48 time=5.31 ms--- baidu.com ping statistics ---37 packets transmitted, 14 received, 62% packet loss, time 36576msrtt min/avg/max/mdev = 5.271/5.413/6.046/0.207 ms##查看丢失60%左右的包，把实验recoverchaosd recover b5f0491d-96e0-4474-912e-d5b6b307e181
2、延迟##延迟2s sudo chaosd attack network delay -H baidu.com --device eth0 --latency 2s##ping baidu.com[root@k8s-master ~]# ping baidu.comPING baidu.com (220.181.38.251) 56(84) bytes of data.64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=1 ttl=48 time=2005 ms
3、重复包sudo chaosd attack network duplicate -H baidu.com --device eth0 --percent 50##ping baidu.com[root@k8s-master ~]# ping baidu.comPING baidu.com (220.181.38.251) 56(84) bytes of data.64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=4 ttl=48 time=5.35 ms64 bytes from 220.181.38.251 (220.181.38.251): icmp_seq=4 ttl=48 time=5.35 ms (DUP!)
4、损坏包sudo chaosd attack network corrupt  -H baidu.com --device eth0 --percent 100
5、分区sudo chaosd attack network partition -H baidu.com -d eth0 --direction from#direction:指定分区的方向，对来自/发送到 hostname 指定的主机或者 ip 指定的地址的数据包进行分区    
6、DNS chaosd attack network dns -i 1.1.1.1 -d baidu.com##ping baidu.com[root@k8s-master ~]# ping baidu.comPING baidu.com (1.1.1.1) 56(84) bytes of data.64 bytes from baidu.com (1.1.1.1): icmp_seq=1 ttl=48 time=53.5 ms
#备注-H 只影响到指定的域名-i:只影响到指定的IP地址
常见问题：1. correlation 和 jitter 是什么意思？correlation 表示与前一次数据包的相关性jitter 表示延迟时间的变化范围2. 如何对一批 ip 地址生效？设置 ip 为网段，例如：192.168.1.1/243. 恢复实验后网络仍然有问题？查看是否有其他网络实验；根据日志中的命令排查。

Chaosd 支持模拟主机关机故障 。文档：https://chaos-mesh.org/zh/docs/next/simulate-host-console-in-physical-nodes/#慎用chaosd attack host shutdown

1、模拟CPU压力命令chaosd attack stress cpu --load 90 --workers 4或chaosd attack stress cpu -l 100 -w 5通过top进行查看
2、模拟内存chaosd attack stress mem --size 1GB通过free -h查看

文档：https://github.com/chaos-mesh/website-zh/blob/main/docs/simulate-jvm-application-chaos-in-physical-nodes.md Chaosd 通过 Byteman 模拟 JVM 应用故障，主要支持以下几种故障类型：抛出自定义异常触发垃圾回收增加方法延迟修改方法返回值设置 Byteman 配置文件触发故障增加 JVM 压力1、启动测试进程代码：https://github.com/WangXiangUSTC/byteman-example/blob/main/example.helloworld/HelloWorld/Main.java2、下载和启动[root@k8s-master chaos-mesh]# wget https://github.com/WangXiangUSTC/byteman-example/blob/main/example.helloworld/HelloWorld/Main.java1、修改返回值./chaosd attack jvm submit return --class Main --method getnum --port 9288  --value 1232、增加延迟./chaosd attack jvm submit latency  --class Main --method sayhello --latency 50003、抛出异常chaosd attack jvm submit exception -c Main -m sayhello --exception 4、增加压力./chaosd attack jvm submit stress --cpu-count 2

1、写负载chaosd attack disk add-payload write --path /tmp/temp --size 100MB2、读负载chaosd attack disk add-payload read --path /tmp/temp  --size 100MB3、填充磁盘chaosd attack disk fill -p /tmp/123 -s 500MB --fallocate=false

chaosd attack clock -p $PID -t 11s

search) 所支持的配置：[root@k8s-master chaos-mesh]# chaosd search -hSearch chaos attack, you can search attacks through the uid or the state of the attack
Usage:  chaosd search UID [flags]
Flags:  -A, --all             list all chaos attacks      --asc             order by CreateTime, default value is false that means order by CreateTime desc  -h, --help            help for search  -k, --kind string     attack kind, supported value: network, process, stress, disk, host, jvm  -l, --limit uint32    limit the count of attacks  -o, --offset uint32   starting to search attacks from offset  -s, --status string   attack status, supported value: created, success, error, destroyed, revoked
Global Flags:      --log-level string   the log level of chaosd. The value can be 'debug', 'info', 'warn' and 'error'运行以下命令可查看搜索命令 (search) 所支持的配置：
search

1、在dashboard上把改实验停止。2、查看通过这个命令看MY_CRD_NAME[root@k8s-master ~]# kubectl get physicalmachinechaos.chaos-mesh.org -ANAMESPACE   NAME                  ACTION         DURATIONdefault     master                stress-cpu     5mdefault     physical-stress-cpu   stress-cpu     10mdefault     test-pd-01            network-loss   5mdefault     test-pd-03            network-loss   3mdefault     test-pd-04            network-loss   5m
3、强制删除（kubectl patch crd/MY_CRD_NAME -p '{"metadata":{"finalizers":[]}}' --type=merge。crd是physicalmachinechaos.chaos-mesh.org，MY_CRD_NAME通过1查看的name）[root@k8s-master ~]# kubectl patch physicalmachinechaos.chaos-mesh.org/physical-stress-cpu -p '{"metadata":{"finalizers":[]}}' --type=mergephysicalmachinechaos.chaos-mesh.org/physical-stress-cpu patched
kubectl patch physicalmachinechaos.chaos-mesh.org/master -p '{"metadata":{"finalizers":[]}}' --type=merge