混沌工程之 ChaosToolkit K8S 使用之删除 POD 实验

sudo yum install python3 python3-venv

gaolou@GaoMacPro ~ % pip3 install pipenv

pip3 install -U chaostoolkitpip3 install -U chaostoolkit-kubernetespip3 install -U chaostoolkit-reporting

python3 -m venv .bundlersource .bundler/bin/activate

(.bundler) [root@s5 chaostoolkit_scenarios]# chaos discover chaostoolkit-kubernetes[2021-06-23 12:18:07 INFO] Attempting to download and install package 'chaostoolkit-kubernetes'[2021-06-23 12:18:08 INFO] Package downloaded and installed in current environment[2021-06-23 12:18:09 INFO] Discovering capabilities from chaostoolkit-kubernetes[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.actions[2021-06-23 12:18:09 INFO] Searching for probes in chaosk8s.probes[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.deployment.actions[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.deployment.probes[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.node.actions[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.node.probes[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.pod.actions[2021-06-23 12:18:09 INFO] Searching for probes in chaosk8s.pod.probes[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.replicaset.actions[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.service.actions[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.service.probes[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.statefulset.actions[2021-06-23 12:18:09 INFO] Searching for probes in chaosk8s.statefulset.probes[2021-06-23 12:18:09 INFO] Searching for actions in chaosk8s.crd.actions[2021-06-23 12:18:09 INFO] Searching for probes in chaosk8s.crd.probes[2021-06-23 12:18:09 INFO] Discovery outcome saved in ./discovery.json(.bundler) [root@s5 chaostoolkit_scenarios]#

(.bundler) [root@s5 chaostoolkit_scenarios]# chaos initYou are about to create an experiment.This wizard will walk you through each step so that you can buildthe best experiment for your needs.
An experiment is made up of three elements:- a steady-state hypothesis [OPTIONAL]- an experimental method- a set of rollback activities [OPTIONAL]
Only the method is required. Also your experiment willnot run unless you define at least one activity (probe or action)within itExperiment's title: E2 #这里是配置一个试验名
A steady state hypothesis defines what 'normality' looks like in your systemThe steady state hypothesis is a collection of conditions that are used,at the beginning of an experiment, to decide if the system is in a recognised'normal' state. The steady state conditions are then used again when your experiment is complete to detect where your system may have deviated in an interesting,weakness-detecting way
Initially you may not know what your steady state hypothesis isand so instead you might create an experiment without oneThis is why the stead state hypothesis is optional.Do you want to define a steady state hypothesis now? [y/N]: y # 创建稳态假说，请注意，这个是混沌工程中的重要概念，但是在其他的大部分混沌工具中都看不到这一步Hypothesis's title: H2
You may now define probes that will determinethe steady-state of your system.Add an activity1) all_microservices_healthy2) deployment_is_fully_available3) deployment_is_not_fully_available4) microservice_available_and_healthy5) microservice_is_not_available6) read_microservices_logs7) service_endpoint_is_initialized8) count_pods9) pod_is_not_available10) pods_in_conditions11) pods_in_phase12) pods_not_in_phase13) read_pod_logs14) statefulset_fully_available15) statefulset_not_fully_available16) get_cluster_custom_object17) get_custom_object18) list_cluster_custom_objects19) list_custom_objectsActivity (0 to escape): 1 # 选择稳态假说的判断点，简单来说，这里就是创建一个预期结果
!!!DEPRECATED!!!1) kill_microservice2) remove_service_endpointDo you want to use this probe? [y/N]: y # 确定是否使用上面选择的探针
A steady-state probe requires a tolerance value, within whichyour system is in a reognised `normal` state.
What is the tolerance for this probe?: normal
You now need to fill the arguments for this activity. Defaultvalues will be shown between brackets. You may simply press returnto use it or not set any value.Argument's value for 'ns' [default]: chaosnamespace # 输入k8s中要操作的命名空间Do you want to select another activity? [y/N]: y # 是否选择一个的操作动作Add an activity1) all_microservices_healthy2) deployment_is_fully_available3) deployment_is_not_fully_available1) kill_microservice4) microservice_available_and_healthy5) microservice_is_not_available6) read_microservices_logs7) service_endpoint_is_initialized8) count_pods9) pod_is_not_available10) pods_in_conditions11) pods_in_phase12) pods_not_in_phase13) read_pod_logs14) statefulset_fully_available15) statefulset_not_fully_available16) get_cluster_custom_object17) get_custom_object18) list_cluster_custom_objects19) list_custom_objectsActivity (0 to escape): 1 # 选择具体的动作
!!!DEPRECATED!!!Do you want to use this probe? [y/N]: y # 确定使用上面选择的动作
You now need to fill the arguments for this activity. Defaultvalues will be shown between brackets. You may simply press returnto use it or not set any value.Argument's value for 'ns' [default]:Do you want to select another activity? [y/N]: N # 是否要添加另一个试验动作，这里我不再添加了
An experiment's method contains actions and probes. Actionsvary real-world events in your system to determine if yoursteady-state hypothesis is maintained when those events occur.
An experimental method can also contain probes to gather additionalinformation about your system as your method is executed.Do you want to define an experimental method? [y/N]: y # 选择一个试验具体方法
Add an activity
1) kill_microservice
2) remove_service_endpoint
3) scale_microservice
4) start_microservice
5) all_microservices_healthy
6) deployment_is_fully_available
7) deployment_is_not_fully_available
8) microservice_available_and_healthy
9) microservice_is_not_available
10) read_microservices_logs
11) service_endpoint_is_initialized
12) create_deployment
13) delete_deployment
14) scale_deployment
15) deployment_available_and_healthy
16) deployment_fully_available
17) deployment_not_fully_available
18) cordon_node
19) create_node
20) delete_nodes
21) drain_nodes
22) uncordon_node
23) get_nodes
24) delete_pods
25) exec_in_pods
26) terminate_pods
27) count_pods
28) pod_is_not_available
29) pods_in_conditions
30) pods_in_phase
31) pods_not_in_phase
32) read_pod_logs
33) delete_replica_set
34) create_service_endpoint
35) delete_service
36) service_is_initialized
37) create_statefulset
38) remove_statefulset
39) scale_statefulset
40) statefulset_fully_available
41) statefulset_not_fully_available
42) create_cluster_custom_object
43) create_custom_object
44) delete_cluster_custom_object
45) delete_custom_object
46) patch_cluster_custom_object
47) patch_custom_object
48) replace_cluster_custom_object
49) replace_custom_object
50) get_cluster_custom_object
51) get_custom_object
52) list_cluster_custom_objects
53) list_custom_objectsActivity (0 to escape): 24 # 这里我选择第24个方法：删除一个POD
!!!DEPRECATED!!!Do you want to use this action? [y/N]: y # 确认选择
You now need to fill the arguments for this activity. Defaultvalues will be shown between brackets. You may simply press returnto use it or not set any value.
Argument's value for 'name': DeleteRedisPOD # 给这个方法命名Argument's value for 'ns' [default]: chaosnamespace # 确定要操作的k8s命名空间Argument's value for 'label_selector' [name in ({name})]: app=redis # 输入要操作对象的标签，以便可以找到操作对象Do you want to select another activity? [y/N]: N # 是否添加另一个动作，这里我不再添加
An experiment may optionally define a set of remedial actionsthat are used to rollback the system to a given state.
Do you want to add some rollbacks now? [y/N]: N # 是否添加回滚动作，这里我是要删除redis的POD，因为k8s会自动拉起来，所以我不用回滚动作
Experiment created and saved in './experiment.json' # 生成了试验文件
(.bundler) [root@s5 chaostoolkit_scenarios]#

(.bundler) [root@s5 chaostoolkit_scenarios]# chaos run experiment.json[2021-06-28 23:03:23 INFO] Validating the experiment's syntax[2021-06-28 23:03:24 INFO] Experiment looks valid[2021-06-28 23:03:24 INFO] Running experiment: E2[2021-06-28 23:03:24 INFO] Steady-state strategy: default[2021-06-28 23:03:24 INFO] Rollbacks strategy: default[2021-06-28 23:03:24 INFO] Steady state hypothesis: H2[2021-06-28 23:03:24 INFO] Probe: all_microservices_healthy[2021-06-28 23:03:24 WARNING] all_microservices_healthy function is DEPRECATED and will be removed in the next         releases, please use all_pods_healthy instead[2021-06-28 23:03:24 INFO] Steady state hypothesis is met![2021-06-28 23:03:24 INFO] Playing your experiment's method now...[2021-06-28 23:03:24 INFO] Action: delete_pods[2021-06-28 23:03:24 INFO] Steady state hypothesis: H2[2021-06-28 23:03:24 INFO] Probe: all_microservices_healthy[2021-06-28 23:03:24 WARNING] all_microservices_healthy function is DEPRECATED and will be removed in the next         releases, please use all_pods_healthy instead[2021-06-28 23:03:24 INFO] Steady state hypothesis is met![2021-06-28 23:03:24 INFO] Let's rollback...[2021-06-28 23:03:24 INFO] No declared rollbacks, let's move on.[2021-06-28 23:03:24 INFO] Experiment ended with status: completed(.bundler) [root@s5 chaostoolkit_scenarios]#

执行试验前：
[root@s5 ~]# kubectl get pods -n chaosnamespace -o wideNAME                                   READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
...........................redis-master-b96c9795b-nqzmr           1/1     Running   0          3d9h    10.100.220.84    s6     <none>           <none>redis-slave-6b8d456947-6r42k           1/1     Running   0          3d9h    10.100.220.86    s6     <none>           <none>redis-slave-6b8d456947-z55m5           1/1     Running   0          3d9h    10.100.53.206    s7     <none>           <none>

执行试验后：
[root@s5 ~]# kubectl get pods -n chaosnamespace -o wideNAME                                   READY   STATUS              RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
...............................
redis-master-b96c9795b-92rc6           0/1     ContainerCreating   0          3s      <none>           s6     <none>           <none>redis-master-b96c9795b-nqzmr           0/1     Terminating         0          3d9h    10.100.220.84    s6     <none>           <none>redis-slave-6b8d456947-5m2xt           0/1     ContainerCreating   0          2s      <none>           s6     <none>           <none>redis-slave-6b8d456947-6r42k           1/1     Terminating         0          3d9h    10.100.220.86    s6     <none>           <none>redis-slave-6b8d456947-fj4xc           0/1     ContainerCreating   0          3s      <none>           s7     <none>           <none>redis-slave-6b8d456947-z55m5           1/1     Terminating         0          3d9h    10.100.53.206    s7     <none>           <none>

POD完全启动后：
[root@s5 ~]# kubectl get pods -n chaosnamespace -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP               NODE   NOMINATED NODE   READINESS GATES
.......................
redis-master-b96c9795b-92rc6           1/1     Running   0          5m43s   10.100.220.89    s6     <none>           <none>
redis-slave-6b8d456947-5m2xt           1/1     Running   0          5m42s   10.100.220.90    s6     <none>           <none>
redis-slave-6b8d456947-fj4xc           1/1     Running   0          5m43s   10.100.53.211    s7     <none>           <none>
[root@s5 ~]#