写点什么

kubernetes 集群安装(二进制)

用户头像
小小文
关注
发布于: 2020 年 07 月 10 日

一、环境准备

1、服务器准备

总共 5 台机器,3 台 master 2 台 node 节点

系统:CentOS Linux release 7.5 内存:8G 磁盘:50G

最小化安装

10.103.22.231 master01 haproxy keepalived

10.103.22.232 master02 haproxy keepalived

10.103.22.233 master03 haproxy keepalived

10.103.22.234 node04

10.103.22.235 node05

2、系统设置

设置主机名

hostnamectl set-hostname <your_hostname>
复制代码

修改 hosts 文件

cat >> /etc/hosts <<EOF10.103.22.231 master0110.103.22.232 master0210.103.22.233 master0310.103.22.234 node0410.103.22.235 node05EOF
复制代码

安装依赖包

yum install -y conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim yum-utils device-mapper-persistent-data lvm2 net-tools
复制代码

设置防火墙为 Iptables 并设置空规则

systemctl stop firewalld && systemctl disable firewalldyum -y install iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save
复制代码

关闭 swap

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
复制代码

关闭 SELINUX

setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
复制代码

调整内核参数,对于 K8S

cat > /etc/sysctl.d/kubernetes.conf <<EOFnet.bridge.bridge-nf-call-iptables=1net.bridge.bridge-nf-call-ip6tables=1net.ipv4.ip_forward=1net.ipv4.tcp_tw_recycle=0vm.swappiness=0 # 禁止使用 swap 空间,只有当系统 OOM 时才允许使用它vm.overcommit_memory=1 # 不检查物理内存是否够用vm.panic_on_oom=0 # 开启 OOMfs.inotify.max_user_instances=8192fs.inotify.max_user_watches=1048576fs.file-max=52706963fs.nr_open=52706963net.ipv6.conf.all.disable_ipv6=1net.netfilter.nf_conntrack_max=2310720EOFsysctl -p /etc/sysctl.d/kubernetes.conf
复制代码

加载 ipvs 模块

vi /etc/sysconfig/modules/ipvs.modules#!/bin/shipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs/"for mod in `ls ${ipvs_mods_dir} | grep -o "^[^.]*"`; do    /sbin/modprobe $moddone
chmod +x /etc/sysconfig/modules/ipvs.modules && sh /etc/sysconfig/modules/ipvs.modules
#检查模块是否加载生效,如果如下所示表示已经加载ipvs模版到内核了。lsmod | grep ip_vsip_vs_wrr 12697 0 ip_vs_wlc 12519 0 ip_vs_sh 12688 0 ip_vs_sed 12519 0 ip_vs_rr 12600 20 ip_vs_pe_sip 12740 0 nf_conntrack_sip 33860 1 ip_vs_pe_sipip_vs_nq 12516 0 ip_vs_lc 12516 0 ip_vs_lblcr 12922 0 ip_vs_lblc 12819 0 ip_vs_ftp 13079 0 nf_nat 26787 4 ip_vs_ftp,nf_nat_ipv4,nf_nat_ipv6,nf_nat_masquerade_ipv4ip_vs_dh 12688 0 ip_vs 141432 44 ip_vs_dh,ip_vs_lc,ip_vs_nq,ip_vs_rr,ip_vs_sh,ip_vs_ftp,ip_vs_sed,ip_vs_wlc,ip_vs_wrr,ip_vs_pe_sip,ip_vs_lblcr,ip_vs_lblcnf_conntrack 133053 10 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_sip,nf_conntrack_ipv4,nf_conntrack_ipv6libcrc32c 12644 4 xfs,ip_vs,nf_nat,nf_conntrack
复制代码

请确保 ipset 也已经安装了,如未安装请执行 yum install -y ipset 安装。


调整系统时区

# 设置系统时区为 中国/上海timedatectl set-timezone Asia/Shanghai# 将当前的 UTC 时间写入硬件时钟timedatectl set-local-rtc 0# 重启依赖于系统时间的服务systemctl restart rsyslogsystemctl restart crond
复制代码

关闭系统不需要服务

systemctl stop postfix && systemctl disable postfix
复制代码

将可执行文件路径 /opt/kubernetes/bin 添加到 PATH 变量

echo 'PATH=/opt/kubernetes/bin:$PATH' >> /etc/profile.d/kubernetes.shsource /etc/profile.d/kubernetes.shmkdir -p /opt/kubernetes/{bin,cert,script}
复制代码

3、准备 Docker 环境

清理原有版本

yum remove docker \    docker-client \    docker-client-latest \    docker-common \    docker-latest \    docker-latest-logrotate \    docker-logrotate \    docker-engine
复制代码

添加 Docker yum 源

yum-config-manager \    --add-repo \    https://download.docker.com/linux/centos/docker-ce.repo
复制代码

安装 docker

目前根据 kubernetes 容器运行时文档,安装如下版本:

yum install -y \  containerd.io-1.2.10 \  docker-ce-19.03.4 \  docker-ce-cli-19.03.4
复制代码

docker-ce:docker 服务器

docker-ce-cli:docker 客户端

containerd.io:用于管理主机系统的完整容器生命周期,从映像传输和存储到容器执行和监视,再到底层存储、网络附件等等


配置 docker daemon.json 文件

mkdir /etc/dockercat > /etc/docker/daemon.json <<EOF{    "registry-mirrors": ["https://zvakwn5b.mirror.aliyuncs.com"],	  "data-root": "/data/docker",    "exec-opts": ["native.cgroupdriver=systemd"],    "log-driver": "json-file",    "log-opts": {    "max-size": "100m"    },    "storage-driver": "overlay2",    "storage-opts": [    "overlay2.override_kernel_check=true"    ]}EOF
复制代码

创建 docker 数据目录

NODE_IPS=("master01" "master02" "master03" "node04" "node05")for node_ip in ${NODE_IPS[@]};do        ssh root@${node_ip} "mkdir -p /data/docker/"done
复制代码

启动 Docker

systemctl enable dockersystemctl daemon-reloadsystemctl start docker
复制代码

二、安装 kubernetes 集群

1、配置 SSH 免密登录

新密钥对

ssh-keygen -t rsa
复制代码

分发公钥到各个节点

# 把id_rsa.pub文件内容copy到其他机器的授权文件中$ cat ~/.ssh/id_rsa.pub# 在其他节点执行下面命令(包括worker节点)mkdir -p ~/.ssh/$ echo "<file_content>" >> ~/.ssh/authorized_keys
复制代码

2、下载二进制包

kubernetes 包下载:

地址:https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.17.md#downloads-for-v1178

cd /root/softwarewget https://dl.k8s.io/v1.17.8/kubernetes-server-linux-amd64.tar.gz
复制代码

etcd 包下载:

地址:https://github.com/etcd-io/etcd/releases/tag/v3.4.9

cd /root/softwarewget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz
复制代码

包名:

kubernetes-server-linux-amd64.tar.gz

etcd-v3.4.9-linux-amd64.tar.gz

解压安装包:

cd /root/softwaremkdir -p mastermkdir -p workertar zxvf kubernetes-server-linux-amd64.tar.gztar zxvf etcd-v3.4.9-linux-amd64.tar.gzcp kubernetes/server/bin/{kubeadm,kube-apiserver,kube-controller-manager,kube-scheduler,kubectl} master/cp kubernetes/server/bin/{kubelet,kube-proxy} worker/cp etcd-v3.4.9-linux-amd64/{etcd,etcdctl} master/
复制代码

3、创建 CA 证书和秘钥

安装 cfssl 工具集

mkdir -p /opt/kubernetes/{bin,cert} &&cd /opt/kuberneteswget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64mv cfssl_linux-amd64 /opt/kubernetes/bin/cfsslwget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64mv cfssljson_linux-amd64 /opt/kubernetes/bin/cfssljsonwget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64mv cfssl-certinfo_linux-amd64 /opt/kubernetes/bin/cfssl-certinfochmod +x /opt/kubernetes/bin/*
复制代码

创建配置文件

cat > /opt/kubernetes/cert/ca-config.json <<EOF{    "signing": {        "default": {            "expiry": "87600h"        },        "profiles": {            "kubernetes": {                "usages": [                    "signing",                    "key encipherment",                    "server auth",                    "client auth"                ],                "expiry": "87600h"            }        }    }}EOF
复制代码

创建证书签名请求文件

cat > /opt/kubernetes/cert/ca-csr.json <<EOF{    "CN": "kubernetes",    "key": {        "algo": "rsa",        "size": 2048    },    "names": [        {            "C": "CN",            "ST": "ShangHai",            "L": "ShangHai",            "O": "ops",            "OU": "zhonggang"        }    ]}EOF
复制代码

生成 CA 证书和私钥

cd /opt/kubernetes/certcfssl gencert -initca ca-csr.json | cfssljson -bare ca
复制代码

分发根证书文件

NODE_IPS=("master01" "master02" "master03" "node04" "node05")for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}"    ssh root@${node_ip} "mkdir -p /opt/kubernetes/cert"    scp /opt/kubernetes/cert/ca*.pem /opt/kubernetes/cert/ca-config.json root@${node_ip}:/opt/kubernetes/certdone
复制代码

4、部署 etcd 集群

创建 etcd 证书和私钥

创建证书签名请求文件

mkdir -p /opt/etcd/certcat > /opt/etcd/cert/etcd-csr.json <<EOF{    "CN": "etcd",    "hosts": [        "127.0.0.1",        "10.103.22.231",        "10.103.22.232",        "10.103.22.233"    ],    "key": {        "algo": "rsa",        "size": 2048    },    "names": [        {            "C": "CN",            "ST": "ShangHai",            "L": "ShangHai",            "O": "ops",            "OU": "zhonggang"        }    ]}EOF
复制代码

生成证书和私钥

cd /opt/etcd/certcfssl gencert -ca=/opt/kubernetes/cert/ca.pem \-ca-key=/opt/kubernetes/cert/ca-key.pem \-config=/opt/kubernetes/cert/ca-config.json \-profile=kubernetes etcd-csr.json | cfssljson -bare etcd
复制代码

分发生成的证书和私钥到各 etcd 节点

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        scp /root/software/master/etcd* root@${node_ip}:/opt/kubernetes/bin        ssh root@${node_ip} "chmod +x /opt/kubernetes/bin/*"        ssh root@${node_ip} "mkdir -p /opt/etcd/cert"        scp /opt/etcd/cert/etcd*.pem root@${node_ip}:/opt/etcd/cert/done
复制代码

创建 etcd 的 systemd unit 模板

cat > /opt/etcd/etcd.service.template <<EOF[Unit]Description=Etcd ServerAfter=network.targetAfter=network-online.targetWants=network-online.targetDocumentation=etcd start[Service]User=rootType=notifyWorkingDirectory=/opt/lib/etcd/ExecStart=/opt/kubernetes/bin/etcd \--data-dir=/opt/lib/etcd \--name ##NODE_NAME## \--cert-file=/opt/etcd/cert/etcd.pem \--key-file=/opt/etcd/cert/etcd-key.pem \--trusted-ca-file=/opt/kubernetes/cert/ca.pem \--peer-cert-file=/opt/etcd/cert/etcd.pem \--peer-key-file=/opt/etcd/cert/etcd-key.pem \--peer-trusted-ca-file=/opt/kubernetes/cert/ca.pem \--peer-client-cert-auth \--client-cert-auth \--listen-peer-urls=https://##NODE_IP##:2380 \--initial-advertise-peer-urls=https://##NODE_IP##:2380 \--listen-client-urls=https://##NODE_IP##:2379,http://127.0.0.1:2379 \--advertise-client-urls=https://##NODE_IP##:2379 \--initial-cluster-token=etcd-cluster-0 \--initial-cluster=etcd0=https://10.103.22.231:2380,etcd1=https://10.103.22.232:2380,etcd2=https://10.103.22.233:2380 \--initial-cluster-state=newRestart=on-failureRestartSec=5LimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF
复制代码

注:

User :指定以 k8s 账户运行;

WorkingDirectory 、 --data-dir :指定工作目录和数据目录为/opt/lib/etcd ,需在启动服务前创建这个目录;

--name :指定节点名称,当 --initial-cluster-state 值为 new 时, --name 的参数值必须位于 --initial-cluster 列表中;

--cert-file 、 --key-file :etcd server 与 client 通信时使用的证书和私钥;

--trusted-ca-file :签名 client 证书的 CA 证书,用于验证 client 证书;

--peer-cert-file 、 --peer-key-file :etcd 与 peer 通信使用的证书和私钥;

--peer-trusted-ca-file :签名 peer 证书的 CA 证书,用于验证 peer 证书;


为各节点创建和分发 etcd systemd unit 文件和 etcd 数据目录

#替换模板文件中的变量,为各节点创建 systemd unit 文件NODE_NAMES=("etcd0" "etcd1" "etcd2")NODE_IPS=("10.103.22.231" "10.103.22.232" "10.103.22.233")for (( i=0; i < 3; i++ ));do        sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/g" -e "s/##NODE_IP##/${NODE_IPS[i]}/g" /opt/etcd/etcd.service.template > /opt/etcd/etcd-${NODE_IPS[i]}.servicedone#分发生成的 systemd unit 和etcd的配置文件:for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        ssh root@${node_ip} "mkdir -p /opt/lib/etcd"        scp /opt/etcd/etcd-${node_ip}.service root@${node_ip}:/etc/systemd/system/etcd.servicedone
复制代码

启动 etcd 服务

NODE_IPS=("10.103.22.231" "10.103.22.232" "10.103.22.233")#启动 etcd 服务for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        ssh root@${node_ip} "systemctl daemon-reload && systemctl enable etcd && systemctl start etcd"done#检查启动结果,确保状态为 active (running)for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        ssh root@${node_ip} "systemctl status etcd|grep Active"done#验证服务状态,输出均为healthy 时表示集群服务正常for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        ETCDCTL_API=3 /opt/kubernetes/bin/etcdctl \--endpoints=https://${node_ip}:2379 \--cacert=/opt/kubernetes/cert/ca.pem \--cert=/opt/etcd/cert/etcd.pem \--key=/opt/etcd/cert/etcd-key.pem endpoint healthdone
复制代码

显示信息:

>>> 10.103.22.231

https://10.103.22.231:2379 is healthy: successfully committed proposal: took = 14.831695ms

>>> 10.103.22.232

https://10.103.22.232:2379 is healthy: successfully committed proposal: took = 21.961696ms

>>> 10.103.22.233

https://10.103.22.233:2379 is healthy: successfully committed proposal: took = 20.714393ms


5、部署 master 节点

① kubernetes master 节点运行如下组件:

kube-apiserver

kube-scheduler

kube-controller-manager

② kube-scheduler 和 kube-controller-manager 可以以集群模式运行,通过 leader 选举产生一个工作进程,其它进程处于阻塞模式。

③ 对于 kube-apiserver,可以运行多个实例(本文档是 3 实例),但对其它组件需要提供统一的访问地址,该地址需要高可用。本文档使用 keepalived 和 haproxy 实现 kube-apiserver VIP 高可用和负载均衡。

④ 因为对 master 做了 keepalived 高可用,所以 3 台服务器都有可能会升成 master 服务器(主 master 宕机,会有从升级为主);因此所有的 master 操作,在 3 个服务器上都要进行。

5.1 将二进制文件拷贝到所有 master 节点

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}"    scp /root/software/master/kube* root@${node_ip}:/opt/kubernetes/bin/    ssh root@${node_ip} "chmod +x /opt/kubernetes/bin/*"done
复制代码

5.2 部署 kubectl 命令行工具


kubectl 是 kubernetes 集群的命令行管理工具,本文档介绍安装和配置它的步骤。

kubectl 默认从 ~/.kube/config 文件读取 kube-apiserver 地址、证书、用户名等信息,如果没有配置,执行 kubectl 命令时可能会出错:

$ kubectl get pods

The connection to the server localhost:8080 was refused - did you specify the right host or port?

本文档只需要部署一次,生成的 kubeconfig 文件与机器无关。


创建 admin 证书和私钥

kubectl 与 apiserver https 安全端口通信,apiserver 对提供的证书进行认证和授权。

kubectl 作为集群的管理工具,需要被授予最高权限。这里创建具有最高权限的 admin 证书。

cat > /opt/kubernetes/cert/admin-csr.json <<EOF{    "CN": "admin",    "hosts": [],    "key": {        "algo": "rsa",        "size": 2048    },    "names": [        {            "C": "CN",            "ST": "ShangHai",            "L": "ShangHai",            "O": "system:masters",            "OU": "zhonggang"        }    ]}EOF
复制代码

注:

① O 为 system:masters ,kube-apiserver 收到该证书后将请求的 Group 设置为 system:masters;

② 预定义的 ClusterRoleBinding cluster-admin 将 Group system:masters 与 Role cluster-admin 绑定,该 Role 授予所有 API 的权限;

③ 该证书只会被 kubectl 当做 client 证书使用,所以 hosts 字段为空;

生成证书和私钥

cfssl gencert -ca=/opt/kubernetes/cert/ca.pem \-ca-key=/opt/kubernetes/cert/ca-key.pem \-config=/opt/kubernetes/cert/ca-config.json \-profile=kubernetes admin-csr.json | cfssljson -bare admin
复制代码

创建和分发 kubeconfig 文件

kubeconfig 为 kubectl 的配置文件,包含访问 apiserver 的所有信息,如 apiserver 地址、CA 证书和自身使用的证书;

① 设置集群参数,(--server=${KUBE_APISERVER} ,指定 IP 和端口;我使用的是 haproxy 的 VIP 和端口;如果没有 haproxy 代理,就用实际服务的 IP 和端口;如:https://10.103.22.231:6443

kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/cert/ca.pem \--embed-certs=true \--server=https://10.103.22.236:8443 \--kubeconfig=/root/.kube/kubectl.kubeconfig
复制代码

② 设置客户端认证参数

kubectl config set-credentials kube-admin \--client-certificate=/opt/kubernetes/cert/admin.pem \--client-key=/opt/kubernetes/cert/admin-key.pem \--embed-certs=true \--kubeconfig=/root/.kube/kubectl.kubeconfig
复制代码

③ 设置上下文参数

kubectl config set-context kube-admin@kubernetes \--cluster=kubernetes \--user=kube-admin \--kubeconfig=/root/.kube/kubectl.kubeconfig
复制代码

④ 设置默认上下文

kubectl config use-context kube-admin@kubernetes --kubeconfig=/root/.kube/kubectl.kubeconfig
复制代码

注:在后续 kubernetes 认证,文章中会详细讲解

--certificate-authority :验证 kube-apiserver 证书的根证书;

--client-certificate 、 --client-key :刚生成的 admin 证书和私钥,连接 kube-apiserver 时使用;

--embed-certs=true :将 ca.pem 和 admin.pem 证书内容嵌入到生成的 kubectl.kubeconfig 文件中(不加时,写入的是证书文件路径);

验证 kubeconfig 文件

kubectl config view --kubeconfig=/root/.kube/kubectl.kubeconfig
apiVersion: v1clusters:- cluster: certificate-authority-data: DATA+OMITTED server: https://10.103.22.236:8443 name: kubernetescontexts:- context: cluster: kubernetes user: kube-admin name: kube-admin@kubernetescurrent-context: ""kind: Configpreferences: {}users:- name: kube-admin user: client-certificate-data: REDACTED client-key-data: REDACTED
复制代码

分发 kubeclt 和 kubeconfig 文件,分发到所有使用 kubectl 命令的节点

mv /root/.kube/kubectl.kubeconfig /root/.kube/configNODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}"    ssh root@${node_ip} "chmod +x /opt/kubernetes/bin/*"    ssh root@${node_ip} "mkdir -p ~/.kube"    scp ~/.kube/config root@${node_ip}:~/.kube/configdone
复制代码

5.3 安装配置 keepalived haproxy

3 台节点安装 keepalived haproxy

yum install -y keepalived haproxy
复制代码

配置 haproxy 配置文件

vim /etc/haproxy/haproxy.cfg

global    log /dev/log local0    log /dev/log local1 notice    chroot /var/lib/haproxy    stats socket /var/run/haproxy-admin.sock mode 660 level admin    stats timeout 30s    user haproxy    group haproxy    daemon    nbproc 1defaults    log global    timeout connect 5000    timeout client 10m    timeout server 10mlisten admin_stats    bind 0.0.0.0:10080    mode http    log 127.0.0.1 local0 err    stats refresh 30s    stats uri /status    stats realm welcome login\ Haproxy    stats auth along:along123    stats hide-version    stats admin if TRUElisten kube-master    bind 0.0.0.0:8443    mode tcp    option tcplog    balance source    server master01 10.103.22.231:6443 check inter 2000 fall 2 rise 2 weight 1    server master02 10.103.22.232:6443 check inter 2000 fall 2 rise 2 weight 1    server master03 10.103.22.233:6443 check inter 2000 fall 2 rise 2 weight 1
复制代码

注:

haproxy 在 10080 端口输出 status 信息;

haproxy 监听所有接口的 8443 端口,该端口与环境变量 ${KUBE_APISERVER} 指定的端口必须一致;

server 字段列出所有 kube-apiserver 监听的 IP 和端口;


下发 haproxy 配置文件;并启动检查 haproxy 服务

vim /opt/kubernetes/script/haproxy.sh

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}" #下发配置文件    scp /etc/haproxy/haproxy.cfg root@${node_ip}:/etc/haproxy #启动检查haproxy服务    ssh root@${node_ip} "systemctl restart haproxy"    ssh root@${node_ip} "systemctl enable haproxy.service"    ssh root@${node_ip} "systemctl status haproxy|grep Active" #检查 haproxy 是否监听8443 端口    ssh root@${node_ip} "netstat -lnpt|grep haproxy"done
复制代码

确保输出类似于:

tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 20027/haproxy

tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 20027/haproxy


配置和启动 keepalived 服务

keepalived 多备(backup)运行模式,使用非抢占模式

这样配置防止 master 出现故障在恢复的时候立即抢占 VIP (apiserver 启动提供服务需要时间,防止在 apiserver 未提供服务期间有请求调度过来)

backup:10.103.22.231、10.103.22.232、10.103.22.233

设置 keeplived 配置文件:

vim /etc/keepalived/keepalived.conf

global_defs {    router_id keepalived_hap}vrrp_script check-haproxy {    script "killall -0 haproxy"    interval 5    weight -5}vrrp_instance VI-kube-master {    state BACKUP    nopreempt    priority 200    dont_track_primary    interface ens160    virtual_router_id 68    advert_int 3    track_script {        check-haproxy    }    virtual_ipaddress {        10.103.22.236    }}
复制代码

注:

我的 VIP 所在的接口 nterface 为 eth1;根据自己的情况改变

使用 killall -0 haproxy 命令检查所在节点的 haproxy 进程是否正常。如果异常则将权重减少(-30),从而触发重新选主过程;

router_id、virtual_router_id 用于标识属于该 HA 的 keepalived 实例,如果有多套 keepalived HA,则必须各不相同;

重点:

1、三个节点的 state 都必须配置为 BACKUP

2、三个节点都必须加上配置 nopreempt

3、其中一个节点的优先级必须要高于另外两个节点的优先级

4、在 master02 和 master03 的配置中只需将 priority 200 调成 150 和 100 即可


开启 keepalived 服务

NODE_IPS=("master01" "master02" "master03")VIP="10.103.22.236"for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}"    ssh root@${node_ip} "systemctl restart keepalived && systemctl enable keepalived"    ssh root@${node_ip} "systemctl status keepalived|grep Active"    ssh ${node_ip} "ping -c 3 ${VIP}"done
复制代码

查看网卡 ip:ip a show ens160

2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000

link/ether 00:50:56:a0:ed:1f brd ff:ff:ff:ff:ff:ff

inet 10.103.22.231/24 brd 10.103.22.255 scope global noprefixroute ens160

validlft forever preferredlft forever

inet 10.103.22.236/32 scope global ens160

validlft forever preferredlft forever

5.4 部署 kube-apiserver 组件

创建证书签名请求:

cat > /opt/kubernetes/cert/kubernetes-csr.json <<EOF{    "CN": "kubernetes",    "hosts": [        "127.0.0.1",        "10.103.22.231",        "10.103.22.232",        "10.103.22.233",        "10.103.22.236",        "10.96.0.1",        "kubernetes",        "kubernetes.default",        "kubernetes.default.svc",        "kubernetes.default.svc.cluster",        "kubernetes.default.svc.cluster.local"    ],    "key": {        "algo": "rsa",        "size": 2048    },    "names": [        {          "C": "CN",          "ST": "ShangHai",          "L": "ShangHai",          "O": "ops",          "OU": "kubernetes"        }    ]}EOF
复制代码

生成证书和私钥

cfssl gencert -ca=/opt/kubernetes/cert/ca.pem \-ca-key=/opt/kubernetes/cert/ca-key.pem \-config=/opt/kubernetes/cert/ca-config.json \-profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes
复制代码

创建加密配置文件

head -c 32 /dev/urandom | base64
复制代码

注意:每台 master 节点需要用一样的 Key


使用这个加密的 key,创建加密配置文件

vim /opt/kubernetes/cert/encryption-config.yaml

kind: EncryptionConfigapiVersion: v1resources:  - resources:      - secrets    providers:      - aescbc:          keys:            - name: key1              secret: uS+YQXYoi1nxvI1pfSc2wRt64h/Iu5/4GxCuSvN+/jI=      - identity: {}
复制代码

将生成的证书和私钥文件、加密配置文件拷贝到 master 节点的/opt/k8s 目录下

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do    echo  ">>> ${node_ip}"    ssh root@${node_ip} "mkdir -p /opt/kubernetes/cert/"    scp /opt/kubernetes/cert/kubernetes*.pem root@${node_ip}:/opt/kubernetes/cert/    scp /opt/kubernetes/cert/encryption-config.yaml root@${node_ip}:/opt/kubernetes/done
复制代码

创建 kube-apiserver systemd unit 模板文件

mkdir -p /opt/apiservercat > /opt/apiserver/kube-apiserver.service.template <<EOF[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target
[Service]ExecStart=/opt/kubernetes/bin/kube-apiserver \--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \--anonymous-auth=false \--experimental-encryption-provider-config=/opt/kubernetes/encryption-config.yaml \--advertise-address=##NODE_IP## \--bind-address=##NODE_IP## \--insecure-port=0 \--authorization-mode=Node,RBAC \--runtime-config=api/all=true \--enable-bootstrap-token-auth \--service-cluster-ip-range=10.96.0.0/16 \--service-node-port-range=32767-50000 \--tls-cert-file=/opt/kubernetes/cert/kubernetes.pem \--tls-private-key-file=/opt/kubernetes/cert/kubernetes-key.pem \--client-ca-file=/opt/kubernetes/cert/ca.pem \--kubelet-client-certificate=/opt/kubernetes/cert/kubernetes.pem \--kubelet-client-key=/opt/kubernetes/cert/kubernetes-key.pem \--service-account-key-file=/opt/kubernetes/cert/ca-key.pem \--etcd-cafile=/opt/kubernetes/cert/ca.pem \--etcd-certfile=/opt/kubernetes/cert/kubernetes.pem \--etcd-keyfile=/opt/kubernetes/cert/kubernetes-key.pem \--etcd-servers=https://10.103.22.231:2379,https://10.103.22.232:2379,https://10.103.22.233:2379 \--enable-swagger-ui=true \--allow-privileged=true \--apiserver-count=3 \--audit-log-maxage=30 \--audit-log-maxbackup=3 \--audit-log-maxsize=100 \--audit-log-path=/var/log/kube-apiserver-audit.log \--event-ttl=1h \--alsologtostderr=true \--logtostderr=false \--log-dir=/opt/log/kubernetes \--v=2Restart=on-failureRestartSec=5Type=notifyUser=rootLimitNOFILE=65536
[Install]WantedBy=multi-user.targetEOF
复制代码

注:

  • --experimental-encryption-provider-config :启用加密特性;

  • --authorization-mode=Node,RBAC : 开启 Node 和 RBAC 授权模式,拒绝未授权的请求;

  • --enable-admission-plugins :启用 ServiceAccount 和 NodeRestriction ;

  • --service-account-key-file :签名 ServiceAccount Token 的公钥文件,kube-controller-manager 的 --service-account-private-key-file 指定私钥文件,两者配对使用;

  • --tls-*-file :指定 apiserver 使用的证书、私钥和 CA 文件。 --client-ca-file 用于验证 client (kue-controller-manager、kube-scheduler、kubelet、kube-proxy 等)请求所带的证书;

  • --kubelet-client-certificate 、 --kubelet-client-key :如果指定,则使用 https 访问 kubelet APIs;需要为证书对应的用户(上面 kubernetes*.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则,否则访问 kubelet API 时提示未授权;

  • --bind-address : 不能为 127.0.0.1 ,否则外界不能访问它的安全端口 6443;

  • --insecure-port=0 :关闭监听非安全端口(8080);

  • --service-cluster-ip-range : 指定 Service Cluster IP 地址段;

  • --service-node-port-range : 指定 NodePort 的端口范围;

  • --runtime-config=api/all=true : 启用所有版本的 APIs,如 autoscaling/v2alpha1;

  • --enable-bootstrap-token-auth :启用 kubelet bootstrap 的 token 认证;

  • --apiserver-count=3 :指定集群运行模式,多台 kube-apiserver 会通过 leader 选举产生一个工作节点,其它节点处于阻塞状态;

  • User=root :使用 k8s 账户运行;


为各节点创建和分发 kube-apiserver systemd unit 文件;启动检查 kube-apiserver 服务

vim /opt/kubernetes/script/apiserver_service.sh

NODE_IPS=("10.103.22.231" "0.103.22.232" "0.103.22.233")#替换模板文件中的变量,为各节点创建 systemd unit 文件for (( i=0; i < 3; i++ ));do    sed "s/##NODE_IP##/${NODE_IPS[i]}/" /opt/apiserver/kube-apiserver.service.template > /opt/apiserver/kube-apiserver-${NODE_IPS[i]}.servicedone#启动并检查 kube-apiserver 服务for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}"    ssh root@${node_ip} "mkdir -p /opt/log/kubernetes && chown -R k8s /opt/log/kubernetes"    scp /opt/apiserver/kube-apiserver-${node_ip}.service root@${node_ip}:/etc/systemd/system/kube-apiserver.service    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver"    ssh root@${node_ip} "systemctl status kube-apiserver |grep 'Active:'"done
复制代码

打印 kube-apiserver 写入 etcd 的数据

ETCDCTL_API=3 etcdctl \--endpoints="https://10.103.22.231:2379,https://10.103.22.231:2379,https://10.103.22.231:2379" \--cacert=/opt/kubernetes/cert/ca.pem \--cert=/opt/etcd/cert/etcd.pem \--key=/opt/etcd/cert/etcd-key.pem \get /registry/ --prefix --keys-only
复制代码

检查集群信息

kubectl cluster-info
Kubernetes master is running at https://10.103.22.236:8443To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
复制代码


kubectl get all --all-namespacesNAMESPACE   NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGEdefault     service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   12h
复制代码


kubectl get componentstatuses
NAME STATUS MESSAGE ERRORscheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused etcd-0 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"}
复制代码

注意:

① 如果执行 kubectl 命令式时输出如下错误信息,则说明使用的 ~/.kube/config 文件不对,请切换到正确的账户后再执行该命令:

The connection to the server localhost:8080 was refused - did you specify the right host or port?

② 执行 kubectl get componentstatuses 命令时,apiserver 默认向 127.0.0.1 发送请求。当 controller-manager、scheduler 以集群模式运行时,有可能和 kube-apiserver 不在一台机器上,这时 controller-manager 或 scheduler 的状态为 Unhealthy,但实际上它们工作正常。

5.5 部署高可用 kube-controller-manager 集群

本文档介绍部署高可用 kube-controller-manager 集群的步骤。

该集群包含 3 个节点,启动后将通过竞争选举机制产生一个 leader 节点,其它节点为阻塞状态。当 leader 节点不可用后,剩余节点将再次进行选举产生新的 leader 节点,从而保证服务的可用性。

为保证通信安全,本文档先生成 x509 证书和私钥,kube-controller-manager 在如下两种情况下使用该证书:

① 与 kube-apiserver 的安全端口通信时;

② 在安全端口(https,10252) 输出 prometheus 格式的 metrics;


创建 kube-controller-manager 证书和私钥

cat > /opt/kubernetes/cert/kube-controller-manager-csr.json <<EOF{    "CN": "system:kube-controller-manager",    "key": {        "algo": "rsa",        "size": 2048    },    "hosts": [        "127.0.0.1",        "10.103.22.231",        "10.103.22.232",        "10.103.22.233"    ],    "names": [        {            "C": "CN",            "ST": "ShangHai",            "L": "ShangHai",            "O": "system:kube-controller-manager",            "OU": "zhonggang"        }    ]}EOF
复制代码

注:

hosts 列表包含所有 kube-controller-manager 节点 IP;

CN 为 system:kube-controller-manager、O 为 system:kube-controller-manager,kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限。


生成证书和私钥

cfssl gencert -ca=/opt/kubernetes/cert/ca.pem \-ca-key=/opt/kubernetes/cert/ca-key.pem \-config=/opt/kubernetes/cert/ca-config.json \-profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager
复制代码

创建 kubeconfig 文件

kubeconfig 文件包含访问 apiserver 的所有信息,如 apiserver 地址、CA 证书和自身使用的证书;

① 执行命令,生产 kube-controller-manager.kubeconfig 文件

kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/cert/ca.pem \--embed-certs=true \--server=https://10.103.22.236:8443 \--kubeconfig=/root/.kube/kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \--client-certificate=/opt/kubernetes/cert/kube-controller-manager.pem \--client-key=/opt/kubernetes/cert/kube-controller-manager-key.pem \--embed-certs=true \--kubeconfig=/root/.kube/kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager@kubernetes --kubeconfig=/root/.kube/kube-controller-manager.kubeconfig
复制代码

② 验证 kube-controller-manager.kubeconfig 文件

kubectl config view --kubeconfig=/root/.kube/kube-controller-manager.kubeconfig
复制代码

分发生成的证书和私钥、kubeconfig 到所有 master 节点

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}"    scp /opt/kubernetes/cert/kube-controller-manager*.pem root@${node_ip}:/opt/kubernetes/cert/    scp /root/.kube/kube-controller-manager.kubeconfig root@${node_ip}:/opt/kubernetes/done
复制代码

创建和分发 kube-controller-manager systemd unit 文件

mkdir -p /opt/controller_managercat > /opt/controller_manager/kube-controller-manager.service <<EOF[Unit]Description=Kubernetes Controller ManagerDocumentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]ExecStart=/opt/kubernetes/bin/kube-controller-manager \--port=0 \--secure-port=10252 \--bind-address=127.0.0.1 \--kubeconfig=/opt/kubernetes/kube-controller-manager.kubeconfig \--service-cluster-ip-range=10.96.0.0/16 \--cluster-name=kubernetes \--cluster-signing-cert-file=/opt/kubernetes/cert/ca.pem \--cluster-signing-key-file=/opt/kubernetes/cert/ca-key.pem \--experimental-cluster-signing-duration=8760h \--root-ca-file=/opt/kubernetes/cert/ca.pem \--service-account-private-key-file=/opt/kubernetes/cert/ca-key.pem \--leader-elect=true \--feature-gates=RotateKubeletServerCertificate=true \--controllers=*,bootstrapsigner,tokencleaner \--horizontal-pod-autoscaler-use-rest-clients=true \--horizontal-pod-autoscaler-sync-period=10s \--tls-cert-file=/opt/kubernetes/cert/kube-controller-manager.pem \--tls-private-key-file=/opt/kubernetes/cert/kube-controller-manager-key.pem \--use-service-account-credentials=true \--alsologtostderr=true \--logtostderr=false \--log-dir=/var/log/kubernetes \--v=2Restart=onRestart=on-failureRestartSec=5User=root
[Install]WantedBy=multi-user.targetEOF
复制代码

注:

  • --port=0:关闭监听 http /metrics 的请求,同时 --address 参数无效,--bind-address 参数有效;

  • --secure-port=10252、--bind-address=0.0.0.0: 在所有网络接口监听 10252 端口的 https /metrics 请求;

  • --kubeconfig:指定 kubeconfig 文件路径,kube-controller-manager 使用它连接和验证 kube-apiserver;

  • --cluster-signing-*-file:签名 TLS Bootstrap 创建的证书;

  • --experimental-cluster-signing-duration:指定 TLS Bootstrap 证书的有效期;

  • --root-ca-file:放置到容器 ServiceAccount 中的 CA 证书,用来对 kube-apiserver 的证书进行校验;

  • --service-account-private-key-file:签名 ServiceAccount 中 Token 的私钥文件,必须和 kube-apiserver 的 --service-account-key-file 指定的公钥文件配对使用;

  • --service-cluster-ip-range :指定 Service Cluster IP 网段,必须和 kube-apiserver 中的同名参数一致;

  • --leader-elect=true:集群运行模式,启用选举功能;被选为 leader 的节点负责处理工作,其它节点为阻塞状态;

  • --feature-gates=RotateKubeletServerCertificate=true:开启 kublet server 证书的自动更新特性;

  • --controllers=*,bootstrapsigner,tokencleaner:启用的控制器列表,tokencleaner 用于自动清理过期的 Bootstrap token;

  • --horizontal-pod-autoscaler-*:custom metrics 相关参数,支持 autoscaling/v2alpha1;

  • --tls-cert-file、--tls-private-key-file:使用 https 输出 metrics 时使用的 Server 证书和秘钥;

  • --use-service-account-credentials=true:

  • User=k8s:使用 k8s 账户运行;

kube-controller-manager 不对请求 https metrics 的 Client 证书进行校验,故不需要指定 --tls-ca-file 参数,而且该参数已被淘汰。


kube-controller-manager 的权限

ClusteRole: system:kube-controller-manager 的权限很小,只能创建 secret、serviceaccount 等资源对象,各 controller 的权限分散到 ClusterRole system:controller:XXX 中。

需要在 kube-controller-manager 的启动参数中添加 --use-service-account-credentials=true 参数,这样 main controller 会为各 controller 创建对应的 ServiceAccount XXX-controller。

内置的 ClusterRoleBinding system:controller:XXX 将赋予各 XXX-controller ServiceAccount 对应的 ClusterRole system:controller:XXX 权限。


分发 systemd unit 文件到所有 master 节点;启动检查 kube-controller-manager 服务

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        scp /opt/controller_manager/kube-controller-manager.service root@${node_ip}:/etc/systemd/system/        ssh root@${node_ip} "mkdir -p /opt/log/kubernetes && chown -R k8s /opt/log/kubernetes"        ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-controller-manager && systemctl start kube-controller-manager"done
for node_ip in ${NODE_IPS[@]};do echo ">>> ${node_ip}" ssh k8s@${node_ip} "systemctl status kube-controller-manager|grep Active"done
复制代码

测试 kube-controller-manager 集群的高可用

1、停掉一个或两个节点的 kube-controller-manager 服务,观察其它节点的日志,看是否获取了 leader 权限。

2、查看当前的 leader

kubectl get endpoints kube-controller-manager --namespace=kube-system -o yaml
apiVersion: v1kind: Endpointsmetadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master03_c2d0a146-c4f4-4a5b-9078-f82b3915baeb","leaseDurationSeconds":15,"acquireTime":"2020-07-07T06:43:23Z","renewTime":"2020-07-07T07:01:51Z","leaderTransitions":0}' creationTimestamp: "2020-07-07T06:43:24Z" name: kube-controller-manager namespace: kube-system resourceVersion: "18500" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager uid: 3c037ba0-bd36-4d7b-a08b-e5afce35f887
复制代码

3、停掉 master03 的 kube-controller-manager 服务,再次查看

kubectl get endpoints kube-controller-manager --namespace=kube-system -o yamlapiVersion: v1kind: Endpointsmetadata:  annotations:    control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master02_0bce6e9f-20d2-44c8-9382-4b943684cd13","leaseDurationSeconds":15,"acquireTime":"2020-07-07T07:02:08Z","renewTime":"2020-07-07T07:02:43Z","leaderTransitions":1}'  creationTimestamp: "2020-07-07T06:43:24Z"  name: kube-controller-manager  namespace: kube-system  resourceVersion: "18556"  selfLink: /api/v1/namespaces/kube-system/endpoints/kube-controller-manager  uid: 3c037ba0-bd36-4d7b-a08b-e5afce35f887
复制代码

可见,当前的 leader 为 master02 节点,(本来是在 master03 节点)完成自动切换

5.6 部署高可用 kube-scheduler 集群

该集群包含 3 个节点,启动后将通过竞争选举机制产生一个 leader 节点,其它节点为阻塞状态。当 leader 节点不可用后,剩余节点将再次进行选举产生新的 leader 节点,从而保证服务的可用性。

为保证通信安全,本文档先生成 x509 证书和私钥,kube-scheduler 在如下两种情况下使用该证书:

① 与 kube-apiserver 的安全端口通信;

② 在安全端口(https,10251) 输出 prometheus 格式的 metrics;


创建 kube-scheduler 证书和私钥

创建证书签名请求:

cat > /opt/kubernetes/cert/kube-scheduler-csr.json <<EOF{    "CN": "system:kube-scheduler",    "hosts": [      "127.0.0.1",      "10.103.22.231",      "10.103.22.232",      "10.103.22.233"    ],    "key": {        "algo": "rsa",        "size": 2048    },    "names": [      {        "C": "CN",        "ST": "ShangHai",        "L": "ShangHai",        "O": "system:kube-scheduler",        "OU": "zhonggang"      }    ]}EOF
复制代码

注:

hosts 列表包含所有 kube-scheduler 节点 IP;

CN 为 system:kube-scheduler、O 为 system:kube-scheduler,kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予 kube-scheduler 工作所需的权限。


生成证书和私钥

cfssl gencert -ca=/opt/kubernetes/cert/ca.pem \-ca-key=/opt/kubernetes/cert/ca-key.pem \-config=/opt/kubernetes/cert/ca-config.json \-profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
复制代码

创建 kubeconfig 文件

kubeconfig 文件包含访问 apiserver 的所有信息,如 apiserver 地址、CA 证书和自身使用的证书;

① 执行命令,生产 kube-scheduler.kubeconfig 文件

kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/cert/ca.pem \--embed-certs=true \--server=https://10.103.22.236:8443 \--kubeconfig=/root/.kube/kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \--client-certificate=/opt/kubernetes/cert/kube-scheduler.pem \--client-key=/opt/kubernetes/cert/kube-scheduler-key.pem \--embed-certs=true \--kubeconfig=/root/.kube/kube-scheduler.kubeconfig
kubectl config set-context system:kube-scheduler@kubernetes \--cluster=kubernetes \--user=system:kube-scheduler \--kubeconfig=/root/.kube/kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler@kubernetes --kubeconfig=/root/.kube/kube-scheduler.kubeconfig
复制代码

② 验证 kube-controller-manager.kubeconfig 文件

kubectl config view --kubeconfig=/root/.kube/kube-scheduler.kubeconfig
apiVersion: v1clusters:- cluster: certificate-authority-data: DATA+OMITTED server: https://10.103.22.236:8443 name: kubernetescontexts:- context: cluster: kubernetes user: system:kube-scheduler name: system:kube-scheduler@kubernetescurrent-context: system:kube-scheduler@kuberneteskind: Configpreferences: {}users:- name: system:kube-scheduler user: client-certificate-data: REDACTED client-key-data: REDACTED
复制代码

分发生成的证书和私钥、kubeconfig 到所有 master 节点

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        scp /opt/kubernetes/cert/kube-scheduler*.pem root@${node_ip}:/opt/kubernetes/cert/        scp /root/.kube/kube-scheduler.kubeconfig root@${node_ip}:/opt/kubernetes/done
复制代码

创建 kube-scheduler systemd unit 文件

mkdir -p /opt/schedulercat > /opt/scheduler/kube-scheduler.service <<EOF[Unit]Description=Kubernetes SchedulerDocumentation=https://github.com/GoogleCloudPlatform/kubernetes
[Service]ExecStart=/opt/kubernetes/bin/kube-scheduler \\ --address=127.0.0.1 \\ --kubeconfig=/opt/kubernetes/kube-scheduler.kubeconfig \\ --leader-elect=true \\ --alsologtostderr=true \\ --logtostderr=false \\ --log-dir=/var/log/kubernetes \\ --v=2Restart=on-failureRestartSec=5User=root
[Install]WantedBy=multi-user.targetEOF
复制代码

注:

--address:在 127.0.0.1:10251 端口接收 http /metrics 请求;kube-scheduler 目前还不支持接收 https 请求;

--kubeconfig:指定 kubeconfig 文件路径,kube-scheduler 使用它连接和验证 kube-apiserver;

--leader-elect=true:集群运行模式,启用选举功能;被选为 leader 的节点负责处理工作,其它节点为阻塞状态;

User=root:使用 root 账户运行


分发 systemd unit 文件到所有 master 节点;启动检查 kube-scheduler 服务

NODE_IPS=("master01" "master02" "master03")for node_ip in ${NODE_IPS[@]};do        echo ">>> ${node_ip}"        scp /opt/scheduler/kube-scheduler.service root@${node_ip}:/etc/systemd/system/        ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-scheduler && systemctl start kube-scheduler"done
for node_ip in ${NODE_IPS[@]};do echo ">>> ${node_ip}" ssh root@${node_ip} "systemctl status kube-scheduler|grep Active"done
复制代码

查看输出的 metric

ss -nutlp |grep kube-scheduler
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.# TYPE apiserver_audit_event_total counterapiserver_audit_event_total 0# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.# TYPE apiserver_audit_requests_rejected_total counterapiserver_audit_requests_rejected_total 0# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.# TYPE apiserver_client_certificate_expiration_seconds histogramapiserver_client_certificate_expiration_seconds_bucket{le="0"} 0apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
复制代码

测试 kube-scheduler 集群的高可用

1、随便找一个或两个 master 节点,停掉 kube-scheduler 服务,看其它节点是否获取了 leader 权限(systemd 日志)。

2、查看当前的 leader

kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1kind: Endpointsmetadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master01_36a4ff6c-aff2-4bd4-a14a-cda89bcc76f2","leaseDurationSeconds":15,"acquireTime":"2020-07-07T08:00:13Z","renewTime":"2020-07-07T08:09:48Z","leaderTransitions":0}' creationTimestamp: "2020-07-07T07:55:40Z" name: kube-scheduler namespace: kube-system resourceVersion: "23879" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler uid: e6148c56-9f65-4d89-89a2-8eaef5029c76
复制代码

3、停掉 master01 的 scheduler 服务,再次查看

kubectl get endpoints kube-scheduler --namespace=kube-system -o yaml
apiVersion: v1kind: Endpointsmetadata: annotations: control-plane.alpha.kubernetes.io/leader: '{"holderIdentity":"master03_e54fdc78-0505-4068-9873-c4c3d45f7439","leaseDurationSeconds":15,"acquireTime":"2020-07-07T08:07:31Z","renewTime":"2020-07-07T08:08:06Z","leaderTransitions":1}' creationTimestamp: "2020-07-07T07:55:40Z" name: kube-scheduler namespace: kube-system resourceVersion: "24250" selfLink: /api/v1/namespaces/kube-system/endpoints/kube-scheduler uid: e6148c56-9f65-4d89-89a2-8eaef5029c76
复制代码

查看 leader 已经由 master01 切换到 master03

6、部署 worker 节点

kubernetes work 节点运行如下组件:

docker

kubelet

kube-proxy

依赖环境可以看之前的环境准备里的系统设置和 Docker 环境准备


将二进制文件拷贝到所有 node 节点

NODE_IPS=("master01" "master02" "master03" "node04" "node05")for node_ip in ${NODE_IPS[@]};do    echo ">>> ${node_ip}"    scp /root/software/worker/kube* root@${node_ip}:/opt/kubernetes/bin/    ssh root@${node_ip} "chmod +x /opt/kubernetes/bin/*"done
复制代码

6.1 部署 kubelet 组件

vim /opt/kubernetes/script/bootstrap_kubeconfig.sh

NODE_NAMES=("master01" "master02" "master03" "node01" "node02")for node_name in ${NODE_NAMES[@]};do    echo ">>> ${node_name}"    # 创建 token    export BOOTSTRAP_TOKEN=$(kubeadm token create \    --description kubelet-bootstrap-token \    --groups system:bootstrappers:${node_name} \    --kubeconfig /root/.kube/config)
# 设置集群参数 kubectl config set-cluster kubernetes \ --certificate-authority=/opt/kubernetes/cert/ca.pem \ --embed-certs=true \ --server=https://10.103.22.236:8443 \ --kubeconfig=/root/.kube/kubelet-bootstrap-${node_name}.kubeconfig
# 设置客户端认证参数 kubectl config set-credentials kubelet-bootstrap \ --token=${BOOTSTRAP_TOKEN} \ --kubeconfig=/root/.kube/kubelet-bootstrap-${node_name}.kubeconfig
# 设置上下文参数 kubectl config set-context default \ --cluster=kubernetes \ --user=kubelet-bootstrap \ --kubeconfig=/root/.kube/kubelet-bootstrap-${node_name}.kubeconfig
# 设置默认上下文 kubectl config use-context default --kubeconfig=/root/.kube/kubelet-bootstrap-${node_name}.kubeconfigdone
复制代码

① 证书中写入 Token 而非证书,证书后续由 controller-manager 创建。

注:

① 证书中写入 Token 而非证书,证书后续由 controller-manager 创建。

查看 kubeadm 为各节点创建的 token:

kubeadm token list --kubeconfig ~/.kube/config
3ko7un.8fygjyumantb15aa 23h 2020-07-08T17:23:24+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:master03f2ax8x.l23iutub0fwpzjd8 23h 2020-07-08T17:23:23+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:master01ktkl5d.8joag290woeqefgp 23h 2020-07-08T17:23:25+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node05l5bhhc.su2gx09zu22h4s98 23h 2020-07-08T17:23:24+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:node04mggzm3.l0jgha3p9f6lkfna 23h 2020-07-08T17:23:23+08:00 authentication,signing kubelet-bootstrap-token system:bootstrappers:master02
复制代码

② 创建的 token 有效期为 1 天,超期后将不能再被使用,且会被 kube-controller-manager 的 tokencleaner 清理(如果启用该 controller 的话);

③ kube-apiserver 接收 kubelet 的 bootstrap token 后,将请求的 user 设置为 system:bootstrap:,group 设置为 system:bootstrappers;

各 token 关联的 Secret:

kubectl get secrets -n kube-system
bootstrap-token-0gp3v4 bootstrap.kubernetes.io/token 7 4m55sbootstrap-token-13j1jc bootstrap.kubernetes.io/token 7 2m21sbootstrap-token-157pyj bootstrap.kubernetes.io/token 7 35mbootstrap-token-2671ps bootstrap.kubernetes.io/token 7 16mbootstrap-token-2wstjd bootstrap.kubernetes.io/token 7 16mbootstrap-token-360ht3 bootstrap.kubernetes.io/token 7 2m21sbootstrap-token-36sz76 bootstrap.kubernetes.io/token 7 2m57s
复制代码

创建 kubelet 参数配置文件

mkdir -p /opt/kubelet

vim /opt/kubelet/kubelet.config.json.template

{  "kind": "KubeletConfiguration",  "apiVersion": "kubelet.config.k8s.io/v1beta1",  "authentication": {    "x509": {      "clientCAFile": "/opt/kubernetes/cert/ca.pem"    },    "webhook": {      "enabled": true,      "cacheTTL": "2m0s"    },    "anonymous": {      "enabled": false    }  },  "authorization": {    "mode": "Webhook",    "webhook": {      "cacheAuthorizedTTL": "5m0s",      "cacheUnauthorizedTTL": "30s"    }  },  "address": "##NODE_IP##",  "port": 10250,  "readOnlyPort": 0,  "cgroupDriver": "systemd",  "hairpinMode": "promiscuous-bridge",  "serializeImagePulls": false,  "featureGates": {    "RotateKubeletClientCertificate": true,    "RotateKubeletServerCertificate": true  },  "clusterDomain": "cluster.local",  "clusterDNS": ["10.96.0.2"]}
复制代码

分发 bootstrap kubeconfig 、kubelet 配置文件到所有 worker 节点

vim /opt/kubernetes/script/scp_kubelet.sh

NODE_IPS=("10.103.22.231" "10.103.22.232" "10.103.22.233" "10.103.22.234" "10.103.22.235")NODE_NAMES=("master01" "master02" "master03" "node04" "node05")for node_name in ${NODE_NAMES[@]};do    echo ">>> ${node_name}"    scp ~/.kube/kubelet-bootstrap-${node_name}.kubeconfig root@${node_name}:/opt/kubernetes/kubelet-bootstrap.kubeconfigdone
for node_ip in ${NODE_IPS[@]};do echo ">>> ${node_ip}" sed -e "s/##NODE_IP##/${node_ip}/" /opt/kubelet/kubelet.config.json.template > /opt/kubelet/kubelet.config-${node_ip}.json scp /opt/kubelet/kubelet.config-${node_ip}.json root@${node_ip}:/opt/kubernetes/kubelet.config.jsondone
复制代码

创建 kubelet systemd unit 文件

vim /opt/kubelet/kubelet.service.template

[Unit]Description=Kubernetes KubeletDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=docker.serviceRequires=docker.service
[Service]WorkingDirectory=/opt/lib/kubeletExecStart=/opt/kubernetes/bin/kubelet \--bootstrap-kubeconfig=/opt/kubernetes/kubelet-bootstrap.kubeconfig \--cert-dir=/opt/kubernetes/cert \--kubeconfig=/opt/kubernetes/kubelet.kubeconfig \--config=/opt/kubernetes/kubelet.config.json \--hostname-override=##NODE_NAME## \--pod-infra-container-image=registry.access.redhat.com/rhel7/pod-infrastructure:latest \--alsologtostderr=true \--logtostderr=false \--log-dir=/opt/log/kubernetes \--network-plugin=cni \--v=2Restart=on-failureRestartSec=5
[Install]WantedBy=multi-user.target
复制代码

Bootstrap Token Auth 和授予权限

1、kublet 启动时查找配置的 --kubeletconfig 文件是否存在,如果不存在则使用 --bootstrap-kubeconfig 向 kube-apiserver 发送证书签名请求 (CSR)。

2、kube-apiserver 收到 CSR 请求后,对其中的 Token 进行认证(事先使用 kubeadm 创建的 token),认证通过后将请求的 user 设置为 system:bootstrap:,group 设置为 system:bootstrappers,这一过程称为 Bootstrap Token Auth。

3、默认情况下,这个 user 和 group 没有创建 CSR 的权限,kubelet 启动失败,错误日志如下:

$ sudo journalctl -u kubelet -a |grep -A 2 'certificatesigningrequests' May 06 06:42:36 kube-node1 kubelet[26986]: F0506 06:42:36.314378 26986 server.go:233] failed to run Kubelet: cannot create certificate signing request: certificatesigningrequests.certificates.k8s.io is forbidden: User "system:bootstrap:lemy40" cannot create certificatesigningrequests.certificates.k8s.io at the cluster scope May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a May 06 06:42:36 kube-node1 systemd[1]: kubelet.service: Failed with result 'exit-code'.

4、解决办法是:创建一个 clusterrolebinding,将 group system:bootstrappers 和 clusterrole system:node-bootstrapper 绑定:

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --group=system:bootstrappers
复制代码

启动 kubelet 服务

vim /opt/kubernetes/script/kubelet.sh

NODE_IPS=("10.103.22.231" "10.103.22.232" "10.103.22.233" "10.103.22.234" "10.103.22.235")NODE_NAMES=("master01" "master02" "master03" "node04" "node05")#分发kubelet systemd unit 文件for node_name in ${NODE_NAMES[@]};do     echo ">>> ${node_name}"    sed -e "s/##NODE_NAME##/${node_name}/" /opt/kubelet/kubelet.service.template > /opt/kubelet/kubelet-${node_name}.service    scp /opt/kubelet/kubelet-${node_name}.service root@${node_name}:/etc/systemd/system/kubelet.servicedone#开启检查kubelet 服务for node_ip in ${NODE_IPS[@]};do    ssh root@${node_ip} "mkdir -p /opt/lib/kubelet"    ssh root@${node_ip} "mkdir -p /opt/log/kubernetes"    ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kubelet && systemctl restart kubelet"    ssh root@${node_ip} "systemctl status kubelet |grep active"done
复制代码

注:

关闭 swap 分区,注意/etc/fstab 要设为开机不启动 swap 分区,否则 kubelet 会启动失败;

必须先创建工作和日志目录;

kubelet 启动后使用 --bootstrap-kubeconfig 向 kube-apiserver 发送 CSR 请求,当这个 CSR 被 approve 后,kube-controller-manager 为 kubelet 创建 TLS 客户端证书、私钥和 --kubeletconfig 文件。

kube-controller-manager 需要配置 --cluster-signing-cert-file 和 --cluster-signing-key-file 参数,才会为 TLS Bootstrap 创建证书和私钥。


approve kubelet CSR 请求

可以手动或自动 approve CSR 请求。推荐使用自动的方式,因为从 v1.8 版本开始,可以自动轮转 approve csr 后生成的证书。

1、手动 approve CSR 请求

(1)查看 CSR 列表:

kubectl get csr
NAME AGE REQUESTOR CONDITIONnode-csr-9lyr14IKuVosJNZvXaxakuCC0W6jNrsVwnUL5iPIvUk 8m29s system:bootstrap:f2ax8x Pendingnode-csr-EJY5yoigTqvOpJjhQJAAJlMTB5f2qyVP06VWRxnELzI 103s system:bootstrap:3ko7un Pendingnode-csr-SYguhKLGRIQHMYz4_AwRUuOl2vsMDWiEIvGKpRcTkZE 103s system:bootstrap:ktkl5d Pendingnode-csr-iZ6Khob7xMhHLzrf9EmuxI80Fc9_wKDdUYl3I8CmXMI <invalid> system:bootstrap:l5bhhc Pendingnode-csr-yjO-RDE08vVft0ljiqYqhtb0K2ZbsbsQne7OeYrwyUg 106s system:bootstrap:mggzm3 Pending
复制代码

三个 work 节点的 csr 均处于 pending 状态;

(2)approve CSR:

kubectl certificate approve node-csr-iZ6Khob7xMhHLzrf9EmuxI80Fc9_wKDdUYl3I8CmXMI
certificatesigningrequest.certificates.k8s.io/node-csr-iZ6Khob7xMhHLzrf9EmuxI80Fc9_wKDdUYl3I8CmXMI approved
复制代码

再次查看 CSR 列表

kubectl get csr
NAME AGE REQUESTOR CONDITIONnode-csr-9lyr14IKuVosJNZvXaxakuCC0W6jNrsVwnUL5iPIvUk 21m system:bootstrap:f2ax8x Pendingnode-csr-EJY5yoigTqvOpJjhQJAAJlMTB5f2qyVP06VWRxnELzI 15m system:bootstrap:3ko7un Pendingnode-csr-SYguhKLGRIQHMYz4_AwRUuOl2vsMDWiEIvGKpRcTkZE 15m system:bootstrap:ktkl5d Pendingnode-csr-iZ6Khob7xMhHLzrf9EmuxI80Fc9_wKDdUYl3I8CmXMI 10m system:bootstrap:l5bhhc Approved,Issuednode-csr-yjO-RDE08vVft0ljiqYqhtb0K2ZbsbsQne7OeYrwyUg 15m system:bootstrap:mggzm3 Pending
复制代码

可以看到已经有一个节点处于 Approved 状态了

(3)查看 Approve 结果

kubectl describe csr node-csr-iZ6Khob7xMhHLzrf9EmuxI80Fc9_wKDdUYl3I8CmXMI
Name: node-csr-iZ6Khob7xMhHLzrf9EmuxI80Fc9_wKDdUYl3I8CmXMILabels: <none>Annotations: <none>CreationTimestamp: Tue, 07 Jul 2020 18:08:08 +0800Requesting User: system:bootstrap:l5bhhcStatus: Approved,IssuedSubject: Common Name: system:node:node04 Serial Number: Organization: system:nodesEvents: <none>
复制代码

自动 approve CSR 请求

(1)创建三个 ClusterRoleBinding,分别用于自动 approve client、renew client、renew server 证书:

cat > /opt/kubelet/csr-crb.yaml <<EOF# Approve all CSRs for the group "system:bootstrappers" kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: auto-approve-csrs-for-group subjects: - kind: Group name:# Approve all CSRs for the group "system:bootstrappers"kind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:  name: auto-approve-csrs-for-groupsubjects:- kind: Group  name: system:bootstrappers  apiGroup: rbac.authorization.k8s.ioroleRef:  kind: ClusterRole  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient  apiGroup: rbac.authorization.k8s.io---# To let a node of the group "system:nodes" renew its own credentialskind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:  name: node-client-cert-renewalsubjects:- kind: Group  name: system:nodes  apiGroup: rbac.authorization.k8s.ioroleRef:  kind: ClusterRole  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient  apiGroup: rbac.authorization.k8s.io---# A ClusterRole which instructs the CSR approver to approve a node requesting a# serving cert matching its client cert.kind: ClusterRoleapiVersion: rbac.authorization.k8s.io/v1metadata:  name: approve-node-server-renewal-csrrules:- apiGroups: ["certificates.k8s.io"]  resources: ["certificatesigningrequests/selfnodeserver"]  verbs: ["create"]---# To let a node of the group "system:nodes" renew its own server credentialskind: ClusterRoleBindingapiVersion: rbac.authorization.k8s.io/v1metadata:  name: node-server-cert-renewalsubjects:- kind: Group  name: system:nodes  apiGroup: rbac.authorization.k8s.ioroleRef:  kind: ClusterRole  name: approve-node-server-renewal-csr  apiGroup: rbac.authorization.k8s.ioEOF
复制代码

注:

auto-approve-csrs-for-group:自动 approve node 的第一次 CSR; 注意第一次 CSR 时,请求的 Group 为 system:bootstrappers;

node-client-cert-renewal:自动 approve node 后续过期的 client 证书,自动生成的证书 Group 为 system:nodes;

node-server-cert-renewal:自动 approve node 后续过期的 server 证书,自动生成的证书 Group 为 system:nodes;

(2)生效配置:

kubectl apply -f /opt/kubelet/csr-crb.yaml
复制代码

查看 kublet 的情况

1、等待一段时间(1-10 分钟),三个节点的 CSR 都被自动 approve:

kubectl get csr
复制代码

kubelet 提供的 API 接口

1、kublet 启动后监听多个端口,用于接收 kube-apiserver 或其它组件发送的请求:

ss -nutlp |grep kubelet
tcp LISTEN 0 128 127.0.0.1:10248 *:* users:(("kubelet",pid=20268,fd=26))tcp LISTEN 0 128 10.103.22.231:10250 *:* users:(("kubelet",pid=20268,fd=36))tcp LISTEN 0 128 127.0.0.1:38991 *:* users:(("kubelet",pid=20268,fd=13))
复制代码

注:

4194: cadvisor http 服务;

10248: healthz http 服务;

10250: https API 服务;注意:未开启只读端口 10255;


2、例如执行 kubectl exec -it nginx-ds-5rmws -- sh 命令时,kube-apiserver 会向 kubelet 发送如下请求:

POST /exec/default/nginx-ds-5rmws/my-nginx?command=sh&input=1&output=1&tty=1


3、kubelet 接收 10250 端口的 https 请求:

/pods、/runningpods

/metrics、/metrics/cadvisor、/metrics/probes

/spec

/stats、/stats/container

/logs

/run/、"/exec/", "/attach/", "/portForward/", "/containerLogs/" 等管理;


4、由于关闭了匿名认证,同时开启了 webhook 授权,所有访问 10250 端口 https API 的请求都需要被认证和授权。

预定义的 ClusterRole system:kubelet-api-admin 授予访问 kubelet 所有 API 的权限:

kubectl describe clusterrole system:kubelet-api-admin
Name: system:kubelet-api-adminLabels: kubernetes.io/bootstrapping=rbac-defaultsAnnotations: rbac.authorization.kubernetes.io/autoupdate: truePolicyRule: Resources Non-Resource URLs Resource Names Verbs --------- ----------------- -------------- ----- nodes/log [] [] [*] nodes/metrics [] [] [*] nodes/proxy [] [] [*] nodes/spec [] [] [*] nodes/stats [] [] [*] nodes [] [] [get list watch proxy]
复制代码

kublet api 认证和授权

1、kublet 配置了如下认证参数:

authentication.anonymous.enabled:设置为 false,不允许匿名访问 10250 端口;

authentication.x509.clientCAFile:指定签名客户端证书的 CA 证书,开启 HTTPs 证书认证;

authentication.webhook.enabled=true:开启 HTTPs bearer token 认证;

同时配置了如下授权参数:

authroization.mode=Webhook:开启 RBAC 授权;

2、kubelet 收到请求后,使用 clientCAFile 对证书签名进行认证,或者查询 bearer token 是否有效。如果两者都没通过,则拒绝请求,提示 Unauthorized:

curl -s --cacert /opt/k8s/cert/ca.pem https://10.103.22.231:10250/metrics
Unauthorized
复制代码


 curl -s --cacert /opt/k8s/cert/ca.pem -H "Authorization: Bearer 123456" https://10.103.22.231:10250/metrics  Unauthorized
复制代码

3、通过认证后,kubelet 使用 SubjectAccessReview API 向 kube-apiserver 发送请求,查询证书或 token 对应的 user、group 是否有操作资源的权限(RBAC);

证书认证和授权:

权限不足的证书;

curl -s --cacert /opt/k8s/cert/ca.pem --cert /opt/k8s/cert/kube-controller-manager.pem --key /opt/k8s/cert/kube-controller-manager-key.pem https://10.103.22.231:10250/metrics
Forbidden (user=system:kube-controller-manager, verb=get, resource=nodes, subresource=metrics)
复制代码

使用部署 kubectl 命令行工具时创建的、具有最高权限的 admin 证书

curl -s --cacert /opt/k8s/cert/ca.pem --cert /opt/k8s/cert/admin.pem --key /opt/k8s/cert/admin-key.pem https://10.103.22.231:10250/metrics|head
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.# TYPE apiserver_client_certificate_expiration_seconds histogramapiserver_client_certificate_expiration_seconds_bucket{le="0"} 0apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0
复制代码

--cacert、--cert、--key 的参数值必须是文件路径,如上面的/opt/k8s/cert/admin.pem 不能省略 ./,否则返回 401 Unauthorized;


4、bear token 认证和授权:

创建一个 ServiceAccount,将它和 ClusterRole system:kubelet-api-admin 绑定,从而具有调用 kubelet API 的权限:

kubectl create sa kubelet-api-testkubectl create clusterrolebinding kubelet-api-test --clusterrole=system:kubelet-api-admin --serviceaccount=default:kubelet-api-testSECRET=$(kubectl get secrets | grep kubelet-api-test | awk '{print $1}')TOKEN=$(kubectl describe secret ${SECRET} | grep -E '^token' | awk '{print $2}')
复制代码


curl -s --cacert /opt/k8s/cert/ca.pem -H "Authorization: Bearer ${TOKEN}" https://10.103.22.231:10250/metrics|head
# HELP apiserver_client_certificate_expiration_seconds Distribution of the remaining lifetime on the certificate used to authenticate a request.# TYPE apiserver_client_certificate_expiration_seconds histogramapiserver_client_certificate_expiration_seconds_bucket{le="0"} 0apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0
复制代码

cadvisor 和 metrics

cadvisor 统计所在节点各容器的资源(CPU、内存、磁盘、网卡)使用情况,分别在自己的 http web 页面(4194 端口)和 10250 以 promehteus metrics 的形式输出。

浏览器访问 http://10.103.22.231:4194/containers/ 可以查看到 cadvisor 的监控页面:

6.2 部署 kube-proxy 组件

kube-proxy 运行在所有 worker 节点上,,它监听 apiserver 中 service 和 Endpoint 的变化情况,创建路由规则来进行服务负载均衡。

本文档讲解部署 kube-proxy 的部署,使用 ipvs 模式。

1、下载和分发 kube-proxy 二进制文件

二进制文件已经下发

2、安装依赖包

各节点需要安装 ipvsadm 和 ipset 命令,加载 ip_vs 内核模块。

依赖参见系统设置里面已经安装

/usr/sbin/modprobe ip_vs


创建 kube-proxy 证书

创建证书签名请求:

cat > /opt/kubernetes/cert/kube-proxy-csr.json << EOF{  "CN": "system:kube-proxy",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "ShangHai",      "L": "ShangHai",      "O": "ops",      "OU": "zhonggang"    }  ]}EOF
复制代码

注:

CN:指定该证书的 User 为 system:kube-proxy;

预定义的 RoleBinding system:node-proxier 将 User system:kube-proxy 与 Role system:node-proxier 绑定,该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限;

该证书只会被 kube-proxy 当做 client 证书使用,所以 hosts 字段为空;


生成证书和私钥

cfssl gencert -ca=/opt/kubernetes/cert/ca.pem \-ca-key=/opt/kubernetes/cert/ca-key.pem \-config=/opt/kubernetes/cert/ca-config.json \-profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
复制代码

创建 kubeconfig 文件

kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/cert/ca.pem \--embed-certs=true \--server=https://10.103.22.236:8443 \--kubeconfig=/root/.kube/kube-proxy.kubeconfig
kubectl config set-credentials kube-proxy \--client-certificate=/opt/kubernetes/cert/kube-proxy.pem \--client-key=/opt/kubernetes/cert/kube-proxy-key.pem \--embed-certs=true \--kubeconfig=/root/.kube/kube-proxy.kubeconfig
kubectl config set-context kube-proxy@kubernetes \--cluster=kubernetes \--user=kube-proxy \--kubeconfig=/root/.kube/kube-proxy.kubeconfig
kubectl config use-context kube-proxy@kubernetes --kubeconfig=/root/.kube/kube-proxy.kubeconfig
复制代码

注:

--embed-certs=true:将 ca.pem 和 admin.pem 证书内容嵌入到生成的 kubectl-proxy.kubeconfig 文件中(不加时,写入的是证书文件路径);


查看配置文件

kubectl config view --kubeconfig=/root/.kube/kube-proxy.kubeconfig
apiVersion: v1clusters:- cluster: certificate-authority-data: DATA+OMITTED server: https://10.103.22.236:8443 name: kubernetescontexts:- context: cluster: kubernetes user: kube-proxy name: kube-proxy@kubernetescurrent-context: kube-proxy@kuberneteskind: Configpreferences: {}users:- name: kube-proxy user: client-certificate-data: REDACTED client-key-data: REDACTED
复制代码

创建 kube-proxy 配置文件

创建 kube-proxy config 文件模板

mkdir -p /opt/kube-proxycat >/opt/kube-proxy/kube-proxy.config.yaml.template <<EOFapiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: ##NODE_IP##clientConnection:  kubeconfig: /root/.kube/kube-proxy.kubeconfigclusterCIDR: 10.98.0.0/16healthzBindAddress: ##NODE_IP##:10256hostnameOverride: ##NODE_NAME##kind: KubeProxyConfigurationmetricsBindAddress: ##NODE_IP##:10249mode: "ipvs"EOF
复制代码

注:

bindAddress: 监听地址;

clientConnection.kubeconfig: 连接 apiserver 的 kubeconfig 文件;

clusterCIDR: kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量,指定 --cluster-cidr 或 --masquerade-all 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT;

hostnameOverride: 参数值必须与 kubelet 的值一致,否则 kube-proxy 启动后会找不到该 Node,从而不会创建任何 ipvs 规则;

mode: 使用 ipvs 模式;


分发 kubeconfig、kube-proxy systemd unit 文件;启动并检查 kube-proxy 服务

创建 kube-proxy systemd unit 文件

cat > /opt/kube-proxy/kube-proxy.service <<EOF[Unit]Description=Kubernetes Kube-Proxy ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target[Service]WorkingDirectory=/opt/kubernetes/binExecStart=/opt/kubernetes/bin/kube-proxy \\  --config=/opt/kubernetes/kube-proxy.config.yaml \\  --logtostderr=true \\  --v=2Restart=on-failureRestartSec=5LimitNOFILE=65536[Install]WantedBy=multi-user.targetEOF
复制代码

分发配置文件和启动文件

vim /opt/kubernetes/script/kube_proxy.sh

NODE_IPS=("10.103.22.231" "10.103.22.232" "10.103.22.233" "10.103.22.234" "10.103.22.235")NODE_NAMES=("master01" "master02" "master03" "node04" "node05")for (( i=0; i < 5; i++ ));do     echo ">>> ${NODE_NAMES[i]}"    sed -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[i]}/" /opt/kube-proxy/kube-proxy.config.yaml.template > /opt/kube-proxy/kube-proxy-${NODE_NAMES[i]}.config.yaml    scp /opt/kube-proxy/kube-proxy-${NODE_NAMES[i]}.config.yaml root@${NODE_NAMES[i]}:/opt/kubernetes/kube-proxy.config.yamldone
for node_ip in ${NODE_IPS[@]};do echo ">>> ${node_ip}" ssh root@${node_ip} "mkdir -p /root/.kube" scp /root/.kube/kube-proxy.kubeconfig root@${node_ip}:/root/.kube scp /opt/kube-proxy/kube-proxy.service root@${node_ip}:/etc/systemd/system/ ssh root@${node_ip} "mkdir -p /opt/lib/kube-proxy" ssh root@${node_ip} "mkdir -p /opt/log/kubernetes" ssh root@${node_ip} "systemctl daemon-reload && systemctl enable kube-proxy && systemctl restart kube-proxy" ssh root@${node_ip} "systemctl status kube-proxy|grep Active"done
复制代码

查看监听端口和 metrics

ss -lntp |grep kube-proxy
LISTEN 0 128 10.103.22.231:10249 *:* users:(("kube-proxy",pid=27957,fd=12))LISTEN 0 128 10.103.22.231:10256 *:* users:(("kube-proxy",pid=27957,fd=10))
复制代码

查看 ipvs 路由规则

/usr/sbin/ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConnTCP 10.96.0.1:443 rr -> 10.103.22.231:6443 Masq 1 0 0 -> 10.103.22.232:6443 Masq 1 0 0 -> 10.103.22.233:6443 Masq 1 0 0
复制代码

可见将所有到 kubernetes cluster ip 443 端口的请求都转发到 kube-apiserver 的 6443 端口;


7、安装网络插件 calico

使用 etcd 数据存储安装 Calico

1、下载 etcd 的 Calico 网络清单。

curl https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico.yaml
复制代码

2、根据环境调整配置

vim calico.yaml#Secret 对象调整,填入命令获取的值  #etcd-key: (cat /opt/etcd/cert/etcd-key.pem | base64 | tr -d '\n')  #etcd-cert: (cat /opt/etcd/cert/etcd.pem | base64 | tr -d '\n')  #etcd-ca: (cat /opt/kubernetes/cert/ca.pem | base64 | tr -d '\n')
#ConfigMap对象调整data: # Configure this with the location of your etcd cluster. #配置etcd集群的地址 etcd_endpoints: "https://10.103.22.231:2379,https://10.103.22.231:2379,https://10.103.22.231:2379"
#ConfigMap 对象调整,去掉注释 etcd_ca: "/calico-secrets/etcd-ca" # "/calico-secrets/etcd-ca" etcd_cert: "/calico-secrets/etcd-cert" # "/calico-secrets/etcd-cert" etcd_key: "/calico-secrets/etcd-key" # "/calico-secrets/etcd-key" #DaemonSet 对象调整 - name: CALICO_IPV4POOL_CIDR value: "10.98.0.0/16" #此处会默认拿kuber-apiservice的service的clusterip地址 #也可以用apiserver的vip地址(keepalived的vip地址,haproxy代理) #- name: KUBERNETES_SERVICE_HOST # value: "10.103.22.236" #- name: KUBERNETES_SERVICE_PORT # value: "8443" #- name: KUBERNETES_SERVICE_PORT_HTTPS # value: "8443"
#安装calicokubectl apply -f calico.yaml
#查看calicokubectl get pods -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScalico-kube-controllers-f64c4b8fb-8xmhh 1/1 Running 0 103m 10.103.22.232 master02 <none> <none>calico-node-cwlpc 1/1 Running 0 103m 10.103.22.235 node05 <none> <none>calico-node-dlvhm 1/1 Running 0 103m 10.103.22.234 node04 <none> <none>calico-node-kmcc8 1/1 Running 0 103m 10.103.22.233 master03 <none> <none>calico-node-trbrn 1/1 Running 0 103m 10.103.22.231 master01 <none> <none>calico-node-xmnl7 1/1 Running 0 103m 10.103.22.232 master02 <none> <none>
复制代码

8、安装 coredns

修改配置文件

wget https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/coredns/coredns.yaml.inmv coredns.yaml.in coredns.yaml#根据自己的环境调整配置Deployment 对象replicas: 2
Service对象spec: selector: k8s-app: kube-dns clusterIP: 10.96.0.2 #安装corednskubectl apply -f coredns.yaml #查看corednskubectl get pods -n kube-system |grep core coredns-55fb8c8645-cr8xz 1/1 Running 0 4h19mcoredns-55fb8c8645-w5bkt 1/1 Running 0 4h19m
复制代码

9、验证集群功能

检查节点状态

kubectl get nodesNAME       STATUS   ROLES    AGE    VERSIONmaster01   Ready    <none>   2d1h   v1.17.8master02   Ready    <none>   2d1h   v1.17.8master03   Ready    <none>   2d1h   v1.17.8node04     Ready    <none>   2d1h   v1.17.8node05     Ready    <none>   2d1h   v1.17.8
复制代码

创建测试文件

mkdir -p /root/manifests/test#编辑文件vim /root/manifests/test/nginx-ds.ymlapiVersion: apps/v1kind: Deploymentmetadata:  name: nginxspec:  replicas: 2  selector:    matchLabels:      app: nginx  template:    metadata:      name: nginx      labels:        app: nginx    spec:      nodeSelector:        node: master      containers:      - name: nginx        image: nginx---apiVersion: v1kind: Servicemetadata:  name: nginx-service  labels:    app: nginxspec:  type: NodePort  selector:    app: nginx  ports:  - name: http    port: 80    targetPort: 80
复制代码

执行文件并查看结果

kubectl apply -f /root/manifests/test/nginx-ds.yml#查看podkubectl get pods -o widenginx-97c9d49cb-b5zgq    1/1     Running   0          3m18s   10.98.235.2    master03   <none>           <none>nginx-97c9d49cb-t4zj4    1/1     Running   0          3m18s   10.98.241.69   master01   <none>           <none>#查看servicekubectl get svcNAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGEnginx-service   NodePort    10.96.3.235   <none>        80:31637/TCP   10m#查看ipvs规则ipvsadm -lnTCP  10.96.3.235:80 rr  -> 10.98.235.2:80               Masq    1      0          0           -> 10.98.241.69:80              Masq    1      0          0         #和上面pod的ip对应
复制代码


用户头像

小小文

关注

还未添加个人签名 2018.02.06 加入

还未添加个人简介

评论

发布
暂无评论
kubernetes集群安装(二进制)