云原生（二十八） | Kubernetes 篇之自建高可用 k8s 集群搭建

2022 年 8 月 28 日
广东
本文字数：22932 字
阅读完需：约 75 分钟

自建高可用 k8s 集群搭建

一、所有节点基础环境

192.168.0.x ：为机器的网段 10.96.0.0/16: 为 Service 网段 196.16.0.0/16: 为 Pod 网段

1、环境准备与内核升级

先升级所有机器内核

#我的机器版本cat /etc/redhat-release # CentOS Linux release 7.9.2009 (Core)#修改域名，一定不是localhosthostnamectl set-hostname k8s-xxx

#集群规划k8s-master1  k8s-master2  k8s-master3 k8s-master-lb k8s-node01  k8s-node02 ... k8s-nodeN
# 每个机器准备域名vi /etc/hosts192.168.0.10 k8s-master1192.168.0.11 k8s-master2192.168.0.12 k8s-master3192.168.0.13 k8s-node1192.168.0.14 k8s-node2192.168.0.15 k8s-node3192.168.0.250 k8s-master-lb # 非高可用，可以不用这个。这个使用keepalive配置

复制代码

# 关闭selinuxsetenforce 0sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinuxsed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config

复制代码

# 关闭swapswapoff -a && sysctl -w vm.swappiness=0sed -ri 's/.*swap.*/#&/' /etc/fstab

复制代码

#修改limitulimit -SHn 65535vi /etc/security/limits.conf# 末尾添加如下内容* soft nofile 655360* hard nofile 131072* soft nproc 655350* hard nproc 655350* soft memlock unlimited* hard memlock unlimited

复制代码

#为了方便以后操作配置ssh免密连接，master1运行ssh-keygen -t rsa
for i in k8s-master1 k8s-master2 k8s-master3 k8s-node1 k8s-node2 k8s-node3;do ssh-copy-id -i .ssh/id_rsa.pub $i;done

复制代码

#安装后续用的一些工具yum install wget git jq psmisc net-tools yum-utils device-mapper-persistent-data lvm2  -y

复制代码

# 所有节点# 安装ipvs工具,方便以后操作ipvs，ipset，conntrack等yum install ipvsadm ipset sysstat conntrack libseccomp -y# 所有节点配置ipvs模块,执行以下命令，在内核4.19+版本改为nf_conntrack， 4.18下改为nf_conntrack_ipv4modprobe -- ip_vsmodprobe -- ip_vs_rrmodprobe -- ip_vs_wrrmodprobe -- ip_vs_shmodprobe -- nf_conntrack
#修改ipvs配置，加入以下内容vi /etc/modules-load.d/ipvs.conf
ip_vsip_vs_lcip_vs_wlcip_vs_rrip_vs_wrrip_vs_lblcip_vs_lblcrip_vs_dhip_vs_ship_vs_foip_vs_nqip_vs_sedip_vs_ftpip_vs_shnf_conntrackip_tablesip_setxt_setipt_setipt_rpfilteript_REJECTipip
# 执行命令systemctl enable --now systemd-modules-load.service  #--now = enable+start
#检测是否加载lsmod | grep -e ip_vs -e nf_conntrack

复制代码

## 所有节点cat <<EOF > /etc/sysctl.d/k8s.confnet.ipv4.ip_forward = 1net.bridge.bridge-nf-call-iptables = 1net.bridge.bridge-nf-call-ip6tables = 1fs.may_detach_mounts = 1vm.overcommit_memory=1net.ipv4.conf.all.route_localnet = 1
vm.panic_on_oom=0fs.inotify.max_user_watches=89100fs.file-max=52706963fs.nr_open=52706963net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600net.ipv4.tcp_keepalive_probes = 3net.ipv4.tcp_keepalive_intvl =15net.ipv4.tcp_max_tw_buckets = 36000net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_max_orphans = 327680net.ipv4.tcp_orphan_retries = 3net.ipv4.tcp_syncookies = 1net.ipv4.tcp_max_syn_backlog = 16768net.ipv4.ip_conntrack_max = 65536net.ipv4.tcp_timestamps = 0net.core.somaxconn = 16768EOFsysctl --system

复制代码

# 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载rebootlsmod | grep -e ip_vs -e nf_conntrack

复制代码

2、安装 Docker

# 安装dockeryum remove docker*yum install -y yum-utilsyum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repoyum install -y docker-ce-19.03.9  docker-ce-cli-19.03.9 containerd.io-1.4.4

复制代码

# 修改docker配置,新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemdmkdir /etc/dockercat > /etc/docker/daemon.json <<EOF{  "exec-opts": ["native.cgroupdriver=systemd"],  "registry-mirrors": ["https://82m9ayutr63.mirror.aliyuncs.com"]}EOFsystemctl daemon-reload && systemctl enable --now docker

复制代码

#也可以自己下载rpm离线包进行安装http://mirrors.aliyun.com/docker-ce/linux/centos/7.9/x86_64/stable/Packages/yum localinstall xxxx

复制代码

二、PKI

百度百科：公钥基础设施_百度百科

Kubernetes 需要 PKI 才能执行以下操作：

Kubelet 的客户端证书，用于 API 服务器身份验证
API 服务器端点的证书
集群管理员的客户端证书，用于 API 服务器身份认证
API 服务器的客户端证书，用于和 Kubelet 的会话
API 服务器的客户端证书，用于和 etcd 的会话
控制器管理器的客户端证书/kubeconfig，用于和 API 服务器的会话
调度器的客户端证书/kubeconfig，用于和 API 服务器的会话
前端代理的客户端及服务端证书

说明： 只有当你运行 kube-proxy 并要支持扩展 API 服务器时，才需要 front-proxy 证书

etcd 还实现了双向 TLS 来对客户端和对其他对等节点进行身份验证

PKI 证书和要求 | Kubernetes

三、证书工具准备

# 准备文件夹存放所有证书信息。看看kubeadm 如何组织有序的结构的# 三个节点都执行mkdir -p /etc/kubernetes/pki

复制代码

1、下载证书工具

# 下载cfssl核心组件wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64wget https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64
#授予执行权限chmod +x cfssl*
#批量重命名for name in `ls cfssl*`; do mv $name ${name%_1.5.0_linux_amd64};  done
#移动到文件mv cfssl* /usr/bin

复制代码

2、ca 根配置

ca-config.json

mkdir -p /etc/kubernetes/pkicd /etc/kubernetes/pkivi ca-config.json

复制代码

{    "signing": {        "default": {            "expiry": "87600h"        },        "profiles": {            "server": {                "expiry": "87600h",                "usages": [                    "signing",                    "key encipherment",                    "server auth"                ]            },            "client": {                "expiry": "87600h",                "usages": [                    "signing",                    "key encipherment",                    "client auth"                ]            },            "peer": {                "expiry": "87600h",                "usages": [                    "signing",                    "key encipherment",                    "server auth",                    "client auth"                ]            },            "kubernetes": {                "expiry": "87600h",                "usages": [                    "signing",                    "key encipherment",                    "server auth",                    "client auth"                ]            },            "etcd": {                "expiry": "87600h",                "usages": [                    "signing",                    "key encipherment",                    "server auth",                    "client auth"                ]            }        }    }}

复制代码

3、ca 签名请求

CSR 是 Certificate Signing Request 的英文缩写，即证书签名请求文件

ca-csr.json

vi /etc/kubernetes/pki/ca-csr.json

复制代码

{  "CN": "kubernetes",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "Beijing",      "L": "Beijing",      "O": "Kubernetes",      "OU": "Kubernetes"    }  ],  "ca": {    "expiry": "87600h"  }}

复制代码

CN(Common Name):

-  公用名（Common Name）必须填写，一般可以是网站域

复制代码

O(Organization):

-  Organization（组织名）是必须填写的，如果申请的是OV、EV型证书，组织名称必须严格和企业在政府登记名称一致，一般需要和营业执照上的名称完全一致。不可以使用缩写或者商标。如果需要使用英文名称，需要有DUNS编码或者律师信证明。

复制代码

OU(Organization Unit)

-  OU单位部门，这里一般没有太多限制，可以直接填写IT DEPT等皆可。

复制代码

C(City)

-  City是指申请单位所在的城市。

复制代码

ST(State/Province)

-  ST是指申请单位所在的省份。

复制代码

C(Country Name）

-  C是指国家名称，这里用的是两位大写的国家代码，中国是CN。

复制代码

4、生成证书

生成 ca 证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -# ca.csr ca.pem(ca公钥) ca-key.pem(ca私钥,妥善保管)

复制代码

5、k8s 集群是如何使用证书的

参考官方文档：PKI 证书和要求 | Kubernetes

四、etcd 高可用搭建

1、etcd 文档

etcd 示例：Demo | etcd 参照示例学习 etcd 使用

etcd 构建：Install | etcd 参照 etcd-k8s 集群量规划指南，大家参照这个标准建立集群

etcd 部署：Operations guide | etcd 参照部署手册，学习 etcd 配置和集群部署

2、下载 etcd

# 给所有master节点，发送etcd包准备部署etcd高可用wget https://github.com/etcd-io/etcd/releases/download/v3.4.16/etcd-v3.4.16-linux-amd64.tar.gz
## 到其他节点for i in k8s-master1 k8s-master2 k8s-master3;do scp etcd-* root@$i:/root/;done

## 解压到 /usr/local/bintar -zxvf etcd-v3.4.16-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.4.16-linux-amd64/etcd{,ctl}

##验证etcdctl #只要有打印就ok

复制代码

3、etcd 证书

Hardware recommendations | etcd安装参考：Hardware recommendations | etcd

生成 etcd 证书

etcd-ca-csr.json

{  "CN": "etcd",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "Beijing",      "L": "Beijing",      "O": "etcd",      "OU": "etcd"    }  ],  "ca": {    "expiry": "87600h"  }}

复制代码

# 生成etcd根ca证书cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/etcd/ca -

复制代码

etcd-itdachang-csr.json

{    "CN": "etcd-itdachang",    "key": {        "algo": "rsa",        "size": 2048    },    "hosts": [          "127.0.0.1",        "k8s-master1",        "k8s-master2",        "k8s-master3",        "192.168.0.10",        "192.168.0.11",        "192.168.0.12"    ],    "names": [        {            "C": "CN",            "L": "beijing",            "O": "etcd",            "ST": "beijing",            "OU": "System"        }    ]}
// 注意：hosts用自己的主机名和ip// 也可以在签发的时候再加上 -hostname=127.0.0.1,k8s-master1,k8s-master2,k8s-master3,// 可以指定受信的主机列表//    "hosts": [//        "k8s-master1",//        "www.example.net"//    ],

复制代码

# 签发itdachang的etcd证书cfssl gencert \   -ca=/etc/kubernetes/pki/etcd/ca.pem \   -ca-key=/etc/kubernetes/pki/etcd/ca-key.pem \   -config=/etc/kubernetes/pki/ca-config.json \   -profile=etcd \   etcd-itdachang-csr.json | cfssljson -bare /etc/kubernetes/pki/etcd/etcd

复制代码

# 把生成的etcd证书，给其他机器
for i in k8s-master2 k8s-master3;do scp -r /etc/kubernetes/pki/etcd root@$i:/etc/kubernetes/pki;done

复制代码

4、etcd 高可用安装

etcd 配置文件示例： Configuration flags | etcd

etcd 高可用安装示例： Clustering Guide | etcd

为了保证启动配置一致性，我们编写 etcd 配置文件，并将 etcd 做成 service 启动

# etcd yaml示例。# This is the configuration file for the etcd server.
# Human-readable name for this member.name: 'default'# Path to the data directory.data-dir:# Path to the dedicated wal directory.wal-dir:# Number of committed transactions to trigger a snapshot to disk.snapshot-count: 10000# Time (in milliseconds) of a heartbeat interval.heartbeat-interval: 100# Time (in milliseconds) for an election to timeout.election-timeout: 1000# Raise alarms when backend size exceeds the given quota. 0 means use the# default quota.quota-backend-bytes: 0# List of comma separated URLs to listen on for peer traffic.listen-peer-urls: http://localhost:2380# List of comma separated URLs to listen on for client traffic.listen-client-urls: http://localhost:2379# Maximum number of snapshot files to retain (0 is unlimited).max-snapshots: 5# Maximum number of wal files to retain (0 is unlimited).max-wals: 5# Comma-separated white list of origins for CORS (cross-origin resource sharing).cors:# List of this member's peer URLs to advertise to the rest of the cluster.# The URLs needed to be a comma-separated list.initial-advertise-peer-urls: http://localhost:2380# List of this member's client URLs to advertise to the public.# The URLs needed to be a comma-separated list.advertise-client-urls: http://localhost:2379# Discovery URL used to bootstrap the cluster.discovery:# Valid values include 'exit', 'proxy'discovery-fallback: 'proxy'# HTTP proxy to use for traffic to discovery service.discovery-proxy:# DNS domain used to bootstrap initial cluster.discovery-srv:# Initial cluster configuration for bootstrapping.initial-cluster:# Initial cluster token for the etcd cluster during bootstrap.initial-cluster-token: 'etcd-cluster'# Initial cluster state ('new' or 'existing').initial-cluster-state: 'new'# Reject reconfiguration requests that would cause quorum loss.strict-reconfig-check: false# Accept etcd V2 client requestsenable-v2: true# Enable runtime profiling data via HTTP serverenable-pprof: true# Valid values include 'on', 'readonly', 'off'proxy: 'off'# Time (in milliseconds) an endpoint will be held in a failed state.proxy-failure-wait: 5000# Time (in milliseconds) of the endpoints refresh interval.proxy-refresh-interval: 30000# Time (in milliseconds) for a dial to timeout.proxy-dial-timeout: 1000# Time (in milliseconds) for a write to timeout.proxy-write-timeout: 5000# Time (in milliseconds) for a read to timeout.proxy-read-timeout: 0client-transport-security:  # Path to the client server TLS cert file.  cert-file:  # Path to the client server TLS key file.  key-file:  # Enable client cert authentication.  client-cert-auth: false  # Path to the client server TLS trusted CA cert file.  trusted-ca-file:  # Client TLS using generated certificates  auto-tls: falsepeer-transport-security:  # Path to the peer server TLS cert file.  cert-file:  # Path to the peer server TLS key file.  key-file:  # Enable peer client cert authentication.  client-cert-auth: false  # Path to the peer server TLS trusted CA cert file.  trusted-ca-file:  # Peer TLS using generated certificates.  auto-tls: false# Enable debug-level logging for etcd.debug: falselogger: zap# Specify 'stdout' or 'stderr' to skip journald logging even when running under systemd.log-outputs: [stderr]# Force to create a new one member cluster.force-new-cluster: falseauto-compaction-mode: periodicauto-compaction-retention: "1"

复制代码

三个 etcd 机器都创建 /etc/etcd 目录，准备存储 etcd 配置信息 undefined

#三个master执行mkdir -p /etc/etcd

复制代码

vi /etc/etcd/etcd.yaml

复制代码

# 我们的yamlname: 'etcd-master3'  #每个机器可以写自己的域名,不能重复data-dir: /var/lib/etcdwal-dir: /var/lib/etcd/walsnapshot-count: 5000heartbeat-interval: 100election-timeout: 1000quota-backend-bytes: 0listen-peer-urls: 'https://192.168.0.12:2380'  # 本机ip+2380端口，代表和集群通信listen-client-urls: 'https://192.168.0.12:2379,http://127.0.0.1:2379' #改为自己的max-snapshots: 3max-wals: 5cors:initial-advertise-peer-urls: 'https://192.168.0.12:2380' #自己的ipadvertise-client-urls: 'https://192.168.0.12:2379'  #自己的ipdiscovery:discovery-fallback: 'proxy'discovery-proxy:discovery-srv:initial-cluster: 'etcd-master1=https://192.168.0.10:2380,etcd-master2=https://192.168.0.11:2380,etcd-master3=https://192.168.0.12:2380' #这里不一样initial-cluster-token: 'etcd-k8s-cluster'initial-cluster-state: 'new'strict-reconfig-check: falseenable-v2: trueenable-pprof: trueproxy: 'off'proxy-failure-wait: 5000proxy-refresh-interval: 30000proxy-dial-timeout: 1000proxy-write-timeout: 5000proxy-read-timeout: 0client-transport-security:  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'  client-cert-auth: true  trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.pem'  auto-tls: truepeer-transport-security:  cert-file: '/etc/kubernetes/pki/etcd/etcd.pem'  key-file: '/etc/kubernetes/pki/etcd/etcd-key.pem'  peer-client-cert-auth: true  trusted-ca-file: '/etc/kubernetes/pki/etcd/ca.pem'  auto-tls: truedebug: falselog-package-levels:log-outputs: [default]force-new-cluster: false

复制代码

三台机器的 etcd 做成 service，开机启动 undefined

vi /usr/lib/systemd/system/etcd.service
[Unit]Description=Etcd ServiceDocumentation=https://etcd.io/docs/v3.4/op-guide/clustering/After=network.target
[Service]Type=notifyExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.yamlRestart=on-failureRestartSec=10LimitNOFILE=65536
[Install]WantedBy=multi-user.targetAlias=etcd3.service

复制代码

# 加载&开机启动systemctl daemon-reloadsystemctl enable --now etcd
# 启动有问题,使用 journalctl -u 服务名排查journalctl -u etcd

复制代码

测试 etcd 访问

# 查看etcd集群状态etcdctl --endpoints="192.168.0.10:2379,192.168.0.11:2379,192.168.0.12:2379" --cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem  endpoint status --write-out=table
# 以后测试命令export ETCDCTL_API=3HOST_1=192.168.0.10HOST_2=192.168.0.11HOST_3=192.168.0.12ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379
## 导出环境变量，方便测试，参照https://github.com/etcd-io/etcd/tree/main/etcdctlexport ETCDCTL_DIAL_TIMEOUT=3sexport ETCDCTL_CACERT=/etc/kubernetes/pki/etcd/ca.pemexport ETCDCTL_CERT=/etc/kubernetes/pki/etcd/etcd.pemexport ETCDCTL_KEY=/etc/kubernetes/pki/etcd/etcd-key.pemexport ETCDCTL_ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379# 自动用环境变量定义的证书位置etcdctl  member list --write-out=table
#如果没有环境变量就需要如下方式调用etcdctl --endpoints=$ENDPOINTS --cacert=/etc/kubernetes/pki/etcd/ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem member list --write-out=table

## 更多etcdctl命令，https://etcd.io/docs/v3.4/demo/#access-etcd

复制代码

五、k8s 组件与证书

1、K8s 离线安装包

https://github.com/kubernetes/kubernetes 找到 changelog 对应版本

# 下载k8s包wget https://dl.k8s.io/v1.21.1/kubernetes-server-linux-amd64.tar.gz

复制代码

2、master 节点准备

# 把kubernetes把给master所有节点for i in k8s-master1 k8s-master2 k8s-master3  k8s-node1 k8s-node2 k8s-node3;do scp kubernetes-server-* root@$i:/root/;done

复制代码

#所有master节点解压kubelet，kubectl等到 /usr/local/bin。tar -xvf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

#master需要全部组件，node节点只需要 /usr/local/bin kubelet、kube-proxy

复制代码

3、apiserver 证书生成

3.1、apiserver-csr.json

//10.96.0. 为service网段。可以自定义 如： 66.66.0.1// 192.168.0.250： 是我准备的负载均衡器地址（负载均衡可以自己搭建，也可以购买云厂商lb。）{    "CN": "kube-apiserver",    "hosts": [      "10.96.0.1",      "127.0.0.1",      "192.168.0.250",      "192.168.0.10",      "192.168.0.11",      "192.168.0.12",      "192.168.0.13",      "192.168.0.14",      "192.168.0.15",      "192.168.0.16",      "kubernetes",      "kubernetes.default",      "kubernetes.default.svc",      "kubernetes.default.svc.cluster",      "kubernetes.default.svc.cluster.local"    ],    "key": {        "algo": "rsa",        "size": 2048    },    "names": [        {            "C": "CN",            "L": "BeiJing",            "ST": "BeiJing",            "O": "Kubernetes",            "OU": "Kubernetes"        }    ]}

复制代码

3.2、生成 apiserver 证书

# 192.168.0.是k8s service的网段，如果说需要更改k8s service网段，那就需要更改192.168.0.1，# 如果不是高可用集群，10.103.236.236为Master01的IP#先生成CA机构vi ca-csr.json{  "CN": "kubernetes",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "Beijing",      "L": "Beijing",      "O": "Kubernetes",      "OU": "Kubernetes"    }  ],  "ca": {    "expiry": "87600h"  }}

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert   -ca=/etc/kubernetes/pki/ca.pem   -ca-key=/etc/kubernetes/pki/ca-key.pem   -config=/etc/kubernetes/pki/ca-config.json   -profile=kubernetes   apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver

复制代码

4、front-proxy 证书生成

官方文档：配置聚合层 | Kubernetes

注意：front-proxy 不建议用新的 CA 机构签发证书，可能导致通过他代理的组件如 metrics-server 权限不可用。如果用新的，api-server 配置添加 --requestheader-allowed-names=front-proxy-client

4.1、front-proxy-ca-csr.json

front-proxy 根 ca

vi front-proxy-ca-csr.json{  "CN": "kubernetes",  "key": {     "algo": "rsa",     "size": 2048  }}

复制代码

#front-proxy 根ca生成cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca

复制代码

4.2、front-proxy-client 证书

vi  front-proxy-client-csr.json  #准备申请client客户端

复制代码

{  "CN": "front-proxy-client",  "key": {     "algo": "rsa",     "size": 2048  }}

复制代码

#生成front-proxy-client 证书cfssl gencert   -ca=/etc/kubernetes/pki/front-proxy-ca.pem   -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem   -config=ca-config.json   -profile=kubernetes   front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client
#忽略警告，毕竟我们不是给网站生成的

复制代码

5、controller-manage 证书生成与配置

5.1、controller-manager-csr.json

vi controller-manager-csr.json

复制代码

{  "CN": "system:kube-controller-manager",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "Beijing",      "L": "Beijing",      "O": "system:kube-controller-manager",      "OU": "Kubernetes"    }  ]}

复制代码

5.2、生成证书

cfssl gencert \   -ca=/etc/kubernetes/pki/ca.pem \   -ca-key=/etc/kubernetes/pki/ca-key.pem \   -config=ca-config.json \   -profile=kubernetes \  controller-manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager

复制代码

5.3、生成配置

# 注意，如果不是高可用集群，192.168.0.250:6443改为master01的地址，6443为apiserver的默认端口# set-cluster：设置一个集群项，kubectl config set-cluster kubernetes \     --certificate-authority=/etc/kubernetes/pki/ca.pem \     --embed-certs=true \     --server=https://192.168.0.250:6443 \     --kubeconfig=/etc/kubernetes/controller-manager.conf
# 设置一个环境项，一个上下文kubectl config set-context system:kube-controller-manager@kubernetes \    --cluster=kubernetes \    --user=system:kube-controller-manager \    --kubeconfig=/etc/kubernetes/controller-manager.conf
# set-credentials 设置一个用户项
kubectl config set-credentials system:kube-controller-manager \     --client-certificate=/etc/kubernetes/pki/controller-manager.pem \     --client-key=/etc/kubernetes/pki/controller-manager-key.pem \     --embed-certs=true \     --kubeconfig=/etc/kubernetes/controller-manager.conf

# 使用某个环境当做默认环境
kubectl config use-context system:kube-controller-manager@kubernetes \     --kubeconfig=/etc/kubernetes/controller-manager.conf          # 后来也用来自动批复kubelet证书

复制代码

6、scheduler 证书生成与配置

6.1、scheduler-csr.json

vi scheduler-csr.json

复制代码

{  "CN": "system:kube-scheduler",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "Beijing",      "L": "Beijing",      "O": "system:kube-scheduler",      "OU": "Kubernetes"    }  ]}

复制代码

6.2、签发证书

cfssl gencert \   -ca=/etc/kubernetes/pki/ca.pem \   -ca-key=/etc/kubernetes/pki/ca-key.pem \   -config=/etc/kubernetes/pki/ca-config.json \   -profile=kubernetes \   scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler

复制代码

6.3、生成配置

# 注意，如果不是高可用集群，192.168.0.250:6443 改为master01的地址，6443是api-server默认端口
kubectl config set-cluster kubernetes \     --certificate-authority=/etc/kubernetes/pki/ca.pem \     --embed-certs=true \     --server=https://192.168.0.250:6443 \     --kubeconfig=/etc/kubernetes/scheduler.conf

kubectl config set-credentials system:kube-scheduler \     --client-certificate=/etc/kubernetes/pki/scheduler.pem \     --client-key=/etc/kubernetes/pki/scheduler-key.pem \     --embed-certs=true \     --kubeconfig=/etc/kubernetes/scheduler.conf
kubectl config set-context system:kube-scheduler@kubernetes \     --cluster=kubernetes \     --user=system:kube-scheduler \     --kubeconfig=/etc/kubernetes/scheduler.conf

kubectl config use-context system:kube-scheduler@kubernetes \     --kubeconfig=/etc/kubernetes/scheduler.conf
#k8s集群安全操作相关

复制代码

7、admin 证书生成与配置

7.1、admin-csr.json

vi admin-csr.json

复制代码

{  "CN": "admin",  "key": {    "algo": "rsa",    "size": 2048  },  "names": [    {      "C": "CN",      "ST": "Beijing",      "L": "Beijing",      "O": "system:masters",      "OU": "Kubernetes"    }  ]}

复制代码

7.2、生成证书

cfssl gencert \   -ca=/etc/kubernetes/pki/ca.pem \   -ca-key=/etc/kubernetes/pki/ca-key.pem \   -config=/etc/kubernetes/pki/ca-config.json \   -profile=kubernetes \   admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin

复制代码

7.3、生成配置

# 注意，如果不是高可用集群，192.168.0.250:6443改为master01的地址，6443为apiserver的默认端口kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=https://192.168.0.250:6443 \--kubeconfig=/etc/kubernetes/admin.conf

kubectl config set-credentials kubernetes-admin \--client-certificate=/etc/kubernetes/pki/admin.pem \--client-key=/etc/kubernetes/pki/admin-key.pem \--embed-certs=true \--kubeconfig=/etc/kubernetes/admin.conf

kubectl config set-context kubernetes-admin@kubernetes \--cluster=kubernetes \--user=kubernetes-admin \--kubeconfig=/etc/kubernetes/admin.conf
kubectl config use-context kubernetes-admin@kubernetes \--kubeconfig=/etc/kubernetes/admin.conf

复制代码

kubelet 将使用 bootstrap 引导机制，自动颁发证书，所以我们不用配置了。要不然，1 万台机器，一个万 kubelet，证书配置到明年去。。。

8、ServiceAccount Key 生成

k8s 底层，每创建一个 ServiceAccount，都会分配一个 Secret，而 Secret 里面有秘钥，秘钥就是由我们接下来的 sa 生成的。所以我们提前创建出 sa 信息

openssl genrsa -out /etc/kubernetes/pki/sa.key 2048
openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub

复制代码

9、发送证书到其他节点

# 在master1上执行for NODE in k8s-master2 k8s-master3do    for FILE in admin.conf controller-manager.conf scheduler.conf    do    scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}    donedone

复制代码

六、高可用配置

高可用配置

-  如果你不是在创建高可用集群，则无需配置haproxy和keepalived

复制代码

-  高可用有很多可选方案

复制代码

    -  nginx

复制代码

    -  haproxy

复制代码

    -  keepalived

复制代码

    -  云供应商提供的负载均衡产品

复制代码

云上安装注意事项

-  云上安装可以直接使用云上的lb，比如阿里云slb，腾讯云elb等

复制代码

-  公有云要用公有云自带的负载均衡，比如阿里云的SLB，腾讯云的ELB，用来替代haproxy和keepalived，因为公有云大部分都是不支持keepalived的。

复制代码

-  阿里云的话，kubectl控制端不能放在master节点，推荐使用腾讯云，因为阿里云的slb有回环的问题，也就是slb代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题。

复制代码

青云使用

-  创建负载均衡器，指定ip地址为我们之前的预留地址

复制代码

-  进入负载均衡器，创建监听器

复制代码

-  选择TCP，6443端口

复制代码

-  添加后端服务器地址与端口

复制代码

七、组件启动

1、所有 master 执行

mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes

#三个master节点kube-xx相关的程序都在 /usr/local/binfor NODE in k8s-master2 k8s-master3do    scp -r /etc/kubernetes/* root@$NODE:/etc/kubernetes/done

复制代码

接下来把 master1 生成的所有证书全部发给 master2,master3

2、配置 apiserver 服务

2.1、配置

所有 Master 节点创建kube-apiserver.service

注意，如果不是高可用集群，192.168.0.250 改为 master01 的地址以下文档使用的 k8s service 网段为10.96.0.0/16，该网段不能和宿主机的网段、Pod 网段的重复特别注意：docker 的网桥默认为 172.17.0.1/16。不要使用这个网段

# 每个master节点都需要执行以下内容# --advertise-address： 需要改为本master节点的ip# --service-cluster-ip-range=10.96.0.0/16： 需要改为自己规划的service网段# --etcd-servers： 改为自己etcd-server的所有地址
vi /usr/lib/systemd/system/kube-apiserver.service
[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/kubernetes/kubernetesAfter=network.target
[Service]ExecStart=/usr/local/bin/kube-apiserver \      --v=2  \      --logtostderr=true  \      --allow-privileged=true  \      --bind-address=0.0.0.0  \      --secure-port=6443  \      --insecure-port=0  \      --advertise-address=192.168.0.10 \      --service-cluster-ip-range=10.96.0.0/16  \      --service-node-port-range=30000-32767  \      --etcd-servers=https://192.168.0.10:2379,https://192.168.0.11:2379,https://192.168.0.12:2379 \      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.pem  \      --etcd-certfile=/etc/kubernetes/pki/etcd/etcd.pem  \      --etcd-keyfile=/etc/kubernetes/pki/etcd/etcd-key.pem  \      --client-ca-file=/etc/kubernetes/pki/ca.pem  \      --tls-cert-file=/etc/kubernetes/pki/apiserver.pem  \      --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem  \      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem  \      --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem  \      --service-account-key-file=/etc/kubernetes/pki/sa.pub  \      --service-account-signing-key-file=/etc/kubernetes/pki/sa.key  \      --service-account-issuer=https://kubernetes.default.svc.cluster.local \      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname  \      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota  \      --authorization-mode=Node,RBAC  \      --enable-bootstrap-token-auth=true  \      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem  \      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem  \      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem  \      --requestheader-allowed-names=aggregator,front-proxy-client  \      --requestheader-group-headers=X-Remote-Group  \      --requestheader-extra-headers-prefix=X-Remote-Extra-  \      --requestheader-username-headers=X-Remote-User      # --token-auth-file=/etc/kubernetes/token.csv
Restart=on-failureRestartSec=10sLimitNOFILE=65535
[Install]WantedBy=multi-user.target

复制代码

2.2、启动 apiserver 服务

systemctl daemon-reload && systemctl enable --now kube-apiserver

#查看状态systemctl status kube-apiserver

复制代码

3、配置 controller-manager 服务

3.1、配置

所有 Master 节点配置 kube-controller-manager.service

文档使用的 k8s Pod 网段为196.16.0.0/16，该网段不能和宿主机的网段、k8s Service 网段的重复，请按需修改; 特别注意：docker 的网桥默认为 172.17.0.1/16。不要使用这个网

# 所有节点执行vi /usr/lib/systemd/system/kube-controller-manager.service
## --cluster-cidr=196.16.0.0/16 ： 为Pod的网段。修改成自己想规划的网段
[Unit]Description=Kubernetes Controller ManagerDocumentation=https://github.com/kubernetes/kubernetesAfter=network.target
[Service]ExecStart=/usr/local/bin/kube-controller-manager \      --v=2 \      --logtostderr=true \      --address=127.0.0.1 \      --root-ca-file=/etc/kubernetes/pki/ca.pem \      --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \      --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \      --service-account-private-key-file=/etc/kubernetes/pki/sa.key \      --kubeconfig=/etc/kubernetes/controller-manager.conf \      --leader-elect=true \      --use-service-account-credentials=true \      --node-monitor-grace-period=40s \      --node-monitor-period=5s \      --pod-eviction-timeout=2m0s \      --controllers=*,bootstrapsigner,tokencleaner \      --allocate-node-cidrs=true \      --cluster-cidr=196.16.0.0/16 \      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \      --node-cidr-mask-size=24      Restart=alwaysRestartSec=10s
[Install]WantedBy=multi-user.target

复制代码

3.2、启动

# 所有master节点执行systemctl daemon-reload
systemctl daemon-reload && systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager

复制代码

4、配置 scheduler

4.1、配置

所有 Master 节点配置 kube-scheduler.service

vi /usr/lib/systemd/system/kube-scheduler.service 


[Unit]Description=Kubernetes SchedulerDocumentation=https://github.com/kubernetes/kubernetesAfter=network.target
[Service]ExecStart=/usr/local/bin/kube-scheduler \      --v=2 \      --logtostderr=true \      --address=127.0.0.1 \      --leader-elect=true \      --kubeconfig=/etc/kubernetes/scheduler.conf
Restart=alwaysRestartSec=10s
[Install]WantedBy=multi-user.target

复制代码

4.2、启动

systemctl daemon-reload
systemctl daemon-reload && systemctl enable --now kube-scheduler
systemctl status kube-scheduler

复制代码

八、TLS 与引导启动原理

1、master1 配置 bootstrap

注意，如果不是高可用集群，192.168.0.250:6443改为 master1 的地址，6443 为 apiserver 的默认端口

#准备一个随机token。但是我们只需要16个字符head -c 16 /dev/urandom | od -An -t x | tr -d ' '# 值如下： 737b177d9823531a433e368fcdb16f5f
# 生成16个字符的head -c 8 /dev/urandom | od -An -t x | tr -d ' '# d683399b7a553977

复制代码

#设置集群kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=https://192.168.0.250:6443 \--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
#设置秘钥kubectl config set-credentials tls-bootstrap-token-user \--token=l6fy8c.d683399b7a553977 \--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf 
#设置上下文kubectl config set-context tls-bootstrap-token-user@kubernetes \--cluster=kubernetes \--user=tls-bootstrap-token-user \--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf
#使用设置kubectl config use-context tls-bootstrap-token-user@kubernetes \--kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf

复制代码

2、master1 设置 kubectl 执行权限

kubectl 能不能操作集群是看 /root/.kube 下有没有 config 文件，而 config 就是我们之前生成的 admin.conf，具有操作权限的

# 只在master1生成，因为生产集群，我们只能让一台机器具有操作集群的权限，这样好控制
mkdir -p /root/.kube ;cp /etc/kubernetes/admin.conf /root/.kube/config

复制代码

#验证kubectl get nodes
# 应该在网络里面开放负载均衡器的6443端口;默认应该不要配置的[root@k8s-master1 ~]# kubectl get nodesNo resources found#说明已经可以连接apiserver并获取资源

复制代码

3、创建集群引导权限文件

# master准备这个文件 vi  /etc/kubernetes/bootstrap.secret.yaml


apiVersion: v1kind: Secretmetadata:  name: bootstrap-token-l6fy8c  namespace: kube-systemtype: bootstrap.kubernetes.io/tokenstringData:  description: "The default bootstrap token generated by 'kubelet '."  token-id: l6fy8c  token-secret: d683399b7a553977  usage-bootstrap-authentication: "true"  usage-bootstrap-signing: "true"  auth-extra-groups:  system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress ---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: kubelet-bootstraproleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:node-bootstrappersubjects:- apiGroup: rbac.authorization.k8s.io  kind: Group  name: system:bootstrappers:default-node-token---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: node-autoapprove-bootstraproleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:certificates.k8s.io:certificatesigningrequests:nodeclientsubjects:- apiGroup: rbac.authorization.k8s.io  kind: Group  name: system:bootstrappers:default-node-token---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: node-autoapprove-certificate-rotationroleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclientsubjects:- apiGroup: rbac.authorization.k8s.io  kind: Group  name: system:nodes---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRolemetadata:  annotations:    rbac.authorization.kubernetes.io/autoupdate: "true"  labels:    kubernetes.io/bootstrapping: rbac-defaults  name: system:kube-apiserver-to-kubeletrules:  - apiGroups:      - ""    resources:      - nodes/proxy      - nodes/stats      - nodes/log      - nodes/spec      - nodes/metrics    verbs:      - "*"---apiVersion: rbac.authorization.k8s.io/v1kind: ClusterRoleBindingmetadata:  name: system:kube-apiserver  namespace: ""roleRef:  apiGroup: rbac.authorization.k8s.io  kind: ClusterRole  name: system:kube-apiserver-to-kubeletsubjects:  - apiGroup: rbac.authorization.k8s.io    kind: User    name: kube-apiserver

复制代码

# 应用此文件资源内容kubectl create -f /etc/kubernetes/bootstrap.secret.yaml

复制代码

九、引导 Node 节点启动

所有节点的 kubelet 需要我们引导启动

1、发送核心证书到节点

master1 节点把核心证书发送到其他节点

cd /etc/kubernetes/  #查看所有信息
#执行所有令牌操作
for NODE in k8s-master2 k8s-master3 k8s-node1 k8s-node2; do     ssh $NODE mkdir -p /etc/kubernetes/pki/etcd     for FILE in ca.pem etcd.pem etcd-key.pem; do       scp /etc/kubernetes/pki/etcd/$FILE $NODE:/etc/kubernetes/pki/etcd/     done     for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.conf; do       scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE} done done

复制代码

2、所有节点配置 kubelet

# 所有节点创建相关目录mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/
## 所有node节点必须有 kubelet kube-proxyfor NODE in k8s-master2 k8s-master3 k8s-node3 k8s-node1 k8s-node2; do     scp -r /etc/kubernetes/* root@$NODE:/etc/kubernetes/ done

复制代码

2.1、创建 kubelet.service

#所有节点，配置kubelet服务
vi  /usr/lib/systemd/system/kubelet.service
[Unit]Description=Kubernetes KubeletDocumentation=https://github.com/kubernetes/kubernetesAfter=docker.serviceRequires=docker.service
[Service]ExecStart=/usr/local/bin/kubelet
Restart=alwaysStartLimitInterval=0RestartSec=10
[Install]WantedBy=multi-user.target

复制代码

# 所有节点配置kubelet service配置文件vi /etc/systemd/system/kubelet.service.d/10-kubelet.conf
[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"Environment="KUBELET_SYSTEM_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/pause:3.4.1"Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "ExecStart=ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS

复制代码

2.2、创建 kubelet-conf.yml 文件

#所有节点，配置kubelet-conf文件vi /etc/kubernetes/kubelet-conf.yml# clusterDNS 为service网络的第10个ip值,改成自己的。如：10.96.0.10

复制代码

apiVersion: kubelet.config.k8s.io/v1beta1kind: KubeletConfigurationaddress: 0.0.0.0port: 10250readOnlyPort: 10255authentication:  anonymous:    enabled: false  webhook:    cacheTTL: 2m0s    enabled: true  x509:    clientCAFile: /etc/kubernetes/pki/ca.pemauthorization:  mode: Webhook  webhook:    cacheAuthorizedTTL: 5m0s    cacheUnauthorizedTTL: 30scgroupDriver: systemdcgroupsPerQOS: trueclusterDNS:- 10.96.0.10clusterDomain: cluster.localcontainerLogMaxFiles: 5containerLogMaxSize: 10MicontentType: application/vnd.kubernetes.protobufcpuCFSQuota: truecpuManagerPolicy: nonecpuManagerReconcilePeriod: 10senableControllerAttachDetach: trueenableDebuggingHandlers: trueenforceNodeAllocatable:- podseventBurst: 10eventRecordQPS: 5evictionHard:  imagefs.available: 15%  memory.available: 100Mi  nodefs.available: 10%  nodefs.inodesFree: 5%evictionPressureTransitionPeriod: 5m0s  #缩小相应的配置failSwapOn: truefileCheckFrequency: 20shairpinMode: promiscuous-bridgehealthzBindAddress: 127.0.0.1healthzPort: 10248httpCheckFrequency: 20simageGCHighThresholdPercent: 85imageGCLowThresholdPercent: 80imageMinimumGCAge: 2m0siptablesDropBit: 15iptablesMasqueradeBit: 14kubeAPIBurst: 10kubeAPIQPS: 5makeIPTablesUtilChains: truemaxOpenFiles: 1000000maxPods: 110nodeStatusUpdateFrequency: 10soomScoreAdj: -999podPidsLimit: -1registryBurst: 10registryPullQPS: 5resolvConf: /etc/resolv.confrotateCertificates: trueruntimeRequestTimeout: 2m0sserializeImagePulls: truestaticPodPath: /etc/kubernetes/manifestsstreamingConnectionIdleTimeout: 4h0m0ssyncFrequency: 1m0svolumeStatsAggPeriod: 1m0s

复制代码

2.3、所有节点启动 kubelet

systemctl daemon-reload && systemctl enable --now kubelet

systemctl status kubelet

复制代码

会提示 "Unable to update cni config"。接下来配置 cni 网络即可

3、kube-proxy 配置

注意，如果不是高可用集群，192.168.0.250:6443改为 master1 的地址，6443 改为 apiserver 的默认端口

3.1、生成 kube-proxy.conf

以下操作在 master1 执行

#创建kube-proxy的sakubectl -n kube-system create serviceaccount kube-proxy
#创建角色绑定kubectl create clusterrolebinding system:kube-proxy \--clusterrole system:node-proxier \--serviceaccount kube-system:kube-proxy
#导出变量，方便后面使用SECRET=$(kubectl -n kube-system get sa/kube-proxy --output=jsonpath='{.secrets[0].name}')JWT_TOKEN=$(kubectl -n kube-system get secret/$SECRET --output=jsonpath='{.data.token}' | base64 -d)PKI_DIR=/etc/kubernetes/pkiK8S_DIR=/etc/kubernetes
# 生成kube-proxy配置# --server: 指定自己的apiserver地址或者lb地址kubectl config set-cluster kubernetes \--certificate-authority=/etc/kubernetes/pki/ca.pem \--embed-certs=true \--server=https://192.168.0.250:6443 \--kubeconfig=${K8S_DIR}/kube-proxy.conf
# kube-proxy秘钥设置kubectl config set-credentials kubernetes \--token=${JWT_TOKEN} \--kubeconfig=/etc/kubernetes/kube-proxy.conf

kubectl config set-context kubernetes \--cluster=kubernetes \--user=kubernetes \--kubeconfig=/etc/kubernetes/kube-proxy.conf

kubectl config use-context kubernetes \--kubeconfig=/etc/kubernetes/kube-proxy.conf

复制代码

#把生成的 kube-proxy.conf 传给每个节点for NODE in k8s-master2 k8s-master3 k8s-node1 k8s-node2 k8s-node3; do      scp /etc/kubernetes/kube-proxy.conf $NODE:/etc/kubernetes/ done

复制代码

3.2、配置 kube-proxy.service

# 所有节点配置 kube-proxy.service 服务，一会儿设置为开机启动vi /usr/lib/systemd/system/kube-proxy.service
[Unit]Description=Kubernetes Kube ProxyDocumentation=https://github.com/kubernetes/kubernetesAfter=network.target
[Service]ExecStart=/usr/local/bin/kube-proxy \  --config=/etc/kubernetes/kube-proxy.yaml \  --v=2
Restart=alwaysRestartSec=10s
[Install]WantedBy=multi-user.target

复制代码

3.3、准备 kube-proxy.yaml

一定注意修改自己的 Pod 网段范围

# 所有机器执行vi /etc/kubernetes/kube-proxy.yaml

复制代码

apiVersion: kubeproxy.config.k8s.io/v1alpha1bindAddress: 0.0.0.0clientConnection:  acceptContentTypes: ""  burst: 10  contentType: application/vnd.kubernetes.protobuf  kubeconfig: /etc/kubernetes/kube-proxy.conf   #kube-proxy引导文件  qps: 5clusterCIDR: 196.16.0.0/16  #修改为自己的Pod-CIDRconfigSyncPeriod: 15m0sconntrack:  max: null  maxPerCore: 32768  min: 131072  tcpCloseWaitTimeout: 1h0m0s  tcpEstablishedTimeout: 24h0m0senableProfiling: falsehealthzBindAddress: 0.0.0.0:10256hostnameOverride: ""iptables:  masqueradeAll: false  masqueradeBit: 14  minSyncPeriod: 0s  syncPeriod: 30sipvs:  masqueradeAll: true  minSyncPeriod: 5s  scheduler: "rr"  syncPeriod: 30skind: KubeProxyConfigurationmetricsBindAddress: 127.0.0.1:10249mode: "ipvs"nodePortAddresses: nulloomScoreAdj: -999portRange: ""udpIdleTimeout: 250ms

复制代码

3.4、启动 kube-proxy

所有节点启动

systemctl daemon-reload && systemctl enable --now kube-proxysystemctl status kube-proxy

复制代码

十、部署 calico

可以参照 calico 私有云部署指南

# 下载官网calicocurl https://docs.projectcalico.org/manifests/calico-etcd.yaml -o calico.yaml## 把这个镜像修改成国内镜像

# 修改一些我们自定义的. 修改etcd集群地址sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.0.10:2379,https://192.168.0.11:2379,https://192.168.0.12:2379"#g' calico.yaml

# etcd的证书内容，需要base64编码设置到yaml中ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.pem | base64 -w 0 `ETCD_CERT=`cat /etc/kubernetes/pki/etcd/etcd.pem | base64 -w 0 `ETCD_KEY=`cat /etc/kubernetes/pki/etcd/etcd-key.pem | base64 -w 0 `
# 替换etcd中的证书base64编码后的内容sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico.yaml

#打开 etcd_ca 等默认设置（calico启动后自己生成）。sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico.yaml
# 修改自己的Pod网段 196.16.0.0/16POD_SUBNET="196.16.0.0/16"sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico.yaml# 一定确定自己是否修改好了
#确认calico是否修改好grep "CALICO_IPV4POOL_CIDR" calico.yaml -A 1

复制代码

# 应用calico配置kubectl apply -f calico.yaml

复制代码

十一、部署 coreDNS

git clone https://github.com/coredns/deployment.gitcd deployment/kubernetes
#10.96.0.10 改为 service 网段的 第 10 个ip./deploy.sh -s -i 10.96.0.10 | kubectl apply -f -

复制代码

十二、给机器打上 role 标签

kubectl label node k8s-master1 node-role.kubernetes.io/master=''kubectl label node k8s-master2 node-role.kubernetes.io/master=''kubectl label node k8s-master3 node-role.kubernetes.io/master=''
kubectl taints node k8s-master1

复制代码

十三、集群验证

验证 Pod 网络可访问性

-  同名称空间，不同名称空间可以使用 ip 互相访问

复制代码

-  跨机器部署的Pod也可以互相访问

复制代码

验证 Service 网络可访问性

-  集群机器使用serviceIp可以负载均衡访问

复制代码

-  pod内部可以访问service域名 serviceName.namespace

复制代码

-  pod可以访问跨名称空间的service

复制代码

# 部署以下内容进行测试
apiVersion: apps/v1kind: Deploymentmetadata:  name:  nginx-01  namespace: default  labels:    app:  nginx-01spec:  selector:    matchLabels:      app: nginx-01  replicas: 1  template:    metadata:      labels:        app:  nginx-01    spec:      containers:      - name:  nginx-01        image:  nginx---apiVersion: v1kind: Servicemetadata:  name: nginx-svc  namespace: defaultspec:  selector:    app:  nginx-01  type: ClusterIP  ports:  - name: nginx-svc    port: 80    targetPort: 80    protocol: TCP---apiVersion: v1kind: Namespacemetadata:  name: hellospec: {}---apiVersion: apps/v1kind: Deploymentmetadata:  name:  nginx-hello  namespace: hello  labels:    app:  nginx-hellospec:  selector:    matchLabels:      app: nginx-hello  replicas: 1  template:    metadata:      labels:        app:  nginx-hello    spec:      containers:      - name:  nginx-hello        image:  nginx---apiVersion: v1kind: Servicemetadata:  name: nginx-svc-hello  namespace: hellospec:  selector:    app:  nginx-hello  type: ClusterIP  ports:  - name: nginx-svc-hello    port: 80    targetPort: 80    protocol: TCP

复制代码

# 给两个master标识为workerkubectl label node k8s-node3 node-role.kubernetes.io/worker=''kubectl label node k8s-master3 node-role.kubernetes.io/worker=''kubectl label node k8s-node1 node-role.kubernetes.io/worker=''kubectl label node k8s-node2 node-role.kubernetes.io/worker=''
# 给master1打上污点。二进制部署的集群，默认master是没有污点的，可以任意调度。我们最好给一个master打上污点，保证master最小可用kubectl label node k8s-master3 node-role.kubernetes.io/master=''kubectl taint nodes k8s-master1 node-role.kubernetes.io/master=:NoSchedule

复制代码

发布于: 3 小时前阅读数: 9

原文链接:【http://xie.infoq.cn/article/dbef63661485033d10645a906】。文章转载请联系作者。

Lansonli

关注

微信公众号：三帮大数据 2022.07.12 加入

CSDN大数据领域博客专家，华为云享专家、阿里云专家博主、腾云先锋（TDP）核心成员、51CTO专家博主，全网六万多粉丝，知名互联网公司大数据高级开发工程师

发布

暂无评论

创作场景

云原生（二十八） | Kubernetes 篇之自建高可用 k8s 集群搭建

自建高可用 k8s 集群搭建

一、所有节点基础环境

1、环境准备与内核升级

2、安装 Docker

二、PKI

三、证书工具准备

1、下载证书工具

2、ca 根配置

3、ca 签名请求

4、生成证书

四、etcd 高可用搭建

1、etcd 文档

2、下载 etcd

3、etcd 证书

4、etcd 高可用安装

五、k8s 组件与证书

1、K8s 离线安装包

2、master 节点准备

3、apiserver 证书生成

4、front-proxy 证书生成

5、controller-manage 证书生成与配置

6、scheduler 证书生成与配置

7、admin 证书生成与配置

8、ServiceAccount Key 生成

9、发送证书到其他节点

六、高可用配置

七、组件启动

1、所有 master 执行

2、配置 apiserver 服务

3、配置 controller-manager 服务

4、配置 scheduler

八、TLS 与引导启动原理

1、master1 配置 bootstrap

2、master1 设置 kubectl 执行权限

3、创建集群引导权限文件

九、引导 Node 节点启动

1、发送核心证书到节点

2、所有节点配置 kubelet

3、kube-proxy 配置

十、部署 calico

十一、部署 coreDNS

十二、给机器打上 role 标签

十三、集群验证

Lansonli

评论