写点什么

Kubernetes 集群监控平台 (kube-prometheus-stack/Metrics Server) 搭建教程

作者:山河已无恙
  • 2022 年 2 月 17 日
  • 本文字数:15425 字

    阅读完需:约 51 分钟

Kubernetes集群监控平台(kube-prometheus-stack/Metrics Server)搭建教程

写在前面


  • 学习K8s涉及,整理笔记记忆

  • 博文偏实战,内容涉及:

  • 集群核心指标(Core Metrics)监控工具Metrics Server的简介

  • Metrics Server的安装Demo

  • 集群自定义指标(Custom Metrics)监控平台简介:

  • Prometheus

  • Grafana

  • NodeExporter

  • 通过helm(kube-prometheus-stack)安装监控平台的Demo


带着凡世的梦想,将污秽的灵魂依偎在纯洁的天边。它们是所有流浪、追寻、渴望与乡愁的永恒象征。 ——赫尔曼·黑塞《彼得.卡门青》


Kubernetes 集群性能监控管理

Kubernetes平台搭建好后,了解Kubernetes平台及在此平台上部署的应用的运行状况,以及处理系统主要告誓及性能瓶颈,这些都依赖监控管理系统


Kubernetes的早期版本依靠Heapster来实现完整的性能数据采集和监控功能, Kubernetes从1.8版本开始,性能数据开始以Metrics APl的方式提供标准化接口,并且从1.10版本开始将Heapster替换为MetricsServer


Kubernetes新的监控体系中:Metrics Server用于提供核心指标(Core Metrics) ,包括Node, Pod的CPU和内存使用指标。对其他自定义指标(Custom Metrics)的监控则由Prometheus等组件来完成。


监控节点状态,我们使用docker的话可以通过docker stats.

┌──[root@vms81.liruilongs.github.io]-[~]└─$docker statsCONTAINER ID   NAME                                                                                                                             CPU %     MEM USAGE / LIMIT     MEM %     NET I/O   BLOCK I/O     PIDS781c898eea19   k8s_kube-scheduler_kube-scheduler-vms81.liruilongs.github.io_kube-system_5bd71ffab3a1f1d18cb589aa74fe082b_18                     0.15%     23.22MiB / 3.843GiB   0.59%     0B / 0B   0B / 0B       7acac8b21bb57   k8s_kube-controller-manager_kube-controller-manager-vms81.liruilongs.github.io_kube-system_93d9ae7b5a4ccec4429381d493b5d475_18   1.18%     59.16MiB / 3.843GiB   1.50%     0B / 0B   0B / 0B       6fe97754d3dab   k8s_calico-node_calico-node-skzjp_kube-system_a211c8be-3ee1-44a0-a4ce-3573922b65b2_14                                            4.89%     94.25MiB / 3.843GiB   2.39%     0B / 0B   0B / 4.1kB    40
复制代码

那使用 k8s 的话,我们可以通过Metrics Server监控PodNodeCPU和内存资源使用数据

Metrics Server:集群性能监控平台

Metrics Server在部署完成后,将通过Kubernetes核心API Server 的/apis/metrics.k8s.io/v1beta1路径提供 Pod 和 Node 的监控数据。

安装 Metrics Server

Metrics Server源代码和部署配置可以在 GitHub 代码库

curl -Ls https://api.github.com/repos/kubernetes-sigs/metrics-server/tarball/v0.3.6 -o metrics-server-v0.3.6.tar.gz
复制代码

相关镜像

docker pull mirrorgooglecontainers/metrics-server-amd64:v0.3.6
复制代码

镜像小伙伴可以下载一下,这里我已经下载好了,直接上传导入镜像

┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$ansible all -m copy -a "src=./metrics-img.tar dest=/root/metrics-img.tar"┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "systemctl restart docker "192.168.26.82 | CHANGED | rc=0 >>
192.168.26.83 | CHANGED | rc=0 >>
192.168.26.81 | CHANGED | rc=0 >>
复制代码

通过 docker 命令导入镜像

┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$ansible all -m shell -a "docker load -i /root/metrics-img.tar"192.168.26.83 | CHANGED | rc=0 >>Loaded image: k8s.gcr.io/metrics-server-amd64:v0.3.6192.168.26.81 | CHANGED | rc=0 >>Loaded image: k8s.gcr.io/metrics-server-amd64:v0.3.6192.168.26.82 | CHANGED | rc=0 >>Loaded image: k8s.gcr.io/metrics-server-amd64:v0.3.6┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$
复制代码

修改 metrics-server-deployment.yaml

┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$mv kubernetes-sigs-metrics-server-d1f4f6f/ metrics┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$cd metrics/┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics]└─$lscmd                 deploy      hack      OWNERS          README.md          versioncode-of-conduct.md  Gopkg.lock  LICENSE   OWNERS_ALIASES  SECURITY_CONTACTSCONTRIBUTING.md     Gopkg.toml  Makefile  pkg             vendor┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics]└─$cd deploy/1.8+/┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics/deploy/1.8+]└─$lsaggregated-metrics-reader.yaml  metrics-apiservice.yaml         resource-reader.yamlauth-delegator.yaml             metrics-server-deployment.yamlauth-reader.yaml                metrics-server-service.yaml
复制代码

这里修改一些镜像获取策略,因为 Githup 上的镜像拉去不下来,或者拉去比较麻烦,所以我们提前上传好

┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics/deploy/1.8+]└─$vim metrics-server-deployment.yaml
复制代码


 31       - name: metrics-server 32         image: k8s.gcr.io/metrics-server-amd64:v0.3.6 33         #imagePullPolicy: Always 34         imagePullPolicy: IfNotPresent 35         command: 36         - /metrics-server 37         - --metric-resolution=30s 38         - --kubelet-insecure-tls 39         - --kubelet-preferred-address-types=InternalIP 40         volumeMounts:
复制代码

运行资源文件,创建相关资源对象

┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics/deploy/1.8+]└─$kubectl apply -f .
复制代码

查看 pod 列表,metrics-server 创建成功

┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics/deploy/1.8+]└─$kubectl  get pods -n kube-systemNAME                                                 READY   STATUS    RESTARTS   AGEcalico-kube-controllers-78d6f96c7b-79xx4             1/1     Running   2          3h15mcalico-node-ntm7v                                    1/1     Running   1          12hcalico-node-skzjp                                    1/1     Running   4          12hcalico-node-v7pj5                                    1/1     Running   1          12hcoredns-545d6fc579-9h2z4                             1/1     Running   2          3h15mcoredns-545d6fc579-xgn8x                             1/1     Running   2          3h16metcd-vms81.liruilongs.github.io                      1/1     Running   1          13hkube-apiserver-vms81.liruilongs.github.io            1/1     Running   2          13hkube-controller-manager-vms81.liruilongs.github.io   1/1     Running   4          13hkube-proxy-rbhgf                                     1/1     Running   1          13hkube-proxy-vm2sf                                     1/1     Running   1          13hkube-proxy-zzbh9                                     1/1     Running   1          13hkube-scheduler-vms81.liruilongs.github.io            1/1     Running   5          13hmetrics-server-bcfb98c76-gttkh                       1/1     Running   0          70m
复制代码

通过kubectl top nodes命令测试,

┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics/deploy/1.8+]└─$kubectl top nodesW1007 14:23:06.102605  102831 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flagNAME                         CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%vms81.liruilongs.github.io   555m         27%    2025Mi          52%vms82.liruilongs.github.io   204m         10%    595Mi           15%vms83.liruilongs.github.io   214m         10%    553Mi           14%┌──[root@vms81.liruilongs.github.io]-[~/ansible/metrics/deploy/1.8+]└─$
复制代码

Prometheus+Grafana+NodeExporter:集群监控平台

在各个计算节点上部署NodeExporter采集CPU、内存、磁盘及IO信息,并将这些信息传输给监控节点上的Prometheus服务器进行存储分析,通过Grafana进行可视化监控,

Prometheus

Prometheus是一款开源的监控解决方案,由SoundCloud公司开发的开源监控系统,是继Kubernetes之后CNCF第 2 个孵化成功的项目,在容器和微服务领域得到了广泛应用,能在监控Kubernetes平台的同时监控部署在此平台中的应用,它提供了一系列工具集及多维度监控指标。Prometheus依赖Grafana实现数据可视化


Prometheus 的主要特点如下:

  • 使用指标名称键值对标识多维度数据模型

  • 采用灵活的查询语言PromQL

  • 不依赖分布式存储,为自治的单节点服务

  • 使用HTTP完成对监控数据的拉取。

  • 支持通过网关推送时序数据

  • 支持多种图形和Dashboard的展示,例如Grafana

Prometheus 生态系统由各种组件组成,用于功能的扩充:

Prometheus 的核心组件Prometheus Server的主要功能包括:


Kubernetes Master 获取需要监控的资源或服务信息;从各种Exporter抓取(Pull)指标数据,然后将指标数据保存在时序数据库(TSDB)中;向其他系统提供HTTP API进行查询;提供基于PromQL语言的数据查询;可以将告警数据推送(Push)给AlertManager,等等。



Prometheus 的系统架构:

NodeExporter

NodeExporter主要用来采集服务器CPU、内存、磁盘、IO等信息,是机器数据的通用采集方案。只要在宿主机上安装NodeExporter和cAdisor容器,通过Prometheus进行抓取即可。它同Zabbix的功能相似.

Grafana

Grafana是一个Dashboard工具,用GoJS开发,它是一个时间序列数据库界面展示层,通过SQL命令查询Metrics并将结果展示出来。它能自定义多种仪表盘,可以轻松实现覆盖多个Docker的宿主机监控信息的展现。

搭建 Prometheus+Grafana+NodeExporter 平台

这里我们通过helm的方式搭建,简单方便快捷,运行之后,相关的镜像都会创建成功.下面是创建成功的镜像列表。

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl get podsNAME                                                    READY   STATUS    RESTARTS      AGEalertmanager-liruilong-kube-prometheus-alertmanager-0   2/2     Running   0             61mliruilong-grafana-5955564c75-zpbjq                      3/3     Running   0             62mliruilong-kube-prometheus-operator-5cb699b469-fbkw5     1/1     Running   0             62mliruilong-kube-state-metrics-5dcf758c47-bbwt4           1/1     Running   7 (32m ago)   62mliruilong-prometheus-node-exporter-rfsc5                1/1     Running   0             62mliruilong-prometheus-node-exporter-vm7s9                1/1     Running   0             62mliruilong-prometheus-node-exporter-z9j8b                1/1     Running   0             62mprometheus-liruilong-kube-prometheus-prometheus-0       2/2     Running   0             61m┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$
复制代码

环境版本

我的 K8s 集群版本

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$kubectl  get nodesNAME                         STATUS   ROLES                  AGE   VERSIONvms81.liruilongs.github.io   Ready    control-plane,master   34d   v1.22.2vms82.liruilongs.github.io   Ready    <none>                 34d   v1.22.2vms83.liruilongs.github.io   Ready    <none>                 34d   v1.22.2
复制代码

hrlm 版本

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm versionversion.BuildInfo{Version:"v3.2.1", GitCommit:"fe51cd1e31e6a202cba7dead9552a6d418ded79a", GitTreeState:"clean", GoVersion:"go1.13.10"}
复制代码

prometheus-operator(旧名字)安装出现的问题

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm  search repo prometheus-operatorNAME                            CHART VERSION   APP VERSION     DESCRIPTIONali/prometheus-operator         8.7.0           0.35.0          Provides easy monitoring definitions for Kubern...azure/prometheus-operator       9.3.2           0.38.1          DEPRECATED Provides easy monitoring definitions...┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm install liruilong  ali/prometheus-operatorError: failed to install CRD crds/crd-alertmanager.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm pull  ali/prometheus-operator┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$
复制代码

解决办法:新版本安装

直接下载kube-prometheus-stack(新)的 chart 包,通过命令安装:

https://github.com/prometheus-community/helm-charts/releases/download/kube-prometheus-stack-30.0.1/kube-prometheus-stack-30.0.1.tgz

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$lsindex.yaml  kube-prometheus-stack-30.0.1.tgz  liruilonghelm  liruilonghelm-0.1.0.tgz  mysql  mysql-1.6.4.tgz┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$helm listNAME    NAMESPACE       REVISION        UPDATED STATUS  CHART   APP VERSION
复制代码

解压 chart 包 kube-prometheus-stack-30.0.1.tgz

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$tar -zxf kube-prometheus-stack-30.0.1.tgz
复制代码

创建新的命名空间

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$cd kube-prometheus-stack/┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl create ns monitoringnamespace/monitoring created┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl config  set-context $(kubectl config current-context) --namespace=monitoringContext "kubernetes-admin@kubernetes" modified.
复制代码

进入文件夹,直接通过helm install liruilong .安装

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$lsChart.lock  charts  Chart.yaml  CONTRIBUTING.md  crds  README.md  templates  values.yaml┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$helm install liruilong .
复制代码

kube-prometheus-admission-create 对应 Pod 的相关镜像下载不下来问题

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$kubectl get podsNAME                                                  READY   STATUS             RESTARTS   AGEliruilong-kube-prometheus-admission-create--1-bn7x2   0/1     ImagePullBackOff   0          33s
复制代码

查看 pod 详细信息,发现是谷歌的一个镜像国内无法下载

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$kubectl  describe pod  liruilong-kube-prometheus-admission-create--1-bn7x2Name:         liruilong-kube-prometheus-admission-create--1-bn7x2Namespace:    monitoringPriority:     0Node:         vms83.liruilongs.github.io/192.168.26.83Start Time:   Sun, 16 Jan 2022 02:43:07 +0800Labels:       app=kube-prometheus-stack-admission-create              app.kubernetes.io/instance=liruilong              app.kubernetes.io/managed-by=Helm              app.kubernetes.io/part-of=kube-prometheus-stack              app.kubernetes.io/version=30.0.1              chart=kube-prometheus-stack-30.0.1              controller-uid=2ce48cd2-a118-4e23-a27f-0228ef6c45e7              heritage=Helm              job-name=liruilong-kube-prometheus-admission-create              release=liruilongAnnotations:  cni.projectcalico.org/podIP: 10.244.70.8/32              cni.projectcalico.org/podIPs: 10.244.70.8/32Status:       PendingIP:           10.244.70.8IPs:  IP:           10.244.70.8Controlled By:  Job/liruilong-kube-prometheus-admission-createContainers:  create:    Container ID:    Image:         k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0@sha256:f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068    Image ID:    Port:          <none>    Host Port:    。。。。。。。。。。。。。。。。。。。。。。。。。。。
复制代码

在 dokcer 仓库里找了一个类似的,通过 kubectl edit修改

image: k8s.gcr.io/ingress-nginx/kube-webhook-certgen:v1.0  替换为 : docker.io/liangjw/kube-webhook-certgen:v1.1.1
复制代码

或者也可以修改配置文件从新 install(记得要把 sha 注释掉)

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$lsindex.yaml  kube-prometheus-stack  kube-prometheus-stack-30.0.1.tgz  liruilonghelm  liruilonghelm-0.1.0.tgz  mysql  mysql-1.6.4.tgz┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create]└─$cd kube-prometheus-stack/┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$lsChart.lock  charts  Chart.yaml  CONTRIBUTING.md  crds  README.md  templates  values.yaml┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$cat values.yaml | grep -A 3 -B 2 kube-webhook-certgen      enabled: true      image:        repository: docker.io/liangjw/kube-webhook-certgen        tag: v1.1.1        #sha: "f3b6b39a6062328c095337b4cadcefd1612348fdd5190b1dcbcb9b9e90bd8068"        pullPolicy: IfNotPresent┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$helm del liruilong;helm install liruilong .
复制代码

之后其他的相关 pod 正常创建中

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl get podsNAME                                                  READY   STATUS              RESTARTS   AGEliruilong-grafana-5955564c75-zpbjq                    0/3     ContainerCreating   0          27sliruilong-kube-prometheus-operator-5cb699b469-fbkw5   0/1     ContainerCreating   0          27sliruilong-kube-state-metrics-5dcf758c47-bbwt4         0/1     ContainerCreating   0          27sliruilong-prometheus-node-exporter-rfsc5              0/1     ContainerCreating   0          28sliruilong-prometheus-node-exporter-vm7s9              0/1     ContainerCreating   0          28sliruilong-prometheus-node-exporter-z9j8b              0/1     ContainerCreating   0          28s
复制代码

kube-state-metrics 这个 pod 的镜像也没有拉取下来。应该也是相同的原因

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl get podsNAME                                                    READY   STATUS             RESTARTS   AGEalertmanager-liruilong-kube-prometheus-alertmanager-0   2/2     Running            0          3m35sliruilong-grafana-5955564c75-zpbjq                      3/3     Running            0          4m46sliruilong-kube-prometheus-operator-5cb699b469-fbkw5     1/1     Running            0          4m46sliruilong-kube-state-metrics-5dcf758c47-bbwt4           0/1     ImagePullBackOff   0          4m46sliruilong-prometheus-node-exporter-rfsc5                1/1     Running            0          4m47sliruilong-prometheus-node-exporter-vm7s9                1/1     Running            0          4m47sliruilong-prometheus-node-exporter-z9j8b                1/1     Running            0          4m47sprometheus-liruilong-kube-prometheus-prometheus-0       2/2     Running            0          3m34s
复制代码

同样 k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0 这个镜像没办法拉取

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl  describe  pod liruilong-kube-state-metrics-5dcf758c47-bbwt4Name:         liruilong-kube-state-metrics-5dcf758c47-bbwt4Namespace:    monitoringPriority:     0Node:         vms82.liruilongs.github.io/192.168.26.82Start Time:   Sun, 16 Jan 2022 02:59:53 +0800Labels:       app.kubernetes.io/component=metrics              app.kubernetes.io/instance=liruilong              app.kubernetes.io/managed-by=Helm              app.kubernetes.io/name=kube-state-metrics              app.kubernetes.io/part-of=kube-state-metrics              app.kubernetes.io/version=2.3.0              helm.sh/chart=kube-state-metrics-4.3.0              pod-template-hash=5dcf758c47              release=liruilongAnnotations:  cni.projectcalico.org/podIP: 10.244.171.153/32              cni.projectcalico.org/podIPs: 10.244.171.153/32Status:       PendingIP:           10.244.171.153IPs:  IP:           10.244.171.153Controlled By:  ReplicaSet/liruilong-kube-state-metrics-5dcf758c47Containers:  kube-state-metrics:    Container ID:    Image:         k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.3.0    Image ID:    Port:          8080/TCP    。。。。。。。。。。。。。。。。。。。。。。
复制代码

同样的,我们通过 docker 仓库找一下相同的,然后通过kubectl edit pod修改一下

k8s.gcr.io/kube-state-metrics/kube-state-metrics 替换为: docker.io/dyrnq/kube-state-metrics:v2.3.0
复制代码

可以先在节点机上拉取一下

┌──[root@vms81.liruilongs.github.io]-[~/ansible]└─$ ansible node -m shell -a "docker pull dyrnq/kube-state-metrics:v2.3.0"192.168.26.82 | CHANGED | rc=0 >>v2.3.0: Pulling from dyrnq/kube-state-metricse8614d09b7be: Pulling fs layer53ccb90bafd7: Pulling fs layere8614d09b7be: Verifying Checksume8614d09b7be: Download completee8614d09b7be: Pull complete53ccb90bafd7: Verifying Checksum53ccb90bafd7: Download complete53ccb90bafd7: Pull completeDigest: sha256:c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030Status: Downloaded newer image for dyrnq/kube-state-metrics:v2.3.0docker.io/dyrnq/kube-state-metrics:v2.3.0192.168.26.83 | CHANGED | rc=0 >>v2.3.0: Pulling from dyrnq/kube-state-metricse8614d09b7be: Pulling fs layer53ccb90bafd7: Pulling fs layere8614d09b7be: Verifying Checksume8614d09b7be: Download completee8614d09b7be: Pull complete53ccb90bafd7: Verifying Checksum53ccb90bafd7: Download complete53ccb90bafd7: Pull completeDigest: sha256:c9137505edaef138cc23479c73e46e9a3ef7ec6225b64789a03609c973b99030Status: Downloaded newer image for dyrnq/kube-state-metrics:v2.3.0docker.io/dyrnq/kube-state-metrics:v2.3.0
复制代码

修改完之后,会发现所有的 pod 都创建成功

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl get podsNAME                                                    READY   STATUS    RESTARTS      AGEalertmanager-liruilong-kube-prometheus-alertmanager-0   2/2     Running   0             61mliruilong-grafana-5955564c75-zpbjq                      3/3     Running   0             62mliruilong-kube-prometheus-operator-5cb699b469-fbkw5     1/1     Running   0             62mliruilong-kube-state-metrics-5dcf758c47-bbwt4           1/1     Running   7 (32m ago)   62mliruilong-prometheus-node-exporter-rfsc5                1/1     Running   0             62mliruilong-prometheus-node-exporter-vm7s9                1/1     Running   0             62mliruilong-prometheus-node-exporter-z9j8b                1/1     Running   0             62mprometheus-liruilong-kube-prometheus-prometheus-0       2/2     Running   0             61m┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$
复制代码

然后我们需要修改 liruilong-grafana SVC 的类型为 NodePort,这样,物理机就可以访问了

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl  get svcNAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGEalertmanager-operated                    ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   33mliruilong-grafana                        ClusterIP   10.99.220.121    <none>        80/TCP                       34mliruilong-kube-prometheus-alertmanager   ClusterIP   10.97.193.228    <none>        9093/TCP                     34mliruilong-kube-prometheus-operator       ClusterIP   10.101.106.93    <none>        443/TCP                      34mliruilong-kube-prometheus-prometheus     ClusterIP   10.105.176.19    <none>        9090/TCP                     34mliruilong-kube-state-metrics             ClusterIP   10.98.94.55      <none>        8080/TCP                     34mliruilong-prometheus-node-exporter       ClusterIP   10.110.216.215   <none>        9100/TCP                     34mprometheus-operated                      ClusterIP   None             <none>        9090/TCP                     33m┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack]└─$kubectl edit svc liruilong-grafanaservice/liruilong-grafana edited┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl  get svcNAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGEalertmanager-operated                    ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   35mliruilong-grafana                        NodePort    10.99.220.121    <none>        80:30443/TCP                 36mliruilong-kube-prometheus-alertmanager   ClusterIP   10.97.193.228    <none>        9093/TCP                     36mliruilong-kube-prometheus-operator       ClusterIP   10.101.106.93    <none>        443/TCP                      36mliruilong-kube-prometheus-prometheus     ClusterIP   10.105.176.19    <none>        9090/TCP                     36mliruilong-kube-state-metrics             ClusterIP   10.98.94.55      <none>        8080/TCP                     36mliruilong-prometheus-node-exporter       ClusterIP   10.110.216.215   <none>        9100/TCP                     36mprometheus-operated                      ClusterIP   None             <none>        9090/TCP                     35m
复制代码



通过 secrets 解密获取用户名密码

┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets | grep grafanaliruilong-grafana                                                  Opaque                                3      38mliruilong-grafana-test-token-q8z8j                                 kubernetes.io/service-account-token   3      38mliruilong-grafana-token-j94p8                                      kubernetes.io/service-account-token   3      38m┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets liruilong-grafana -o yamlapiVersion: v1data:  admin-password: cHJvbS1vcGVyYXRvcg==  admin-user: YWRtaW4=  ldap-toml: ""kind: Secretmetadata:  annotations:    meta.helm.sh/release-name: liruilong    meta.helm.sh/release-namespace: monitoring  creationTimestamp: "2022-01-15T18:59:40Z"  labels:    app.kubernetes.io/instance: liruilong    app.kubernetes.io/managed-by: Helm    app.kubernetes.io/name: grafana    app.kubernetes.io/version: 8.3.3    helm.sh/chart: grafana-6.20.5  name: liruilong-grafana  namespace: monitoring  resourceVersion: "1105663"  uid: c03ff5f3-deb5-458c-8583-787f41034469type: Opaque┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets liruilong-grafana -o jsonpath='{.data.admin-user}}'| base64 -dadminbase64: 输入无效┌──[root@vms81.liruilongs.github.io]-[~/ansible/k8s-helm-create/kube-prometheus-stack/templates]└─$kubectl get secrets liruilong-grafana -o jsonpath='{.data.admin-password}}'| base64 -dprom-operatorbase64: 输入无效
复制代码

得到用户名密码:admin/prom-operator






发布于: 2022 年 02 月 17 日阅读数: 5
用户头像

CSDN博客专家,华为云云享专家,RHCE/CKA认证 2022.01.04 加入

Java 后端一枚,技术不高,前端、Shell、Python 也可以写一点.纯种屌丝,不热爱生活,热爱学习,热爱工作,喜欢一直忙,不闲着。喜欢篆刻,喜欢吃好吃的,喜欢吃饱了晒太阳。

评论

发布
暂无评论
Kubernetes集群监控平台(kube-prometheus-stack/Metrics Server)搭建教程