极客时间运维进阶训练营第七周作业

作者：9527

2022-12-08
美国
本文字数：6365 字
阅读完需：约 21 分钟

Mandatory

了解 Prometheus 各组件的功能，熟悉 Prometheus 的数据采集流程
基于 Docker 或二进制部署 Prometheus Server
基于 Docker 或二进制部署 node-exporter，并通过 Prometheus 收集 node-exporter 指标数据
安装 Grafana 并添加 Prometheus 数据源，导入模板可以图形显示指标数据
掌握 Prometheus 的 PromQL 语句的简单使用
部署 Prometheus 联邦集群并实现指标数据收集

Prometheus Introduction

Monitoring

The value of monitoring system

Monitoring type

Process

Prometheus

Introduction

Architecture

Data collection flow

TSDB

Install Prometheus and node-exporter on K8s

Check K8s cluster status

$ k get nsNAME              STATUS   AGEdefault           Active   13mkube-node-lease   Active   13mkube-public       Active   13mkube-system       Active   13m

复制代码

Add prometheus Helm repo

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts"prometheus-community" has been added to your repositories
$ helm repo add grafana https://grafana.github.io/helm-charts"grafana" has been added to your repositories

复制代码

Deploy prometheus

$ k create namespace prometheusnamespace/prometheus created
$ helm install prometheus prometheus-community/prometheus     --namespace prometheus     --set alertmanager.persistentVolume.storageClass="gp2"     --set server.persistentVolume.storageClass="gp2"NAME: prometheusLAST DEPLOYED: Wed Dec  7 18:23:05 2022NAMESPACE: prometheusSTATUS: deployedREVISION: 1NOTES:The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:prometheus-server.prometheus.svc.cluster.local

Get the Prometheus server URL by running these commands in the same shell:  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")  kubectl --namespace prometheus port-forward $POD_NAME 9090

The Prometheus alertmanager can be accessed via port  on the following DNS name from within your cluster:prometheus-%!s(<nil>).prometheus.svc.cluster.local

Get the Alertmanager URL by running these commands in the same shell:  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus,component=" -o jsonpath="{.items[0].metadata.name}")  kubectl --namespace prometheus port-forward $POD_NAME 9093#######################################################################################   WARNING: Pod Security Policy has been disabled by default since    ###########            it deprecated after k8s 1.25+. use                        ###########            (index .Values "prometheus-node-exporter" "rbac"          ########### .          "pspEnabled") with (index .Values                         ###########            "prometheus-node-exporter" "rbac" "pspAnnotations")       ###########            in case you still need it.                                ######################################################################################

The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:prometheus-prometheus-pushgateway.prometheus.svc.cluster.local

Get the PushGateway URL by running these commands in the same shell:  export POD_NAME=$(kubectl get pods --namespace prometheus -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")  kubectl --namespace prometheus port-forward $POD_NAME 9091
For more information on running Prometheus, visit:https://prometheus.io/

复制代码

Check the installation

$ helm list -n prometheusNAME      	NAMESPACE 	REVISION	UPDATED                                	STATUS  	CHART            	APP VERSIONprometheus	prometheus	1       	2022-12-07 18:23:05.496767122 +0000 UTC	deployed	prometheus-19.0.1	v2.40.5
$  k get po -n prometheusNAME                                                 READY   STATUS    RESTARTS   AGEprometheus-alertmanager-0                            1/1     Running   0          22mprometheus-kube-state-metrics-7c668ff876-h9crw       1/1     Running   0          22mprometheus-prometheus-node-exporter-899pz            1/1     Running   0          22mprometheus-prometheus-pushgateway-57d6d7494c-wh2wj   1/1     Running   0          22mprometheus-server-74c69b74f5-t8p8x                   1/2     Running   0          22m

复制代码

Forward port and check UI

$ k port-forward -n prometheus deploy/prometheus-server 8080:9090 --address 0.0.0.0Forwarding from 0.0.0.0:8080 -> 9090Handling connection for 8080Handling connection for 8080

复制代码

Install Grafana on K8s

Install

$ mkdir -p ${HOME}/environment/grafana
$ cat << EoF > ${HOME}/environment/grafana/grafana.yamldatasources:  datasources.yaml:    apiVersion: 1    datasources:    - name: Prometheus      type: prometheus      url: http://prometheus-server.prometheus.svc.cluster.local      access: proxy      isDefault: trueEoF
$ k create namespace grafananamespace/grafana created
$ helm install grafana grafana/grafana \>     --namespace grafana \>     --set persistence.storageClassName="gp2" \>     --set persistence.enabled=true \>     --set adminPassword='awesome' \>     --values ${HOME}/environment/grafana/grafana.yaml \>     --set service.type=LoadBalancerW1207 19:03:28.382865   12279 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+W1207 19:03:28.550183   12279 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+NAME: grafanaLAST DEPLOYED: Wed Dec  7 19:03:27 2022NAMESPACE: grafanaSTATUS: deployedREVISION: 1NOTES:1. Get your 'admin' user password by running:
   kubectl get secret --namespace grafana grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:
   grafana.grafana.svc.cluster.local
   Get the Grafana URL to visit by running these commands in the same shell:   NOTE: It may take a few minutes for the LoadBalancer IP to be available.        You can watch the status of by running 'kubectl get svc --namespace grafana -w grafana'     export SERVICE_IP=$(kubectl get svc --namespace grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}')     http://$SERVICE_IP:80
3. Login with the password from step 1 and the username: admin

复制代码

Check deployment

$ helm list -n grafanaNAME   	NAMESPACE	REVISION	UPDATED                                	STATUS  	CHART         	APP VERSIONgrafana	grafana  	1       	2022-12-07 19:03:27.413111235 +0000 UTC	deployed	grafana-6.45.1	9.3.0
$ k get po -n grafanaNAME                       READY   STATUS    RESTARTS   AGEgrafana-5878dccf6d-bjzdn   1/1     Running   0          38s
$ k get svc -n grafanaNAME      TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)        AGEgrafana   LoadBalancer   10.100.77.114   a475d6b973c544aa7ba79675a7bac425-736003492.us-east-1.elb.amazonaws.com   80:31766/TCP   50s

复制代码

Access UI

$ export ELB=$(kubectl get svc -n grafana grafana -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
$ echo "http://$ELB"

复制代码

Config Grafana

Add "Prometheus" data source

Install template

Check Dashboard

PromQL Statement

Introduction

Official documentation: https://prometheus.io/docs/prometheus/latest/querying/basics/
Instance vector

$ curl http://127.0.0.1:8080/api/v1/query --data 'query=node_memory_MemFree_bytes' | jq{  "status": "success",  "data": {    "resultType": "vector",    "result": [      {        "metric": {          "__name__": "node_memory_MemFree_bytes",          "app_kubernetes_io_component": "metrics",          "app_kubernetes_io_instance": "prometheus",          "app_kubernetes_io_managed_by": "Helm",          "app_kubernetes_io_name": "prometheus-node-exporter",          "app_kubernetes_io_part_of": "prometheus-node-exporter",          "app_kubernetes_io_version": "1.5.0",          "helm_sh_chart": "prometheus-node-exporter-4.8.0",          "instance": "192.168.0.232:9100",          "job": "kubernetes-service-endpoints",          "namespace": "prometheus",          "node": "ip-192-168-0-232.ec2.internal",          "service": "prometheus-prometheus-node-exporter"        },        "value": [          1670459005.891,          "248516608"        ]      }    ]  }}

复制代码

Range vector

$ curl http://127.0.0.1:8080/api/v1/query --data 'query=node_memory_MemFree_bytes[5m]' | jq{  "status": "success",  "data": {    "resultType": "matrix",    "result": [      {        "metric": {          "__name__": "node_memory_MemFree_bytes",          "app_kubernetes_io_component": "metrics",          "app_kubernetes_io_instance": "prometheus",          "app_kubernetes_io_managed_by": "Helm",          "app_kubernetes_io_name": "prometheus-node-exporter",          "app_kubernetes_io_part_of": "prometheus-node-exporter",          "app_kubernetes_io_version": "1.5.0",          "helm_sh_chart": "prometheus-node-exporter-4.8.0",          "instance": "192.168.0.232:9100",          "job": "kubernetes-service-endpoints",          "namespace": "prometheus",          "node": "ip-192-168-0-232.ec2.internal",          "service": "prometheus-prometheus-node-exporter"        },        "values": [          [            1670458779.648,            "250081280"          ],          [            1670458839.648,            "250404864"          ],          [            1670458899.648,            "247250944"          ],          [            1670458959.648,            "248516608"          ],          [            1670459019.648,            "247308288"          ]        ]      }    ]  }}

复制代码

scalar

$ curl http://127.0.0.1:8080/api/v1/query --data 'query=scalar(node_memory_MemFree_bytes)' | jq{  "status": "success",  "data": {    "resultType": "scalar",    "result": [      1670459171.963,      "246968320"    ]  }}

复制代码

Others

tag matching

Time range

Operator

Aggregate

Prometheus Fedaration for K8s

Introduction

Install the Prometheus Operator on each cluster
Ref: https://github.com/kemilad/prometheus-federation-and-thanos-k8s-webinar
Add the Bitnami charts repository to Helm: helm repo add bitnami https://charts.bitnami.com/bitnami
Install the Prometheus Operator in the first “data producer” cluster using the command below:

helm install prometheus-operator \  --set prometheus.thanos.create=true \  --set operator.service.type=ClusterIP \  --set prometheus.service.type=ClusterIP \  --set alertmanager.service.type=ClusterIP \  --set prometheus.thanos.service.type=LoadBalancer \  --set prometheus.externalLabels.cluster="<CLUSTER-LABEL>" \  bitnami/prometheus-operator

复制代码

Then get the sidecar service public ip address:

$ kubectl get svc | grep prometheus-operator-prometheus-thanos or kgs | grep prometheus-operator-prometheus-thanos

复制代码

Do the same thing for your second data producer cluster.
Install and configure Thanos
Get the values.yaml file in this repository; do the correct configurations according to that file.
Install Thanos using the command below:

helm install thanos bitnami/thanos \  --values values.yaml

复制代码

Wait for the deployment to complete and note the DNS name and port number for the Thanos Querier service in the deployment output.
Follow the instructions according to that output and check it on your browser.
Configure Grafana to use Thanos as a data source
From the Grafana dashboard, click the “Add data source” button.
On the “Choose data source type” page, select “Prometheus”.
On the “Settings” page, set the URL for the Prometheus server to http://NAME:PORT, where NAME is the DNS name for the Thanos service obtained at the end of Step 2 and PORT is the corresponding service port. Leave all other values at their default.
Click “Save & Test” to save and test the configuration. If everything is configured correctly, you should see a success message.

Optional

Prometheus pushgateway

Introduction

Pod YAML

apiVersion: v1kind: Podmetadata:  name: 'pushgateway'  labels:    function: 'pushgateway'spec:  containers:  - name: 'pushgateway'    image: prom/pushgateway---apiVersion: v1kind: Servicemetadata:  name: 'pushgateway'  labels:    function: 'pushgateway-target'spec:  type: 'ClusterIP'  selector:    function: 'pushgateway'  ports:    # 'name' is important, it will be referred by ServiceMonitor configuration    - name: 'pushgateway'      protocol: 'TCP'      port: 9091      targetPort: 9091---apiVersion: v1kind: Servicemetadata:  name: 'pushgateway-lb'spec:  type: 'LoadBalancer'  selector:    function: 'pushgateway'  ports:    - name: 'pushgateway'      protocol: 'TCP'      port: 9091      targetPort: 9091EOF

复制代码

Install

$ k create -f pushgateway.yml

复制代码

Reference: https://kb.novaordis.com/index.php/Prometheus_Pushgateway_Installation

发布于: 刚刚阅读数: 4

原文链接:【http://xie.infoq.cn/article/22c723a56e01886e44a684333】。文章转载请联系作者。

9527

关注

还未添加个人签名 2020-04-22 加入

还未添加个人简介

发布

暂无评论

创作场景

极客时间运维进阶训练营第七周作业

Mandatory

Prometheus Introduction

Monitoring

Prometheus

Install Prometheus and node-exporter on K8s

Install Grafana on K8s

Config Grafana

PromQL Statement

Prometheus Fedaration for K8s

Optional

Prometheus pushgateway

9527

评论