整合 Micrometer 与 Prometheus & ElasticSearch

# 整合Micrometer与Prometheus & ElasticSearch

为什么要做这个整合？其实主要目的是想要建立一个应用指标的监控系统，比如我要看最近一个小时新注册的用户数，这类指标是无法从通用监控系统获得数据的，都需要自己在应用代码内进行开发。而Micrometer是springboot2采用的统一监控框架，类似slf4j定义了统一的log接口，micrometer定义了统一的各种meter监控接口，背后支持各种实现，我们这里只讨论两种：elasticsearch和prometheus.

#### Prometheus

Prometheus可以被理解为一个时序数据库，是一个时序数据的存储和查询引擎，监控的话一般和需要和grafana配合。

#### Elasticsearch

ElasticSearch作为一个众所周知的搜索引擎，其本身也和spring框架一样，衍生和扩展出了很多系统和工具，比如elastic APM。 由于我们的CICD系统已经使用了elastic APM，所以看起来使用elasticsearch来完成应用指标监控是个很自然的选择，这样在kibana上就可以同时有APM和业务指标监控了。

## 1. Micrometer整合Prometheus

Promethes是通过定时从`http://host:port/actuator/prometheus`这个端点来读取meter数据。参考prometheus的配置：

```java

scrape_configs:

# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

- job_name: 'prometheus'

# metrics_path defaults to '/metrics'

# scheme defaults to 'http'.

static_configs:

- targets: ['localhost:9090']

- job_name: 'portal-bff'

metrics_path: '/actuator/prometheus'

static_configs:

- targets: ['192.168.48.22:18000']

```

#### 引入依赖

```java

<dependency>

<groupId>org.springframework.boot</groupId>

<artifactId>spring-boot-starter-actuator</artifactId>

</dependency>

<!--

Prometheus整合(需要引入spring-boot-actuator包)

参考spring-boot-actuator-autoconfigure中的自动配置类：org.springframework.boot.actuate.autoconfigure.metrics.export.prometheus.PrometheusMetricsExportAutoConfiguration

默认创建的prometheus endpoint为：http://host:${app_port}/actuator/prometheus

-->

<dependency>

<groupId>io.micrometer</groupId>

<artifactId>micrometer-registry-prometheus</artifactId>

</dependency>

```

引入这个依赖的话，springboot的autoconfiguration会自动注册于prometheus相关的bean，尤其`io.micrometer.prometheus.PrometheusMeterRegistry`

#### Counter

```java

Counter saleCounter = Counter.builder("product.search.group").tag("category", "test-skywalking")

.register(this.meterRegistry);

saleCounter.increment();

```

```java

# HELP product_search_group_total

# TYPE product_search_group_total counter

product_search_group_total{category="test-skywalking",} 5.0

```

#### Gauge

```java

Gauge gauge = Gauge.builder("gauge.product.search.group", () -> {

double value = new Random().nextInt(10) + 10.0;

logger.debug("gauge value:{}", value);

return value;

}).register(this.meterRegistry);

```

```java

# HELP gauge_product_search_group

# TYPE gauge_product_search_group gauge

gauge_product_search_group 12.0

```

#### Timer

```java

Timer.Sample sample = (Timer.Sample) ctx.get(PrometheusPreFilter.PROM_SAMPLE);

if (sample == null) {

return null;

}

// 移除timer

ctx.remove(PrometheusPreFilter.PROM_SAMPLE);

// 设置uri等tag

String catfishCommand = ctx.getRequest().getHeader(HEADER_CATFISH_COMMAND);

String uri = ctx.getRequest().getRequestURI();

String uriTag = catfishCommand != null ? simplifyCommandName(catfishCommand.trim()) : uri.replace("//", "/");

// 设置请求处理是否成功的tag...参考{@link com.netflix.zuul.http.ZuulServlet.error}

boolean isSuccess = isSuccess(ctx, catfishCommand);

sample.stop(this.meterRegistry.timer("portal.http.server.requests", "uri", uriTag, "success", isSuccess + ""));

```

```java

# HELP portal_http_server_requests_seconds

# TYPE portal_http_server_requests_seconds summary

portal_http_server_requests_seconds_count{success="true",uri="/api/v4/search-group",} 5.0

portal_http_server_requests_seconds_sum{success="true",uri="/api/v4/search-group",} 13.205301777

```

## 2. Micrometer整合Elasticsearch

* micrometer-elastic的属性：`org.springframework.boot.actuate.autoconfigure.metrics.export.elastic.ElasticProperties`

* autoconfiguration：`org.springframework.boot.actuate.autoconfigure.metrics.export.elastic.ElasticMetricsExportAutoConfiguration`

* MeterRegistry: `io.micrometer.elastic.ElasticMeterRegistry`

#### 引入依赖

```java

<dependency>

<groupId>io.micrometer</groupId>

<artifactId>micrometer-registry-elastic</artifactId>

<version>1.5.4</version>

</dependency>

```

ElasticMeterRegistry会主动向Elasticsearch推送meter，所以在应用程序内需要配置:

```java

management.metrics.export.elastic.enabled=true

management.metrics.export.elastic.host=http://172.20.7.161:9200

management.metrics.export.elastic.index=portalbiz

```

#### Counter

```java

// ElasticMeterRegistry注册的counter是StepCounter，这个counter不不会一直累加计数，而是和rate()函数有点象，默认以1分钟为step，超过1分钟counter就清零。

// ElasticMeterRegistry的构造函数会启动一个定时调度线程，每step时长执行一次，调用io.micrometer.elastic.ElasticMeterRegistry.publish()方法来向elastic

// 发送meter数据。

Counter saleCounter = Counter.builder("product.search.group").tag("category", "test-skywalking")

.register(this.meterRegistry);

saleCounter.increment();

```

> 通过wireshark抓包，获得提交到elasticsearch的http请求

```java

{"@timestamp":"2020-09-15T01:46:19.495Z","name":"product_search_group","type":"counter","category":"test-skywalking","count":3.0}

```

#### Gauge

```java

Gauge gauge = Gauge.builder("gauge.product.search.group", () -> {

double value = new Random().nextInt(10) + 10.0;

logger.debug("gauge value:{}", value);

return value;

}).register(this.meterRegistry);

```

```java

{"@timestamp":"2020-09-15T01:46:19.494Z","name":"gauge_product_search_group","type":"gauge","value":13.0}

```

#### Timer

```java

Timer.Sample sample = (Timer.Sample) ctx.get(PrometheusPreFilter.PROM_SAMPLE);

if (sample == null) {

return null;

}

... ...

sample.stop(this.meterRegistry.timer("portal.http.server.requests", "uri", uriTag, "success", isSuccess + ""));

```

```java

{"@timestamp":"2020-09-15T01:46:19.495Z","name":"portal_http_server_requests","type":"timer","success":"true","uri":"/api/v4/search-group","count":3,"sum":7802.271546,"mean":2600.757182,"max":7739.304955}

```

#### Kibana监控示例