运维进阶训练营 -W06H
作者：赤色闪电
- 2022-12-03 北京
本文字数：2810 字
阅读完需：约 9 分钟
1、基于 logstash filter 功能将 nginx 默认的访问日志及 error log 转换为 json 格式并写入 elasticsearch
input { file { path => "/apps/nginx/logs/access.log" type => "nginx-accesslog" stat_interval => "1" start_position => "beginning" }
file { path => "/apps/nginx/logs/error.log" type => "nginx-errorlog" stat_interval => "1" start_position => "beginning" }
}
filter { if [type] == "nginx-accesslog" { grok { match => { "message" => ["%{IPORHOST:clientip} - %{DATA:username} \[%{HTTPDATE:request-time}\] \"%{WORD:request-method} %{DATA:request-uri} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent_bytes} \"%{DATA:referrer}\" \"%{DATA:useragent}\""] } remove_field => "message" add_field => { "project" => "magedu"} } mutate { convert => [ "[response_code]", "integer"] } } if [type] == "nginx-errorlog" { grok { match => { "message" => ["(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:loglevel}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IPV4:clientip}, server: %{GREEDYDATA:server}, request: \"(?:%{WORD:request-method} %{NOTSPACE:request-uri}(?: HTTP/%{NUMBER:httpversion}))\", host: %{GREEDYDATA:domainname}"]} remove_field => "message" } }}
output { if [type] == "nginx-accesslog" { elasticsearch { hosts => ["172.31.2.101:9200"] index => "magedu-nginx-accesslog-%{+yyyy.MM.dd}" user => "magedu" password => "123456" }}
if [type] == "nginx-errorlog" { elasticsearch { hosts => ["172.31.2.101:9200"] index => "magedu-nginx-errorlog-%{+yyyy.MM.dd}" user => "magedu" password => "123456" }}
}input { file { path => "/var/log/nginx/access.log" start_position => "end" type => "nginx-json-accesslog" stat_interval => "1" codec => json }}
output { if [type] == "nginx-json-accesslog" { elasticsearch { hosts => ["172.31.2.101:9200"] index => "nginx-accesslog-2.107-%{+YYYY.MM.dd}" user => "magedu" password => "123456" }}}复制代码
2、基于 logstash 收集 json 格式的 nginx 访问日志
input { file { path => "/var/log/nginx/access.log" start_position => "end" type => "nginx-json-accesslog" stat_interval => "1" codec => json }}
output { if [type] == "nginx-json-accesslog" { elasticsearch { hosts => ["172.31.2.101:9200"] index => "nginx-accesslog-2.107-%{+YYYY.MM.dd}" user => "magedu" password => "123456" }}}复制代码
3、基于 logstash 收集 java 日志并实现多行合并
input { file { path => "/data/eslogs/magedu-es-cluster1.log" type => "eslog" stat_interval => "1" start_position => "beginning" codec => multiline { #pattern => "^\[" pattern => "^\[[0-9]{4}\-[0-9]{2}\-[0-9]{2}" negate => "true" what => "previous" } }}
output { if [type] == "eslog" { elasticsearch { hosts => ["172.31.2.102:9200"] index => "magedu-eslog-%{+YYYY.ww}" user => "magedu" password => "123456" }}}
复制代码
4、基于 logstash 收集 syslog 类型日志 (以 haproxy 替代网络设备)
input{ syslog { type => "rsyslog-haproxy" port => "514" #监听一个本地的端口}}
output{ if [type] == "rsyslog-haproxy" { elasticsearch { hosts => ["172.31.2.102:9200"] index => "magedu-rsyslog-haproxy-%{+YYYY.ww}" user => "magedu" password => "123456" }}}复制代码
5、logstash 收集日志并写入 Redis、再通过其它 logstash 消费至 elasticsearch 并保持 json 格式日志的解析
input { file { path => "/var/log/nginx/access.log" type => "magedu-nginx-accesslog" start_position => "beginning" stat_interval => "1" codec => "json" #对json格式日志进行json解析 }
file { path => "/apps/nginx/logs/error.log" type => "magedu-nginx-errorlog" start_position => "beginning" stat_interval => "1" }}
filter { if [type] == "magedu-nginx-errorlog" { grok { match => { "message" => ["(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:loglevel}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IPV4:clientip}, server: %{GREEDYDATA:server}, request: \"(?:%{WORD:request-method} %{NOTSPACE:request-uri}(?: HTTP/%{NUMBER:httpversion}))\", host: %{GREEDYDATA:domainname}"]} remove_field => "message" #删除源日志 } }}
output { if [type] == "magedu-nginx-accesslog" { redis { data_type => "list" key => "magedu-nginx-accesslog" host => "172.31.2.105" port => "6379" db => "0" password => "123456" } } if [type] == "magedu-nginx-errorlog" { redis { data_type => "list" key => "magedu-nginx-errorlog" host => "172.31.2.105" port => "6379" db => "0" password => "123456" } }}
复制代码
6、基于 docker-compose 部署单机版本 ELK
基于docker-compose部署ELK： # git clone https://gitee.com/jiege-gitee/elk-docker-compose.git # cd docker-elk-compose # docker-compose up -d elasticsearch #运行elasticsearch容器 # docker exec -it elasticsearch /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive #设置账户密码magedu123 修改kibana连接elasticsearch的账户密码： # vim kibana/config/kibana.yml 修改logstash连接elasticsearch的账户密码 # vim logstash/config/logstash.conf 修改Logstash输入输出规则 # vim logstash/config/logstash.conf # docker-compose up -d复制代码
划线
评论
复制
发布于: 刚刚阅读数: 4
版权声明: 本文为 InfoQ 作者【赤色闪电】的原创文章。
原文链接:【http://xie.infoq.cn/article/fa6f2859b777d96f9f16d2716】。未经作者许可，禁止转载。
赤色闪电
关注
还未添加个人签名 2018-05-30 加入
还未添加个人简介
评论