ELK-logstash 使用总结
作者：忙着长大#
- 2022-12-07 北京
本文字数：2117 字
阅读完需：约 7 分钟
1.基于 logstash filter 功能将 nginx 默认的访问日志及 error log 转换为 json 格式并写入 elasticsearch
nginx 配置
worker_processes 1;error_log logs/error.log;events { worker_connections 1024;}http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log logs/access.log main; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root html; index index.html index.htm; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } }}复制代码
logstash 配置
input { file { path => "/usr/local/nginx/logs/access.log" type => "nginx-accesslog" stat_interval => "1" start_position => "beginning" }
file { path => "/usr/local/nginx/logs/error.log" type => "nginx-errorlog" stat_interval => "1" start_position => "beginning" }
}
filter { if [type] == "nginx-accesslog" { grok { match => { "message" => ["%{IPORHOST:clientip} - %{DATA:username} \[%{HTTPDATE:request-time}\] \"%{WORD:request-method} %{DATA:request-uri} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent_bytes} \"%{DATA:referrer}\" \"%{DATA:useragent}\""] } remove_field => "message" add_field => { "project" => "test"} } mutate { convert => [ "[response_code]", "integer"] } } if [type] == "nginx-errorlog" { grok { match => { "message" => ["(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:loglevel}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:message}, client: %{IPV4:clientip}, server: %{GREEDYDATA:server}, request: \"(?:%{WORD:request-method} %{NOTSPACE:request-uri}(?: HTTP/%{NUMBER:httpversion}))\", host: %{GREEDYDATA:domainname}"]} remove_field => "message" } }}
output { if [type] == "nginx-accesslog" { elasticsearch { hosts => ["192.168.131.131:9200"] index => "test-nginx-accesslog-%{+yyyy.MM.dd}" user => "test" password => "123456" }}
if [type] == "nginx-errorlog" { elasticsearch { hosts => ["192.168.131.131:9200"] index => "test-nginx-errorlog-%{+yyyy.MM.dd}" user => "test" password => "123456" }}
}复制代码
2.基于 logstash 收集 json 格式的 nginx 访问日志
nginx 配置文件
worker_processes 1;error_log logs/error.log;events { worker_connections 1024;}http { include mime.types; default_type application/octet-stream; log_format access_json '{"@timestamp":"$time_iso8601",' '"host":"$server_addr",' '"clientip":"$remote_addr",' '"size":$body_bytes_sent,' '"responsetime":$request_time,' '"upstreamtime":"$upstream_response_time",' '"upstreamhost":"$upstream_addr",' '"http_host":"$host",' '"uri":"$uri",' '"domain":"$host",' '"xff":"$http_x_forwarded_for",' '"referer":"$http_referer",' '"tcp_xff":"$proxy_protocol_addr",' '"http_user_agent":"$http_user_agent",' '"status":"$status"}'; access_log logs/access.log access_json; sendfile on; keepalive_timeout 65; server { listen 80; server_name localhost; location / { root html; index index.html index.htm; } error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } }}复制代码
logstash 配置文件
input { file { path => "/usr/local/nginx/logs/access.log" start_position => "end" type => "nginx-json-accesslog" stat_interval => "1" codec => json }}
output { if [type] == "nginx-json-accesslog" { elasticsearch { hosts => ["192.168.131.133:9200"] index => "nginx-accesslog-133-%{+YYYY.MM.dd}" user => "test" password => "123456" }}}复制代码
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
划线
评论
复制
发布于: 刚刚阅读数: 2
忙着长大#
关注
还未添加个人签名 2022-02-09 加入
还未添加个人简介
评论