Logstash 之 filebeats

简单流程

  1. Download filebeats

  2. 配置 filebeat.yml

    1
    2
    3
    4
    5
    6
    filebeat.prospectors:
    	- input_type: log
    paths:
    	- /var/log/simple.log
    output.logstash:
    	hosts: ["localhost:5043"]

  3. 执行 sudo ./filebeat -e -c filebeat.yml -d "publish"， filebeat 会一直尝试连接 5043 端口

  4. 配置 logstash.conf

    1
    2
    3
    4
    5
    6
    7
    8
    input {
    	beats {
    			port => "5043"
    	}
    }
    output {
    	stdout { codec => rubydebug }
    }

  5. 验证配置是否合法 logstash -f logstash.conf --config.test_and_exit

  6. 启动 Logstash logstash -f logstash.conf --config.reload.automatic，这时会看到 simple.log 的日志被逐个输出
    添加 --config.reload.automatic 参数，当修改 logstash.conf 时，logstash 会自动重启

  7. 向 simple.log 追加一条日志 echo "这是一条追加日志" >> logstash-tutorial.log，可以看到 logstash 输出了新追加的内容

Apache 日志加工

  1. 配置 logstash.conf

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    input {
    	beats {
    			port => "5043"
    	}
    }

    filter {
    	# apache 日志过滤
    	grok {
    		match => { "message" => "%{COMBINEDAPACHELOG}"}
    	}
    	# 通过 ip 确认经纬度
    	geoip {
    		source => "clientip"
    	}
    }

    output {
    	stdout { codec => rubydebug }
    }

  2. 停止 filebeat ，删除 filebeat/data 文件，该文件记录了之前读取记录，所以需要删除掉

  3. 重新启动 filebeat sudo ./filebeat -e -c filebeat.yml -d "publish"

  4. 向 simple.log 追加一条 apache 日志 

    1
    10.63.9.126 - - [04/Jan/2016:05:13:42 +0000] "PUT /septenary/api/user/update HTTP/1.1" 200 203023 "http://septenary.cn/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

  5. logstash 输出类似以下内容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
{
	"request" => "/presentations/logstash-monitorama-2013/images/kibana-search.png",
	"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
	"geoip" => {
		"timezone" => "Europe/Moscow",
		"ip" => "83.149.9.216",
		"latitude" => 55.7485,
		"continent_code" => "EU",
		"city_name" => "Moscow",
		"country_code2" => "RU",
		"country_name" => "Russia",
		"country_code3" => "RU",
		"region_name" => "Moscow",
		"location" => [
			[0] 37.6184,
			[1] 55.7485
		],
		"postal_code" => "101194",
		"longitude" => 37.6184,
		"region_code" => "MOW"
	},
	"offset" => 650,
	"auth" => "-",
	"ident" => "-",
	"input_type" => "log",
	"verb" => "GET",
	"source" => "/Users/renyufeng/Documents/ES/practice-logstash/log/simple.log",
	"message" => "83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] \"GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1\" 200 203023 \"http://semicomplete.com/presentations/logstash-monitorama-2013/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36\"",
	"type" => "log",
	"tags" => [
		[0] "beats_input_codec_plain_applied"
	],
	"referrer" => "\"http://semicomplete.com/presentations/logstash-monitorama-2013/\"",
	"@timestamp" => 2017-03-10T08:09:49.065Z,
	"response" => "200",
	"bytes" => "203023",
	"clientip" => "83.149.9.216",
	"@version" => "1",
	"beat" => {
		"hostname" => "renyufengdeMacBook-Pro.local",
		"name" => "renyufengdeMacBook-Pro.local",
		"version" => "5.2.2"
	},
	"host" => "renyufengdeMacBook-Pro.local",
	"httpversion" => "1.1",
	"timestamp" => "04/Jan/2015:05:13:42 +0000"
}

索引日志到 ElasticSearch 中

  1. 配置 logstash.conf

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    input {
    	beats {
    		port => "5043"
    	}
    }

    filter {
    	grok {
    		match => { "message" => "%{COMBINEDAPACHELOG}"}
    	}
    	geoip {
    		source => "clientip"
    	}
    }

    output {
    	stdout { codec => rubydebug }
    		elasticsearch {
    			hosts => [ "localhost:9200" ]
    		}
    }

  2. 启动以下服务

    • 重启 filebeat
    • 启动 elasticsearch
    • 启动 kibana

  3. 此时运行的服务如下图

  4. 查看 elasticsearch 所有索引 curl -XGET 'localhost:9200/_cat/indices?v&pretty'

  5. 通过索引名查询入库的日志 curl -XGET 'localhost:9200/logstash-2017.03.10/_search?pretty'

  6. 同样可以在 kibana 上查看

官方指南