filebeat收集k8s日志

1、Filebeat概述

????Filebeat是用于轉發和集中日志數據的輕量級傳送程序。作為服務器上的代理安裝，Filebeat監視您指定的日志文件或位置，收集日志事件，并將其轉發給[Elasticsearch]或 [Logstash]進行索引。

????Filebeat的工作方式如下：啟動Filebeat時，它將啟動一個或多個輸入，這些輸入將在為日志數據指定的位置中查找。對于Filebeat所找到的每個日志，Filebeat都會啟動收集器。每個收割機都讀取單個日志以獲

取新內容，并將新日志數據發送到libbeat，libbeat將聚集事件并將聚集的數據發送到為Filebeat配置的輸出。

2、在Kubernetes上運行Filebeat

????將Filebeat部署為DaemonSet，以確保集群的每個節點上都有一個正在運行的實例。Docker日志主機文件夾（/var/lib/docker/containers）安裝在Filebeat容器上。Filebeat會開始輸入文件，并在文件出現在

文件夾中后立即開始收集它們。

????這里使用官方提供的方式部署：

curl -L -O https://raw.githubusercontent.com/elastic/beats/7.5/deploy/kubernetes/filebeat-kubernetes.yaml

3、設置

默認情況下，Filebeat將事件發送到現有的Elasticsearch部署（如果存在）。要指定其他目標，請在清單文件中更改以下參數：

env:
-?name:?ELASTICSEARCH_HOST
??value:?elasticsearch
-?name:?ELASTICSEARCH_PORT
??value:?"9200"
-?name:?ELASTICSEARCH_USERNAME
??value:?elastic
-?name:?ELASTICSEARCH_PASSWORD
??value:?changeme
-?name:?ELASTIC_CLOUD_ID
??value:
-?name:?ELASTIC_CLOUD_AUTH
??value:

輸出到logstash：

---
apiVersion:?v1
kind:?ConfigMap
metadata:
??name:?filebeat-config
??namespace:?kube-system
??labels:
????k8s-app:?filebeat
data:
??filebeat.yml:?|-
????filebeat.config:
??????inputs:
????????#?Mounted?`filebeat-inputs`?configmap:
????????path:?${path.config}/inputs.d/*.yml
????????#?Reload?inputs?configs?as?they?change:
????????reload.enabled:?false
??????modules:
????????path:?${path.config}/modules.d/*.yml
????????#?Reload?module?configs?as?they?change:
????????reload.enabled:?false

????#?To?enable?hints?based?autodiscover,?remove?`filebeat.config.inputs`?configuration?and?uncomment?this:
????#filebeat.autodiscover:
????#??providers:
????#????-?type:?kubernetes
????#??????hints.enabled:?true

????processors:
??????-?add_cloud_metadata:

????#cloud.id:?${ELASTIC_CLOUD_ID}
????#cloud.auth:?${ELASTIC_CLOUD_AUTH}

????#output.elasticsearch:
??????#hosts:?['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
??????#username:?${ELASTICSEARCH_USERNAME}
??????#password:?${ELASTICSEARCH_PASSWORD}
????output.logstash:
??????hosts:?["192.168.0.104:5044"]
---
apiVersion:?v1
kind:?ConfigMap
metadata:
??name:?filebeat-inputs
??namespace:?kube-system
??labels:
????k8s-app:?filebeat
data:
??kubernetes.yml:?|-
????-?type:?log????#設置類型為log
????????paths:
??????????-?/var/lib/docker/containers/*/*.log
????????#fields:
??????????#app:?k8s
??????????#type:?docker-log
????????fields_under_root:?true
????????json.keys_under_root:?true
????????json.overwrite_keys:?true
????????encoding:?utf-8
????????fields.sourceType:?docker-log?????????#索引名格式
---
apiVersion:?extensions/v1beta1
kind:?DaemonSet
metadata:
??name:?filebeat
??namespace:?kube-system
??labels:
????k8s-app:?filebeat
spec:
??template:
????metadata:
??????labels:
????????k8s-app:?filebeat
????spec:
??????serviceAccountName:?filebeat
??????terminationGracePeriodSeconds:?30
??????containers:
??????-?name:?filebeat
????????image:?docker.elastic.co/beats/filebeat:6.5.4???????#提前準備好鏡像，需要***下載
????????args:?[
??????????"-c",?"/etc/filebeat.yml",
??????????"-e",
????????]
????????securityContext:
??????????runAsUser:?0
??????????#?If?using?Red?Hat?OpenShift?uncomment?this:
??????????#privileged:?true
????????resources:
??????????limits:
????????????memory:?200Mi
??????????requests:
????????????cpu:?100m
????????????memory:?100Mi
????????volumeMounts:
????????-?name:?config
??????????mountPath:?/etc/filebeat.yml
??????????readOnly:?true
??????????subPath:?filebeat.yml
????????-?name:?inputs
??????????mountPath:?/usr/share/filebeat/inputs.d
??????????readOnly:?true
????????-?name:?data
??????????mountPath:?/usr/share/filebeat/data
????????-?name:?varlibdockercontainers
??????????mountPath:?/var/lib/docker/containers
??????????readOnly:?true
??????volumes:
??????-?name:?config
????????configMap:
??????????defaultMode:?0600
??????????name:?filebeat-config
??????-?name:?varlibdockercontainers
????????hostPath:
??????????path:?/var/lib/docker/containers
??????-?name:?inputs
????????configMap:
??????????defaultMode:?0600
??????????name:?filebeat-inputs
??????#?data?folder?stores?a?registry?of?read?status?for?all?files,?so?we?don't?send?everything?again?on?a?Filebeat?pod?restart
??????-?name:?data
????????hostPath:
??????????path:?/var/lib/filebeat-data
??????????type:?DirectoryOrCreate
---
apiVersion:?rbac.authorization.k8s.io/v1beta1
kind:?ClusterRoleBinding
metadata:
??name:?filebeat
subjects:
-?kind:?ServiceAccount
??name:?filebeat
??namespace:?kube-system
roleRef:
??kind:?ClusterRole
??name:?filebeat
??apiGroup:?rbac.authorization.k8s.io
---
apiVersion:?rbac.authorization.k8s.io/v1beta1
kind:?ClusterRole
metadata:
??name:?filebeat
??labels:
????k8s-app:?filebeat
rules:
-?apiGroups:?[""]?#?""?indicates?the?core?API?group
??resources:
??-?namespaces
??-?pods
??verbs:
??-?get
??-?watch
??-?list
---
apiVersion:?v1
kind:?ServiceAccount
metadata:
??name:?filebeat
??namespace:?kube-system
??labels:
????k8s-app:?filebeat

創建并運行：

能看到上面日志，則表示啟動成功。

4、排錯

如果沒啟動成功，查看logstash的日志，報錯如下

[2019-12-20T19:53:14,049][ERROR][logstash.outputs.elasticsearch]?Could?not?index?event?to?Elasticsearch.?{:status=>400,?:action=>["index",?{:_id=>nil,?:_index=>"dev-%{[fields][sourceType]}-2019-12-20",?:_type=>"doc",?:routing=>nil},?
#<LogStash::Event:0x4c8737db>],?:response=>{"index"=>{"_index"=>"dev-%{[fields][sourceType]}-2019-12-20",?"_type"=>"doc",?"_id"=>nil,?"status"=>400,?"error"=>{"type"=>"invalid_index_name_exception",?"reason"=>"Invalid?index?name?
[dev-%{[fields][sourceType]}-2019-12-20],?must?be?lowercase",?"index_uuid"=>"_na_",?"index"=>"dev-%{[fields][sourceType]}-2019-12-20"}}}}
[2019-12-20T19:53:14,049][ERROR][logstash.outputs.

原因是logstash中output的index不能有大寫：

我原來的logstash的conf文件

output?{
????elasticsearch?{
????????????hosts?=>?["localhost:9200"]
????????????index?=>?'%{platform}-%{[fields][sourceType]}-%{+YYYY-MM-dd}'
????????????template?=>?"/opt/logstash-6.5.2/config/af-template.json"
????????????template_overwrite?=>?true
????}
}

修改后的

output?{
????elasticsearch?{
????????????hosts?=>?["localhost:9200"]
????????????index?=>?"k8s-log-%{+YYYY.MM.dd}"
????????????template?=>?"/opt/logstash-6.5.2/config/af-template.json"
????????????template_overwrite?=>?true
????}
}

完美結束！無坑