K8S通过log-pilot采集应用日志（定制化tomcat多行）

说明：

    可以采集控制台日志，也可以采集log日志文件

说明文档：

搭建：

 wget https://acs-logging.oss-cn-hangzhou.aliyuncs.com/log-pilot.yml

    b.如果是自己搭建的elasticsearch集群，修改

        env:
          - name: "FILEBEAT_OUTPUT"
            value: "elasticsearch"  
          - name: "ELASTICSEARCH_HOST"
            value: "elasticsearch-api"  #修改为自己的es主机地址
          - name: "ELASTICSEARCH_PORT"
            value: "9200"             #修改es主机端口
          - name: "ELASTICSEARCH_USER"
            value: "elastic"            #如果有es账号密码配置，如果没有删除
          - name: "ELASTICSEARCH_PASSWORD"
            value: "changeme"

    c.应用：

kubectl apply -f log-pilot.yml

[root@iZwz9iqbscb8v5z26k21o3Z ~]# kubectl get pods -n kube-system -o wide|grep log-pilot
log-pilot-2pgck                                              1/1       Running   0          44m       172.16.1.148   cn-shenzhen.i-wz9iqbscb8v5z26k21o5   <none>
log-pilot-2qftj                                              1/1       Running   0          44m       172.16.0.131   cn-shenzhen.i-wz90ky5oya6sv3ftbtjp   <none>
log-pilot-8ln9x                                              1/1       Running   0          44m       172.16.3.151   cn-shenzhen.i-wz9iqbscb8v5z26k21o8   <none>
log-pilot-cdcsh                                              1/1       Running   0          44m       172.16.4.144   cn-shenzhen.i-wz9iqbscb8v5z26k21o3   <none>
log-pilot-jb7wl                                              1/1       Running   0          44m       172.16.4.22    cn-shenzhen.i-wz9iqbscb8v5z26k21o6   <none>
log-pilot-nks57                                              1/1       Running   0          44m       172.16.2.21    cn-shenzhen.i-wz9iqbscb8v5z26k21o2   <none>
log-pilot-nnpjq                                              1/1       Running   0          44m       172.16.1.3     cn-shenzhen.i-wz9j27sd0gs85wsdmrsk   <none>
log-pilot-nq2kp                                              1/1       Running   0          44m       172.16.0.8     cn-shenzhen.i-wz9ihky27ka1nnykchwp   <none>
log-pilot-pj7ms                                              1/1       Running   0          44m       172.16.3.18    cn-shenzhen.i-wz9iqbscb8v5z26k21o7   <none>
log-pilot-xwtnl                                              1/1       Running   0          44m       172.16.2.143   cn-shenzhen.i-wz9iqbscb8v5z26k21o4   <none>

apiVersion: v1
kind: Pod
metadata:
  name: tomcat
  namespace: default
  labels:
    name: tomcat
spec:
  containers:
  - image: tomcat
    name: tomcat-test
    volumeMounts:
    - mountPath: /usr/local/tomcat/logs
      name: accesslogs
    env:
     - name: aliyun_logs_catalina     ##索引名字是 catalina   ，catalina开始不能有_                     
       value: "stdout"             ##采集标准输出日志
     - name: aliyun_logs_access   ##索引名字是 access  
       value: "/usr/local/tomcat/logs/catalina.*.log"   ## 采集容器内日志文件 
  volumes:
    - name: accesslogs
      emptyDir: {}

多行日志：

        a.clone代码

git clone https://github.com/AliyunContainerService/log-pilot.git

vim log-pilot/assets/filebeat/filebeat.tpl

{{range .configList}}
- type: log
  enabled: true
  paths:
      - {{ .HostDir }}/{{ .File }}
  multiline.pattern: '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]'   #新增正则条件
  multiline.negate: true                                             #新增
  multiline.match: after                                             #新增
  multiline.max_lines: 10000                                         #新增
  scan_frequency: 10s
  fields_under_root: true
  {{if .Stdout}}
  docker-json: true
  {{end}}
  {{if eq .Format "json"}}
  json.keys_under_root: true
  {{end}}
  fields:
      {{range $key, $value := .Tags}}
      {{ $key }}: {{ $value }}
      {{end}}
      {{range $key, $value := $.container}}
      {{ $key }}: {{ $value }}
      {{end}}
  tail_files: false
  close_inactive: 2h
  close_eof: false
  close_removed: true
  clean_removed: true
  close_renamed: false

{{end}}

        c.从新打包镜像,第一次估计有点久            

cd log-pilot/ && ./build-image.sh

docker tag log-pilot registry.cn-shenzhen.aliyuncs.com/hr_plub/log-pilot:v2
docker push registry.cn-shenzhen.aliyuncs.com/hr_plub/log-pilot:v2