ELK单台日志收集系统的搭建

环境：

    1.nginx web server：centos 6.7 X86-64  192.168.80.101

    2.ELK：centos 6.7 X86-64   IP:192.168.80.100

安装：

    elk 192.168.80.100安装

        先安装JDK

        

yum install -y java-1.8.0-openjdk.x86_64

        1.安装Elasticsearch     

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.rpm
yum localinstall elasticsearch-5.1.1.rpm -y

          修改Elasticsearch 配置文件：

vim /etc/elasticsearch/elasticsearch.yml
cluster.name: myelk                          #设置群集名
node.name: node-1                          #设置节点名
node.attr.rack: r1               
path.data: /data/elk                          #存储elk文件路径
path.logs: /data/logs                          #存储日志路径
network.host: 0.0.0.0                         #监控IP
http.port: 9200                                  #端口

          建立目录设置目录权限，启动Elasticsearch (启动报错的话见最下面错误解决)

mkdir -pv /data/{elk,logs}
chown -R elasticsearch.elasticsearch /data/
/etc/init.d/elasticsearch start

查看监听状态

[root@cloud indices]# netstat -tlunp|grep java
tcp        0      0 :::9200                     :::*                        LISTEN      2341/java           
tcp        0      0 :::9300                     :::*                        LISTEN      2341/java 

装完可以web访问可以看到一个简单的web界面，表示正常

http://192.168.80.100:9200/

         2.安装kibana     

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.1.1-x86_64.rpm
yum localinstall tools/kibana-5.1.1-x86_64.rpm -y

          修改配置文件

/etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.80.100:9200"

          启动：

/etc/init.d/kibana start
[root@cloud indices]# netstat -tlunp
tcp        0      0 0.0.0.0:5601                0.0.0.0:*                   LISTEN      2478/node

  访问web界面可以看到http://192.168.80.100:9200/

2.nginx web服务器安装logstash收集nginx访问日志，并发给elk服务器，nginx访问日志路径/home/wwwlogs/y.log

yum install -y java-1.8.0-openjdk.x86_64
wegt https://artifacts.elastic.co/downloads/logstash/logstash-5.1.1.rpm
yum localinstall -y logstash-5.1.1.rpm 

     增加配置文件监控日志文件发给elk

[root@VM_87_96_centos ~]# vim /etc/logstash/conf.d/nginx.conf
input{
        file{
                type => "nginxaccesslog"
                path => "/home/wwwlogs/y.log"
                start_position => "beginning"
        }}
output{
        file{
                path => "/tmp/123.txt"
        }
        elasticsearch{
                hosts => ["203.160.54.16"]
                index => "nginx-access-%{+yyyy.MM.dd}"
        }}

 /home/wwwlogs/y.log文件需要有logstash权限，不然无法收集

       启动

[root@VM_87_96_centos bin]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/nginx.conf &

最后，打开http://192.168.80.100:5601/配置监控

 

最后界面

elasticsearch常见错误：

1.内存小于2G需要修改参数，不然报错

        -Xmx768m

2.启动报ERROR: bootstrap checks failed

max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]
解决：切换到root用户，编辑limits.conf 添加类似如下内容
vi /etc/security/limits.conf 

添加如下内容:

* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

3.max number of threads [1024] for user [lish] likely too low, increase to at least [2048]
解决：切换到root用户，进入limits.d目录下修改配置文件。
vi /etc/security/limits.d/90-nproc.conf 
修改如下内容：

* soft nproc 1024
#修改为
* soft nproc 2048

4.max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
解决：切换到root用户修改配置sysctl.conf
vi /etc/sysctl.conf 
添加下面配置：
vm.max_map_count=655360
并执行命令：
sysctl -p
然后，重新启动elasticsearch，即可启动成功。