使用Filebeat来导入Apache访问日志并添加响应时间字段

1 年 ago
逸, 科
5 minutes

使用Filebeat的apache2模块导入访问日志时，默认情况下似乎没有导入响应时间，因此尝试对其进行处理以导入响应时间。

暂时只是个备忘录。

安装

将Elasticsearch、Kibana和Filebeat的安装所需仓库进行配置。

[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

安装下列物品。

yum install -y java-1.8.0-openjdk-devel elasticsearch kibana filebeat httpd

Elasticsearch的配置

将Elasticsearch配置为可从外部连接，并启动。
在/etc/elasticsearch/elasticsearch.yml进行设置。

- #network.host: 192.168.0.1
+  network.host: 0.0.0.0

[root@es ~]# systemctl restart elasticsearch
[root@es ~]# curl http://127.0.0.1:9200
{
  "name" : "4FH6KEs",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "HTHEBXGFQlO4TzABbjuvCg",
  "version" : {
    "number" : "6.3.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "eb782d0",
    "build_date" : "2018-06-29T21:59:26.107521Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

安装在Filebeat所需的Elasticsearch插件。

[root@es ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-user-agent
[root@es ~]# /usr/share/elasticsearch/bin/elasticsearch-plugin install ingest-geoip
[root@es ~]# systemctl restart elasticsearch

Kibana配置

进行设置以使Kibana能够从外部进行连接，并启动。

/kibana.yml位于/etc/kibana/目录下。

# Kibana is served by a back end server. This setting specifies the port to use.
- #server.port: 5601
+  server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
- #server.host: "localhost"
+  server.host: "0.0.0.0"

[root@es ~]# systemctl restart kibana
[root@es ~]# curl http://127.0.0.1:5601
<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>

Apache的配置

设置访问日志输出响应时间（微秒），并启动httpd服务。

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
-   LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
+   LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

[root@es ~]# systemctl restart httpd
[root@es ~]# curl http://127.0.0.1
[root@es ~]# tail /var/log/httpd/access_log
127.0.0.1 - - [13/Jul/2018:13:28:53 +0000] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0" 534

Filebeat的配置

将注册在apache2的pipeline的Elasticsearch进行修改，以获取响应时间。

以下是对 “/usr/share/filebeat/module/apache2/access/ingest/default.json”的中文本地化释义：

默认情况下，此路径指向的是 “/usr/share/filebeat/module/apache2/access/ingest/default.json” 文件。

{
  "description": "Pipeline for parsing Apache2 access logs. Requires the geoip and user_agent plugins.",
  "processors": [{
    "grok": {
      "field": "message",
      "patterns":[
+        "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"%{WORD:apache2.access.method} %{DATA:apache2.access.url} HTTP/%{NUMBER:apache2.access.http_version}\" %{NUMBER:apache2.access.response_code} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( \"%{DATA:apache2.access.referrer}\")?( \"%{DATA:apache2.access.agent}\")?( %{NUMBER:apache2.access.response_time})?",
-        "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"%{WORD:apache2.access.method} %{DATA:apache2.access.url} HTTP/%{NUMBER:apache2.access.http_version}\" %{NUMBER:apache2.access.response_code} (?:%{NUMBER:apache2.access.body_sent.bytes}|-)( \"%{DATA:apache2.access.referrer}\")?( \"%{DATA:apache2.access.agent}\")?",
        "%{IPORHOST:apache2.access.remote_ip} - %{DATA:apache2.access.user_name} \\[%{HTTPDATE:apache2.access.time}\\] \"-\" %{NUMBER:apache2.access.response_code} -"
        ],
      "ignore_missing": true
    }

在创建Elasticsearch的Filebeat索引中添加Apache2的response_time字段。

/etc/filebeat/fields.yml 可以进行改写为:
/filebeat/fields.yml

- key: apache2
  title: "Apache2"
  description: >
    Apache2 Module
  short_config: true
  fields:
    - name: apache2
      type: group
      description: >
        Apache2 fields.
      fields:
        - name: access
          type: group
          description: >
            Contains fields for the Apache2 HTTPD access logs.
          fields:
            - name: remote_ip
              type: keyword
              description: >
                Client IP address.
            - name: user_name
              type: keyword
              description: >
                The user name used when basic authentication is used.
            - name: method
              type: keyword
              example: GET
              description: >
                The request HTTP method.
            - name: url
              type: keyword
              description: >
                The request HTTP URL.
            - name: http_version
              type: keyword
              description: >
                The HTTP version.
            - name: response_code
              type: long
              description: >
                The HTTP response code.
+           - name: response_time
+             type: long
+             description: >
+               The HTTP response time.
            - name: body_sent.bytes
              type: long
              format: bytes
              description: >
                The number of bytes of the server response body.

启用apache2模块并设置日志文件路径。

[root@es ~]# filebeat modules enable apache2

/etc/filebeat模块.d/apache2.yml

- module: apache2
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
-   #var.paths:
+    var.paths: ["/var/log/httpd/access_log**"]

  # Error logs
  error:
-   enabled: true
+   enabled: false

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

启动 Filebeat。

[root@es ~]# systemctl restart filebeat

在Kibana进行确认

为了生成日志，预先访问几次。

[root@es ~]# curl http://127.0.0.1
[root@es ~]# curl http://127.0.0.1
[root@es ~]# curl http://127.0.0.1

在[服务器IP]:5601上连接到Kibana。
由于已创建了filebeat-6.3.1-2018.07.13的索引，需要将其注册到Kibana的索引模式中。

image.png
image.png

到头来

希望Filebeat的Apache2默认拥有响应时间字段…
可是，Elasticsearch和Kibana都升级到了6.3版本，新增了很多功能，变得更方便了。

✖
广告
将在 10 秒后关闭
bannerAds