使用Ansible和Docker，在CentOS 7上构建ELK（Elasticsearch，Logstash，Kibana）系统

12 月 ago
科, 雅
9 minutes

使用Ansible和Docker，在CentOS 7上构建ELK（Elasticsearch，Logstash，Kibana）。

制作的原因

    • ELKを手動で構築するのが面倒なので、自動化したかった。

 

    ELKを構築するために必要なコンフィグの編集、コンテナ起動時に様々なオプションを指定などを行うのが面倒なので、自動化したかった。

目标受众

    CentOS 7 にて、ELKの構築を簡単に行いたい方

前提条件的中文释义是指在考虑某种情况时，需要首先满足的必要条件或先决条件。

    • Ansibleはインストール済みであるとする

 

    • config、inventoryの設定は完了済みであるとする

 

    • 鍵生成、鍵交換、疎通確認は完了済みであるとする

 

    今回は、proxy環境下ではないものとする

执行环境

# cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)
# ansible --version
ansible 2.8.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /bin/ansible
  python version = 2.7.5 (default, Aug  7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

在这本playbook中所涵盖的内容简述

    • yum updateの実施

 

    • 古いdockerをアンインストール

 

    • 必要なパッケージのインストール

 

    • dockerリポジトリを追加

 

    • docker-ceをインストール

 

    • dockerの自動起動有効化 & dockerの起動

 

    • Elasticsearch, Logstash, Kibana, nginxの取得

 

    • ja-JP.jsonのダウンロード

 

    • logstash.conf, kibana.ymlの設定

 

    • nginx, Elasticsearch, Logstash, Kibanaコンテナの起動

 

    filebeatのインストール, 自動起動有効化, filebeatの起動

在执行本手册后获得的结果。

    • ELKが構築される

 

    • KibanaのGUIが日本語表示になる

 

    filebeatがインストールされた状態になる

执行本playbook后获得的环境版本信息。

項目VersionDocker18.06.1-ceElasticsearch7.4.0Logstash7.4.0Kibana7.4.0filebeat7.4.0nginx1.17.4

如何执行本Playbook

# ansible-playbook SetupELK.yml

構築するplaybook：Elasticsearch、Logstash、Kibana

- name: Setup Elasticsearch, Logstash, Kibana
  hosts: localhost
  vars:
    elk_shared_directory:
    - log/nginx
    - log/logstash
    - elasticsearch
    - elasticsearch/data/nodes
    - logstash/pipeline
    - kibana
    - kibana/config
  tasks:
  - name: Upgrade all packages
    yum:
      name: '*'
      state: latest

  - name: Remove the Docker package
    yum:
      name:
      - docker
      - docker-client
      - docker-client-latest
      - docker-common
      - docker-latest
      - docker-latest-logrotate
      - docker-logrotate
      - docker-selinux
      - docker-engine-selinux
      - docker-engine
      state: absent

  - name: Install a list of packages
    yum:
      name:
        - device-mapper-persistent-data
        - lvm2
        - yum-utils
        - python-pip
      state: latest

  - name: Add docker repo
    shell: "yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo"
    args:
      chdir: "/etc/yum.repos.d"
      creates: docker-ce.repo

  - name: Install a list of packages
    yum:
      name: docker-ce
      state: latest

  - name: Systemctl enabled docker & systemctl start docker
    systemd:
      name: docker
      state: started
      enabled: yes

  - name: Install docker python package
    pip:
      name: docker

  - name: Pull an elasticsearch image
    docker_image:
      name: docker.elastic.co/elasticsearch/elasticsearch
      source: pull
      tag: 7.4.0

  - name: Pull a logstash-docker image
    docker_image:
      name: docker.elastic.co/logstash/logstash
      source: pull
      tag: 7.4.0

  - name: Pull a kibana image
    docker_image:
      name: docker.elastic.co/kibana/kibana
      source: pull
      tag: 7.4.0

  - name: Pull a nginx image
    docker_image:
      name: nginx
      source: pull
      tag: latest

  - name: Ensure group "elk" exists
    group:
      name: elk
      state: present

  - name: Add the user 'elk' with a bash shell, appending the group 'elk' to the user's groups
    user:
      name: elk
      shell: /bin/bash
      groups: elk
      append: yes

  - name: Create a shared folder for ELK
    file:
      path: /usr/share/{{ item }}
      state: directory
      owner: elk
      group: elk
      mode: '0777'
    with_items: "{{ elk_shared_directory }}"

  - name: Download ja-JP.json
    get_url:
      url: https://github.com/elastic/kibana/raw/master/x-pack/plugins/translations/translations/ja-JP.json
      dest: /usr/share/kibana
      mode: '0755'
    become: yes
    become_user: root

  - name: Insert/Update kibana.yml in /usr/share/kibana/config/kibana.yml
    blockinfile:
      path: /usr/share/kibana/config/kibana.yml
      create: yes
      regexp: '^server.name'
      insertafter: '^#server.name'
      block: |
        server.name: kibana
        server.host: "0"
        elasticsearch.hosts: [ "http://elasticsearch:9200" ]
        xpack.monitoring.ui.container.elasticsearch.enabled: true
        i18n.locale: "ja-JP"

  - name: Insert/Update logstash.conf in /usr/share/logstash/pipeline/logstash.conf
    blockinfile:
      path: /usr/share/logstash/pipeline/logstash.conf
      create: yes
      regexp: '^input { '
      insertafter: '^#input { '
      block: |
        input {
          file {
            path => "/var/log/nginx/access.log"
            start_position => beginning
          }
        }
        filter {
          grok {
            match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
            remove_field => "message"
          }
          mutate {
            add_field => { "read_timestamp" => "%{@timestamp}" }
          }
          date {
            match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
            remove_field => "[nginx][access][time]"
          }
          useragent {
            source => "[nginx][access][agent]"
            target => "[nginx][access][user_agent]"
            remove_field => "[nginx][access][agent]"
          }
          geoip {
            source => "[nginx][access][remote_ip]"
            target => "[nginx][access][geoip]"
          }
        }
        output {
          elasticsearch {
            hosts => [ 'elasticsearch' ]
            index => "access_log1"
          }
        }

  - name: Create a nginx container
    docker_container:
      name: elk_nginx
      image: nginx
      restart_policy: always
      state: started
      ports:
        - 10080:80
      volumes:
        - /usr/share/log/nginx:/var/log/nginx

  - name: Create an elasticsearch container
    docker_container:
      name: elasticsearch
      image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
      restart_policy: always
      state: started
      ports:
        - 9200:9200
        - 9300:9300
      volumes:
        - /usr/share/elasticsearch:/usr/share/elasticsearch/data
      env:
        discovery.type: "single-node"

  - name: Create an logstash container
    docker_container:
      name: logstash
      image: docker.elastic.co/logstash/logstash:7.4.0
      restart_policy: always
      state: started
      ports:
        - 5044:5044
        - 9600:9600
      links:
        - elasticsearch:elasticsearch
      volumes:
        - /usr/share/logstash/pipeline:/usr/share/logstash/pipeline
        - /usr/share/log/logstash:/var/log

  - name: Create an kibana container
    docker_container:
      name: kibana
      image: docker.elastic.co/kibana/kibana:7.4.0
      restart_policy: always
      state: started
      ports:
        - 5601:5601
      links:
        - elasticsearch:elasticsearch
      volumes:
        - /usr/share/kibana/ja-JP.json:/usr/share/kibana/node_modules/x-pack/plugins/translations/translations/ja-JP.json
        - /usr/share/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml

  - name: Download filebeat
    get_url:
      url: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.4.0-x86_64.rpm
      dest: /usr/local/src
      mode: '0755'
    become: yes
    become_user: root

  - name: Install filebeat
    yum:
      name: /usr/local/src/filebeat-7.4.0-x86_64.rpm
      state: present

  - name: filebeat modules enable elasticsearch
    shell: filebeat modules enable elasticsearch

  - name: Waits for port 9200 of localhost to close active connections, don't start checking for 10 seconds
    wait_for:
      host: localhost
      port: 9200
      timeout: 300
      delay: 10
      state: drained

  - name: Pause for 3 minutes to build app cache.
    pause:
      minutes: 3

  - name: filebeat setup
    shell: filebeat setup

  - name: Systemctl enabled filebeat & systemctl start filebeat
    systemd:
      name: filebeat
      state: started
      enabled: yes

执行Playbook时的日志记录

# ansible-playbook SetupELK.yml 

PLAY [Setup Elasticsearch, Logstash, Kibana] **********************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Upgrade all packages] ***************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Remove the Docker package] **********************************************************************************************************************************************************************************
changed: [localhost]

TASK [Install a list of packages] *********************************************************************************************************************************************************************************
ok: [localhost]

TASK [Add docker repo] ********************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Install a list of packages] *********************************************************************************************************************************************************************************
changed: [localhost]

TASK [Systemctl enabled docker & systemctl start docker] **********************************************************************************************************************************************************
changed: [localhost]

TASK [Install docker python package] ******************************************************************************************************************************************************************************
ok: [localhost]

TASK [Pull an elasticsearch image] ********************************************************************************************************************************************************************************
changed: [localhost]

TASK [Pull a logstash-docker image] *******************************************************************************************************************************************************************************
changed: [localhost]

TASK [Pull a kibana image] ****************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Pull a nginx image] *****************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Ensure group "elk" exists] **********************************************************************************************************************************************************************************
ok: [localhost]

TASK [Add the user 'elk' with a bash shell, appending the group 'elk' to the user's groups] ***********************************************************************************************************************
ok: [localhost]

TASK [Create a shared folder for ELK] *****************************************************************************************************************************************************************************
changed: [localhost] => (item=log/nginx)
changed: [localhost] => (item=log/logstash)
changed: [localhost] => (item=elasticsearch)
changed: [localhost] => (item=elasticsearch/data/nodes)
changed: [localhost] => (item=logstash/pipeline)
changed: [localhost] => (item=kibana)
changed: [localhost] => (item=kibana/config)

TASK [Download ja-JP.json] ****************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Insert/Update kibana.yml in /usr/share/kibana/config/kibana.yml] ********************************************************************************************************************************************
changed: [localhost]

TASK [Insert/Update logstash.conf in /usr/share/logstash/pipeline/logstash.conf] **********************************************************************************************************************************
changed: [localhost]

TASK [Create a nginx container] ***********************************************************************************************************************************************************************************
changed: [localhost]

TASK [Create an elasticsearch container] **************************************************************************************************************************************************************************
changed: [localhost]

TASK [Create an logstash container] *******************************************************************************************************************************************************************************
changed: [localhost]

TASK [Create an kibana container] *********************************************************************************************************************************************************************************
changed: [localhost]

TASK [Download filebeat] ******************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Install filebeat] *******************************************************************************************************************************************************************************************
changed: [localhost]

TASK [filebeat modules enable elasticsearch] **********************************************************************************************************************************************************************
changed: [localhost]

TASK [Waits for port 9200 of localhost to close active connections, don't start checking for 10 seconds] **********************************************************************************************************
ok: [localhost]

TASK [Pause for 3 minutes to build app cache.] ********************************************************************************************************************************************************************
Pausing for 180 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]

TASK [filebeat setup] *********************************************************************************************************************************************************************************************
changed: [localhost]

TASK [Systemctl enabled filebeat & systemctl start filebeat] ******************************************************************************************************************************************************
changed: [localhost]

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=29   changed=20   unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

确认容器的状态和镜像

首先，让我们检查一下容器的状态和镜像。

# docker ps -a
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
67080a477ab5        docker.elastic.co/kibana/kibana:7.4.0                 "/usr/local/bin/dumb…"   17 minutes ago      Up 17 minutes       0.0.0.0:5601->5601/tcp                           kibana
424e6b9642a5        docker.elastic.co/logstash/logstash:7.4.0             "/usr/local/bin/dock…"   17 minutes ago      Up 17 minutes       0.0.0.0:5044->5044/tcp, 0.0.0.0:9600->9600/tcp   logstash
cc6b8a1cd7fc        docker.elastic.co/elasticsearch/elasticsearch:7.4.0   "/usr/local/bin/dock…"   17 minutes ago      Up 17 minutes       0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch
fad3c5ec3136        nginx                                                 "nginx -g 'daemon of…"   17 minutes ago      Up 17 minutes       0.0.0.0:10080->80/tcp                            elk_nginx

# docker image list
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
nginx                                           latest              5a9061639d0a        5 days ago          126MB
docker.elastic.co/logstash/logstash             7.4.0               c2c1ac6b995b        3 weeks ago         889MB
docker.elastic.co/kibana/kibana                 7.4.0               0328df36f79f        3 weeks ago         1.1GB
docker.elastic.co/elasticsearch/elasticsearch   7.4.0               dd156dd42341        3 weeks ago         859MB

确认对Kibana的访问

kibana_002.png

确认对Elasticsearch的访问

elasticsearch_001.png

确认访问Nginx

nginx_001.png

汇总

使用Ansible和Docker，在CentOS 7上成功地部署了ELK（Elasticsearch, Logstash, Kibana）。接下来，我打算通过阅读ELK相关的文章，使用curl命令发送虚拟数据来进行可视化和仪表板的创建，以进一步深入了解ELK的使用方法和理解。

请提供参考网址

这次使用的模块

    • 【docs.ansible.com】yum – Manages packages with the yum package manager

 

    • 【docs.ansible.com】shell – Execute shell commands on targets

 

    • 【docs.ansible.com】systemd – Manage services

 

    • 【docs.ansible.com】pip – Manages Python library dependencies

 

    • 【docs.ansible.com】docker_image – Manage docker images

 

    • 【docs.ansible.com】group – Add or remove groups

 

    • 【docs.ansible.com】user – Manage user accounts

 

    • 【docs.ansible.com】file – Manage files and file properties

 

    • 【docs.ansible.com】get_url – Downloads files from HTTP, HTTPS, or FTP to node

 

    • 【docs.ansible.com】blockinfile – Insert/update/remove a text block surrounded by marker lines

 

    • 【docs.ansible.com】docker_container – manage docker containers

 

    • 【docs.ansible.com】wait_for – Waits for a condition before continuing

 

    【docs.ansible.com】pause – Pause playbook execution

Ansible相关

    Ansibleのタスクを適度に待たせて適切なタイミングで実行したい

Docker、Elasticsearch、Logstash和Kibana相关

    • Docker @ Elastic

 

    • CentOS7.5に最新バージョンのDockerをインストールする

 

    • vagrant上のcentosにansibleでdocker-ceをインストールする

 

    • DockerでELK(Elasticsearch Logstash Kibana) を構築してみました

 

    • Elasticsearch + KibanaをDocker Composeで動かす

 

    • Install Elasticsearch with Dockeredit

 

    • Running Kibana on Dockeredit

 

    • Configuring Kibana on Dockeredit

 

    • Specify data directory permissions for Docker containers like Elasticsearch

 

    • Kibana の 日本語化

 

    • KIBANA 日本語対応

 

    • はじめての Elasticsearch

 

    Elasticsearch、Logstash 、Kibana、によるログの可視化　～価値のある情報を導き出す方法のご紹介 ～
✖
广告
将在 10 秒后关闭
bannerAds