用于Elastalert的Kibana插件可将警报发送到Gmail邮箱

Elasticsearch中有一种基于输入数据发送警报的机制，即elastalert。
https://elastalert.readthedocs.io/en/latest/index.html

虽然不是Elasticsearch默认的公式功能，但由于广泛使用，我想试试看。
另外，这次是通过Kibana插件执行的。

机器规格：
操作系统：Ubuntu 18.04，安装在VirtualBox上
内存：8196 MB
CPU核心数：2
Elasticsearch和Kibana版本：7.6.2

测试环境的设置

为了在本地环境中运行，我们可以方便地使用Docker来设置环境。

文件夹结构

├── docker-compose.yml
├── elastalert
│   ├── bin
│   │   ├── elastalert-start.sh
│   │   └── elastic_search_status.sh
│   ├── config
│   │   ├── config.json
│   │   ├── elastalert-test.yaml
│   │   └── elastalert.yaml
│   ├── Dockerfile
│   ├── pass
│   │   └── smtp_auth_user.yaml
│   ├── rules
│   └── rule_templates
├── elasticsearch
│   ├── config
│   │   └── elasticsearch.yml
│   └── Dockerfile
├── kibana
│   ├── config
│   │   └── kibana.xml
│   ├── Dockerfile
│   └── plugin
│       └── elastalert-kibana-plugin-1.1.0-7.6.2.zip

Elasticsearch: 弹性搜索

FROM docker.elastic.co/elasticsearch/elasticsearch:7.6.2

cluster.name: "docker-cluster"
node.name: "node1"
node.master: true
node.data: true
network.host: 0.0.0.0
network.publish_host: _local_
discovery.seed_hosts: ["172.40.0.2"]
cluster.initial_master_nodes: ["node1"]

Kibana 可视化工具

FROM docker.elastic.co/kibana/kibana:7.6.2

在 kibana.xml 文件中声明使用 elastalert-kibana-plugin。

server.name: kibana
server.host: "0"
elasticsearch.hosts: [ "http://doc-elastic101:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true
elasticsearch.requestTimeout: 60000

# elastalert-kibana-plugin
elastalert-kibana-plugin.serverHost: elastalert
elastalert-kibana-plugin.serverPort: 3030

插件/elastalert-kibana-plugin-1.1.0-7.6.2.zip 将基于前一个版本进行创建。

# Download necessary files
cd /tmp
curl -L -O https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.1.0/elastalert-kibana-plugin-1.1.0-7.5.0.zip
curl -L -O https://raw.githubusercontent.com/mmguero-dev/Malcolm/development/kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# update elasticsearch package to 7.6.2
mv elastalert.js elastalert-server-routes.js
mv elastalert-kibana-plugin-1.1.0-7.5.0.zip elastalert-kibana-plugin-1.1.0-7.6.2.zip
unzip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json
sed -i "s/7\.5\.0/7\.6\.2/g" kibana/elastalert-kibana-plugin/package.json
mkdir -p kibana/elastalert-kibana-plugin/server/routes/
mv /tmp/elastalert-server-routes.js kibana/elastalert-kibana-plugin/server/routes/elastalert.js
zip elastalert-kibana-plugin-1.1.0-7.6.2.zip kibana/elastalert-kibana-plugin/package.json kibana/elastalert-kibana-plugin/server/routes/elastalert.js

# delete remaining directory
rm -rf kibana

# copy the created package to your workspace
cp /tmp/elastalert-kibana-plugin-1.1.0-7.6.2.zip {your_workspace}/kibana/plugin

弹性警报

FROM bitsensor/elastalert:3.0.0-beta.1

USER root

RUN apk update && \
    apk add bash curl && \
    rm -rf /var/cache/apk/*

ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/elastalert-start.sh 

USER node

ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]

#!/bin/bash

set -e

echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."

elastic_search_status.sh

echo "Starting ElastAlert!"
npm start

#!/bin/bash

set -e

if [ $# -gt 0 ]; then
  ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
  ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
  ES_URL="http://$ES_HOST:$ES_PORT"
else
  ES_URL="http://doc-elastic101:9200"
fi

until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
  # printf '+' >&2
  sleep 1
done

echo "Elasticsearch is up and healthy at "$ES_URL"" >&2

{
  "appName": "elastalert-server",
  "port": 3030,
  "wsport": 3333,
  "elastalertPath": "/opt/elastalert",
  "verbose": true,
  "es_debug": false,
  "debug": false,
  "rulesPath": {
    "relative": true,
    "path": "/rules"
  },
  "templatesPath": {
    "relative": true,
    "path": "/rule_templates"
  },
  "es_host": "elasticsearch",
  "es_port": 9200,
  "writeback_index": "elastalert_status"
}

# NOTE: This config is used when testing a rule

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: doc-elastic101

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: doc-elastic101

# The elasticsearch port
es_port: 9200

# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules

# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
  seconds: 5

# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
  minutes: 1

# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch

# Connect with TLS to elasticsearch
#use_ssl: True

# Verify TLS certificates
#verify_certs: True

# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET

# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword

# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status

# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
  days: 2

稍后会提到，由于我想要发送邮件到gmail，我会准备一个包含验证用户名和密码的文件。

user: "xxxx@gmail.com"
password: "xxxx"

version: '3.2'
services:
    elasticsearch:
        build:
            context: elasticsearch/
        hostname: doc-elastic101
        container_name: elastic1
        ports:
            - "9200:9200/tcp"
            - "9300:9300/tcp"
        networks:
            elk_nw:
                ipv4_address: 172.60.0.2
        volumes:
          - type: bind
            source: ./elasticsearch/config/elasticsearch.yml
            target: /usr/share/elasticsearch/config/elasticsearch.yml
            read_only: true
          - type: volume
            source: elasticsearch-data
            target: /usr/share/elasticsearch/data
        extra_hosts:
            - "doc-kibana101:172.60.0.4"

    kibana:
        build:
            context: kibana/
        hostname: doc-kibana101
        container_name: kibana1
        command: sh -c './bin/kibana-plugin list | grep elastalert-kibana-plugin@1.1.0; result=`echo $$?`; if [ $$result = 1 ]; then  ./bin/kibana-plugin install file:///usr/share/kibana/work/elastalert-kibana-plugin-1.1.0-7.6.2.zip && exec /usr/local/bin/kibana-docker; else exec /usr/local/bin/kibana-docker; fi'
        ports:
            - "5601:5601/tcp"
        networks:
            elk_nw:
                ipv4_address: 172.60.0.4
        volumes:
            - type: bind
              source: ./kibana/config/kibana.xml
              target: /usr/share/kibana/config/kibana.yml
              read_only: true
            - type: bind
              source: ./kibana/plugin
              target: /usr/share/kibana/work
              read_only: true
        extra_hosts:
            - "doc-elastic101:172.60.0.2"
        depends_on:
            - elasticsearch

    elastalert:
        container_name: elastalert
        build:
            context: .
            dockerfile: elastalert/Dockerfile
        image: elastalert:0.2.1
        ports:
            - 3030:3030
            - 3333:3333
        depends_on:
            - elasticsearch
            - kibana
        networks:
            elk_nw:
                ipv4_address: 172.60.0.5
        volumes:
            - ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
            - ./elastalert/config/elastalert-test.yaml:/opt/elastalert/config-test.yaml
            - ./elastalert/config/config.json:/opt/elastalert-server/config/config.json
            - ./elastalert/rules:/opt/elastalert/rules
            - ./elastalert/rule_templates:/opt/elastalert/rule_templates
            - ./elastalert/pass:/opt/elastalert/pass
        extra_hosts:
            - "doc-elastic101:172.60.0.2"
            - "doc-kibana101:172.60.0.4"

volumes:
    elasticsearch-data:
        driver: local

networks:
    elk_nw:
        driver: bridge
        ipam:
            driver: default
            config:
                - subnet: 172.60.0.0/16

执行

使用以下命令：
1. 下载基础 DockerImage
2. 生成 DockerImage
3. 启动每个容器
它会为您完成上述操作。
※在我的环境中，下载可能会花费一些时间，但在容器启动后，所以服务启动完成需要约15分钟左右。

$ docker-compose up --build -d

暂时先等待直到以下这样3个容器启动的状态。

$ docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                            NAMES
9b38f592fd57        elastalert:0.2.1          "/usr/local/bin/elas…"   x days ago        Up 3 minutes        0.0.0.0:3030->3030/tcp, 0.0.0.0:3333->3333/tcp   elastalert
696929e31b45        dockerelk_kibana          "/usr/local/bin/dumb…"   x days ago          Up 3 minutes        0.0.0.0:5601->5601/tcp                           kibana1
a06d24afc2d3        dockerelk_elasticsearch   "/usr/local/bin/dock…"   x days ago          Up 3 minutes        0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elastic1

Kibana 用户界面

制定规则

使用 “创建规则” 按钮来创建规则。
有关语法，请参考以下链接。
https://elastalert.readthedocs.io/en/latest/ruletypes.html

测试规则名称: test

es_host: doc-elastic101
es_port: 9200
name: First rule
type: frequency
index: alert*
is_enabled: true

num_events: 1
timeframe:
    hours: 10
filter: []

#subject to email
alert_subject: "Search Spike"

alert:
- "email"
email:
    - "xxxx@gmail.com"
smtp_host: "smtp.gmail.com"
smtp_port: 465 
smtp_ssl: true
from_addr: "xxxx@gmail.com"
smtp_auth_file: "/opt/elastalert/pass/smtp_auth_user.yaml"

• filter: [] とすることで、(アラートを送信する条件を設けず) elasticsearch にメッセージが届いただけでアラートを飛ばすようにしています。

 

送信元と送信先のメールアドレスは同じでOK

考试

允许访问安全性较低的Gmail应用程序

请点击以下链接，将低安全性应用的权限设置为“启用”。
https://myaccount.google.com/lesssecureapps

创建测试数据：用于生成索引模式。

我将通过Kibana UI发送以下查询。
@timestamp将选择最近的合适时间。

PUT /alert/_doc/1
{
  "title": "Test Alart1",
  "name": {
    "first": "test",
    "last": "taro"
  },
  "@timestamp": "2020-MM-ddThh:mm:ss+0900"
}

生成索引模式(This is a direct translation, the phrase is already in Chinese)

创建测试数据：用于发送警报的。

我将在Kibana UI中提交以下查询。
@timestamp 指定了从当前时间开始的后续时间（例如1分钟后）。

PUT /alert/_doc/2
{
  "title": "Test Alart2",
  "name": {
    "first": "test",
    "last": "taro"
  },
  "@timestamp": "2020-MM-ddThh:mm:ss+0900"
}

只要指定的电子邮件地址收到以下这样的消息，则表示成功。

First rule

At least 1 events occurred between 2020-MM-dd hh:mm UTC and 2020-MM-dd hh:mm UTC

@timestamp: 2020-MM-dd hh:mm:ss+09:00
_id: 2
_index: alert
_type: _doc
name: {
    "first": "test",
    "last": "taro"
}
num_hits: 1
num_matches: 1
title: Test Alart2

我卡住了的部分

• 安全性の低いアプリの許可: 無効 だとメール送信できない

 

• セキュアな設定があると思われますが、未確認です

 

• ドキュメント内に @timestamp のフィールドが含まれていないと動かない

 

• ProcessController: ERROR:root:Error running query: RequestError(400, u’search_phase_execution_exception’, u’No mapping found for [@timestamp] in order to sort on’)

 

• ドキュメントの到着時刻を elastalert が判断するために必要

 

• elastalert config で “debug”: true だとアラート送信しない

 

• デバッグモードだと検知してもアラート送信しませんでした（公式ドキュメントでは未確認）

 

• ElastAlert Kibana Plugin の UI から Edit rule を選択すると編集モードになる

 

• 編集モード内に Test ボタンがあるが、実行してもアラートは送信されない

 

以下のようにルールに match するメッセージが何件あったか表示される

Successfully loaded First rule

Would have written the following documents to writeback index (default is elastalert_status):

silence - {'rule_name': 'First rule', '@timestamp': ...}

elastalert_status - {'hits': 2, 'matches': 2, '@timestamp': ...}

• docker logs elastalert -f でログが表示できるが、”verbose”: true にしておかないと詳細なログが表示されず問題に気づけない。

 

• AttributeError: ‘Namespace’ object has no attribute ‘verbose’

 

• Docker の version: 3.0.0-beta.0 では動作しなかった

 

-> 3.0.0-beta.1 に変更することで解消した

下一个任务

• フィルター条件を具体的に指定してみる

 

メールのコンテンツをカスタマイズしてみる

请提供更详细的上下文，以便我能够为您提供高质量的回答。

故障排除

Elastalert无法通过电子邮件进行警报
如何解决elastalert中的“连接到SMTP主机时出错”问题？
我的elastalert正常运行，但我没有收到任何电子邮件警报
Elastalert测试规则未发送任何警报
elastalert未发送电子邮件
elastalert中的电子邮件发送错误。SMTPSenderRefused：（530，“5.5.1需要身份验证”）
找到匹配项但未发送任何警报
无法使用正确的用户名和密码连接smtp服务器
如何修复“发送邮件：授权失败534 5.7.14”
调查基于Elasticsearch数据进行警报通知的方法
bitsensor/elastalert
希望使用elastalert自动检测和通知应用程序异常
在Praeco（ElastAlert GUI）中使用Elasticsearch日志进行警报
bitsensor/elastalert-kibana-plugin
在执行args/conf读取期间出现elastalert-test-rule错误

提示

首次使用ElastAlert和配置
将ElastAlert邮件告警与Elasticsearch集成
ElastAlert文档
ElastAlert-配置和频率规则类型到电子邮件
规则类型和配置选项