使用docker-compose同时启动Elasticsearch/Kibana/ElastAlert，以便在第一次启动时创建ElastAlert的索引
目标版本
・Elasticsearch 7.6.2
・Kibana 7.6.2
・servercentral/elastalert:latest
→ ElastAlert 服务器和 ElastAlert 0.2.1
错误消息输出示例
出现大量错误消息，并且ElastAlert的索引在Elasticsearch中没有创建，因此需要重新启动ElastAlert。
16:42:28.793Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
16:42:59.055Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
16:43:06.877Z ERROR elastalert-server:
ProcessController: WARNING:elasticsearch:GET http://elasticsearch:9200/elastalert_status/_search?size=1000 [status:404 request:0.009s]
16:43:06.877Z ERROR elastalert-server:
ProcessController: ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2020-04-25T16:43:06.867791Z', 'to': '2020-04-27T16:43:06.867819Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1625, in find_recent_pending_alerts
res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 819, in search
"GET", _make_path(index, "_search"), params=params, body=body
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request
timeout=timeout,
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request
self._raise_error(response.status_code, raw_data)
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)
16:43:29.504Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
16:43:59.813Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
16:44:06.871Z ERROR elastalert-server:
ProcessController: WARNING:elasticsearch:GET http://elasticsearch:9200/elastalert_status/_search?size=1000 [status:404 request:0.007s]
16:44:06.874Z ERROR elastalert-server:
ProcessController: ERROR:root:Error finding recent pending alerts: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias) {'query': {'bool': {'must': {'query_string': {'query': '!_exists_:aggregate_id AND alert_sent:false'}}, 'filter': {'range': {'alert_time': {'from': '2020-04-25T16:44:06.864223Z', 'to': '2020-04-27T16:44:06.864250Z'}}}}}, 'sort': {'alert_time': {'order': 'asc'}}}
Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1625, in find_recent_pending_alerts
res = self.writeback_es.search(index=self.writeback_index, body=query, size=1000)
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 819, in search
"GET", _make_path(index, "_search"), params=params, body=body
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/transport.py", line 350, in perform_request
timeout=timeout,
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 156, in perform_request
self._raise_error(response.status_code, raw_data)
File "/home/node/.local/lib/python3.6/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
status_code, error_message, additional_info
elasticsearch.exceptions.NotFoundError: NotFoundError(404, 'index_not_found_exception', 'no such index [elastalert_status]', elastalert_status, index_or_alias)
16:44:30.100Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
如果重启的话会被创建出来…
但是我希望它从一开始就被创建出来。
> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh
16:46:38.792Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:46:38.794Z INFO elastalert-server: Config: Proceeding to look for normal config file.
16:46:38.795Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:46:38.804Z INFO elastalert-server: Router: Listening for GET request on /.
16:46:38.804Z INFO elastalert-server: Router: Listening for GET request on /status.
16:46:38.805Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
16:46:38.805Z INFO elastalert-server: Router: Listening for GET request on /rules.
16:46:38.807Z INFO elastalert-server: Router: Listening for GET request on /rules/:id*.
16:46:38.807Z INFO elastalert-server: Router: Listening for POST request on /rules/:id*.
16:46:38.807Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id*.
16:46:38.807Z INFO elastalert-server: Router: Listening for GET request on /templates.
16:46:38.807Z INFO elastalert-server: Router: Listening for GET request on /templates/:id*.
16:46:38.808Z INFO elastalert-server: Router: Listening for POST request on /templates/:id*.
16:46:38.808Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id*.
16:46:38.808Z INFO elastalert-server: Router: Listening for PUT request on /folders/:type/:path*.
16:46:38.808Z INFO elastalert-server: Router: Listening for DELETE request on /folders/:type/:path*.
16:46:38.808Z INFO elastalert-server: Router: Listening for POST request on /test.
16:46:38.808Z INFO elastalert-server: Router: Listening for POST request on /silence/:path*.
16:46:38.808Z INFO elastalert-server: Router: Listening for GET request on /config.
16:46:38.809Z INFO elastalert-server: Router: Listening for POST request on /config.
16:46:38.809Z INFO elastalert-server: Router: Listening for POST request on /download.
16:46:38.809Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert.
16:46:38.809Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_status.
16:46:38.809Z INFO elastalert-server: Router: Listening for GET request on /metadata/silence.
16:46:38.809Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_error.
16:46:38.809Z INFO elastalert-server: Router: Listening for GET request on /metadata/past_elastalert.
16:46:38.809Z INFO elastalert-server: Router: Listening for GET request on /indices.
16:46:38.810Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
16:46:38.810Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
16:46:38.810Z INFO elastalert-server: Router: Listening for GET request on /config.
16:46:38.814Z INFO elastalert-server: ProcessController: Starting ElastAlert
16:46:38.814Z INFO elastalert-server: ProcessController: Creating index
16:46:43.074Z INFO elastalert-server:
ProcessController: Elastic Version: 7.6.2
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
16:46:43.075Z INFO elastalert-server: ProcessController: Index create exited with code 0
16:46:43.076Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
16:46:43.087Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 36)
16:46:43.091Z INFO elastalert-server: Server: Server listening on port 3030
16:46:43.092Z INFO elastalert-server: Server: Websocket listening on port 3333
16:46:43.095Z INFO elastalert-server: Server: Server started
16:47:06.693Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
16:47:37.013Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
16:48:07.292Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
处理
确保Elasticsearch处于连接状态后，启动ElastAlert。
|--docker-compose.yml
|--Dockerfiles
| |--Dockerfile-elastalert
|
|--es
| |--config
| | |--elasticsearch.yml
| |--data
|
|--kibana
| |--config
| | |--kibana.yml
|
|--elastalert
| |--bin
| | |--elastalert-start.sh
| | |--elastic_search_status.sh
| |--config
| | |--api.config.json
| | |--elastalert.yaml
| |--rule_templates
| |--rules
version: "3.7"
services:
elasticsearch:
container_name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
ports:
- 9200:9200
- 9300:9300
environment:
- ES_JAVA_OPTS=-Xms256m -Xmx256m
- discovery.type=single-node
restart: always
volumes:
- ./es/data:/usr/share/elasticsearch/data
- ./es/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:9200 || exit 1"]
interval: 30s
timeout: 15s
retries: 3
start_period: 180s
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.6.2
ports:
- 5601:5601
depends_on:
- elasticsearch
restart: always
volumes:
- ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:5601/api/status || exit 1"]
interval: 30s
timeout: 15s
retries: 3
start_period: 200s
elastalert:
container_name: elastalert
build:
context: .
dockerfile: Dockerfiles/Dockerfile-elastalert
image: elastalert:0.2.1
ports:
- 3030:3030
- 3333:3333
depends_on:
- elasticsearch
- kibana
restart: always
volumes:
- ./elastalert/config/elastalert.yaml:/opt/elastalert/config.yaml
- ./elastalert/config/api.config.json:/opt/elastalert-server/config/config.json
- ./elastalert/rules:/opt/elastalert/rules
- ./elastalert/rule_templates:/opt/elastalert/rule_templates
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:3030 || exit 1"]
interval: 30s
timeout: 15s
retries: 3
start_period: 200s
cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true
FROM servercentral/elastalert:latest
USER root
RUN apk update && \
apk add bash curl && \
rm -rf /var/cache/apk/*
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/elastalert-start.sh && chmod +x /usr/local/bin/elastic_search_status.sh
USER node
ENTRYPOINT ["/usr/local/bin/elastalert-start.sh"]
#!/bin/bash
set -e
if [ $# -gt 0 ]; then
ES_URL="$1"
elif [[ -n $ELASTICSEARCH_URL ]]; then
ES_URL="$ELASTICSEARCH_URL"
elif [[ -n $ES_HOST ]] && [[ -n $ES_PORT ]]; then
ES_URL="http://$ES_HOST:$ES_PORT"
else
ES_URL="http://elasticsearch:9200"
fi
until [[ "$(curl -fsSL "$ES_URL/_cat/health?h=status" | sed -r 's/^[[:space:]]+|[[:space:]]+$//g')" =~ ^(yellow|green)$ ]]; do
# printf '+' >&2
sleep 1
done
echo "Elasticsearch is up and healthy at "$ES_URL"" >&2
#!/bin/bash
set -e
echo "Giving Elasticsearch at $ELASTICSEARCH_URL time to start..."
elastic_search_status.sh
echo "Starting ElastAlert!"
npm start
es/config/elasticsearch.yml
{
"appName": "elastalert-server",
"port": 3030,
"wsport": 3333,
"elastalertPath": "/opt/elastalert",
"verbose": false,
"es_debug": false,
"debug": false,
"rulesPath": {
"relative": true,
"path": "/rules"
},
"templatesPath": {
"relative": true,
"path": "/rule_templates"
},
"es_host": "elasticsearch",
"es_port": 9200,
"es_username": "",
"es_password": "",
"es_ssl": false,
"writeback_index": "elastalert_status"
}
# The elasticsearch hostname for metadata writeback
# Note that every rule can have its own elasticsearch host
es_host: elasticsearch
# The elasticsearch port
es_port: 9200
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: rules
# How often ElastAlert will query elasticsearch
# The unit can be anything from weeks to seconds
run_every:
seconds: 60
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 1
# Optional URL prefix for elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for elasticsearch
#es_username: someusername
#es_password: somepassword
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
skip_invalid: True
$ chmod 777 es/data
$ chmod 777 elastalert/rules
$ chmod 777 elastalert/rule_templates
$ docker-compose up -d
$ docker logs -f elastalert
Giving Elasticsearch at time to start...
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused
Elasticsearch is up and healthy at http://elasticsearch:9200
Starting ElastAlert!
> @bitsensor/elastalert@0.0.14 start /opt/elastalert-server
> sh ./scripts/start.sh
16:53:28.919Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
16:53:28.927Z INFO elastalert-server: Config: Proceeding to look for normal config file.
16:53:28.931Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
16:53:28.942Z INFO elastalert-server: Router: Listening for GET request on /.
16:53:28.943Z INFO elastalert-server: Router: Listening for GET request on /status.
16:53:28.944Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
16:53:28.944Z INFO elastalert-server: Router: Listening for GET request on /rules.
16:53:28.947Z INFO elastalert-server: Router: Listening for GET request on /rules/:id*.
16:53:28.948Z INFO elastalert-server: Router: Listening for POST request on /rules/:id*.
16:53:28.949Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id*.
16:53:28.949Z INFO elastalert-server: Router: Listening for GET request on /templates.
16:53:28.950Z INFO elastalert-server: Router: Listening for GET request on /templates/:id*.
16:53:28.951Z INFO elastalert-server: Router: Listening for POST request on /templates/:id*.
16:53:28.952Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id*.
16:53:28.952Z INFO elastalert-server: Router: Listening for PUT request on /folders/:type/:path*.
16:53:28.953Z INFO elastalert-server: Router: Listening for DELETE request on /folders/:type/:path*.
16:53:28.954Z INFO elastalert-server: Router: Listening for POST request on /test.
16:53:28.958Z INFO elastalert-server: Router: Listening for POST request on /silence/:path*.
16:53:28.959Z INFO elastalert-server: Router: Listening for GET request on /config.
16:53:28.960Z INFO elastalert-server: Router: Listening for POST request on /config.
16:53:28.961Z INFO elastalert-server: Router: Listening for POST request on /download.
16:53:28.962Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert.
16:53:28.963Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_status.
16:53:28.964Z INFO elastalert-server: Router: Listening for GET request on /metadata/silence.
16:53:28.964Z INFO elastalert-server: Router: Listening for GET request on /metadata/elastalert_error.
16:53:28.965Z INFO elastalert-server: Router: Listening for GET request on /metadata/past_elastalert.
16:53:28.966Z INFO elastalert-server: Router: Listening for GET request on /indices.
16:53:28.967Z INFO elastalert-server: Router: Listening for GET request on /mapping/:index.
16:53:28.967Z INFO elastalert-server: Router: Listening for POST request on /search/:index.
16:53:28.968Z INFO elastalert-server: Router: Listening for GET request on /config.
16:53:28.973Z INFO elastalert-server: ProcessController: Starting ElastAlert
16:53:28.974Z INFO elastalert-server: ProcessController: Creating index
16:53:35.304Z INFO elastalert-server:
ProcessController: Elastic Version: 7.6.2
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!
16:53:35.304Z INFO elastalert-server: ProcessController: Index create exited with code 0
16:53:35.307Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
16:53:35.332Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 241)
16:53:35.352Z INFO elastalert-server: Server: Server listening on port 3030
16:53:35.353Z INFO elastalert-server: Server: Websocket listening on port 3333
16:53:35.358Z INFO elastalert-server: Server: Server started
16:53:45.843Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
16:54:16.146Z INFO elastalert-server: Routes: Successfully handled GET request for '/'.
