将使用Docker的Elasticsearch可视化为Kibana中的NetFlow

使用Docker来构建一个环境，以便在Kibana中显示由Fluentd获取的NetFlow v5数据。

作为前提条件，需要安装Docker和Docker-compose。

从Ubuntu16.04LTS到Docker

2. 提前增加内存映射的上限值

vm.max_map_count=262144

3. 将docker-compose.yml文件按以下方式进行编写。

fluentd:
  build: fluentd
  ports:
    - 24224:24224
    - 24224:24224/udp
  volumes:
    - ./fluentd:/fluentd/etc
    - ./fluentd/plugins:/fluentd/plugins
  links:
      - elasticsearch

elasticsearch:
  image: elasticsearch:5.6.4
  ports:
    - 9200:9200
    - 9300:9300
  expose:
    - 9300

kibana:
  image: kibana:5.6.4
  ports:
    - 5601:5601
  environment:
      - ELASTICSEARCH_URL=http://172.17.0.2:9200
#      - ELASTICSEARCH_URL=http://elasticsearch:9200
  links:
      - elasticsearch

创建从Fluentd到Elasticsearch的发送配置

FROM fluent/fluentd:latest-onbuild

#USER fluent

WORKDIR ~/netflow-compose/fluentd
ENV PATH ~/.gem/ruby/2.3.0/bin:$PATH
RUN apk add --no-cache --update --virtual=build-dependencies ruby-dev build-base && \
  gem install fluent-plugin-rewrite-tag-filter && \
  apk del build-dependencies && \
  rm -rf /tmp/* /var/tmp/* /var/cache/apt/*
RUN gem install fluent-plugin-netflow
RUN gem install fluent-plugin-secure-forward
RUN gem install fluent-plugin-elasticsearch # Elasticsearch連携

#EXPOSE 24284
EXPOSE 24224

CMD fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins -vv

<source>
  type netflow
  tag netflow.event
  port 24224
</source>
<match netflow.event>
  @type copy
  <store>
    @type file
    path         /fluentd/log/data.*.log
    symlink_path /fluentd/log/data.log
    append       true
  </store>
  <store>
    @type elasticsearch
#    host localhost
#    host elasticsearch
    host 172.17.0.2
    port 9200
    type_name netflow
    logstash_format true
    logstash_prefix flow
    logstash_dateformat %Y%m%d
  </store>
</match>

5. 其余的就是运行 docker-compose up。

请运行以下命令：sudo docker-compose up

请参考以下链接

【Docker】使用Fluentd、Elasticsearch和Re:dash进行日志管理
链接：https://qiita.com/narumi_/items/59237554ea1c0e713dd9

・Docker 官方
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
http://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#docker-prod-cluster-composefile

– Docker 官方网站
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
http://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#docker-prod-cluster-composefile

将缓冲区中的所有块排队

关于Fluentd的Ruby引入，可以参考这里的链接：