Elasticsearch-curator

什么是es-curator?

你有需要定期删除es数据的需求吧,有需要把过期的index删除的需求吧,你不再需要手工书写脚本了,官方出了一个curator!

它提供各种参数配置,安装即可使用,命令行模式。

见源码 链接

使用pip安装

当然如果你的机器有外网ip(即可联网),可以方便快捷安装

运行以下命令即可(请保证你的python是2.7以上)

pip install elasticsearch-curator

源码安装

准备好以下安装包,想办法下载吧:

certifi-2017.1.23.tar.gz click-6.3.tar.gz curator-4.2.6.tar.gz elasticsearch-curator-4.2.6.tar.gz elasticsearch-py-2.4.1.tar.gz pip-9.0.1.tar.gz Python-2.7.8.tgz PyYAML-3.11.tar.gz setuptools-0.6c11.tar.gz urllib3-1.12.tar.gz voluptuous-0.9.3.tar.gz

1、安装python2.7

安装之前,请为系统安装zib,zib-devel,因为后面setuptools安装需要用到

yum install -y zib zib-devel

如果忘记了可以重新编译安装python2.7

wget https://www.python.org/ftp/python/2.7.8/Python-2.7.8.tgz
tar xf Python-2.7.8.tgz
cd Python-2.7.8
./configure --prefix=/usr/local
make && make install

2、安装setuptools

tar -xf setuptools-0.6c11.tar.gz
cd  setuptools-0.6c11 
python setup.py install

3、安装pip

tar -xf pip-9.0.1.tar.gz
cd pip-9.0.1
python setup.py install

接下来正式安装curator必需安装的依赖

请参考官网 Installation from source

4、安装certifi时需要注意,安装比较新的版本

如果报类似的以下错误信息,需得上网下载对应时间的版本

Collecting certifi>=2017.1.23 (from elasticsearch-curator==4.2.6)
  Could not fetch URL https://pypi.python.org/simple/certifi/: There was a problem confirming the ssl certificate: Can't connect to HTTPS URL because the SSL module is not available. - skipping
  Could not find a version that satisfies the requirement certifi>=2017.1.23 (from elasticsearch-curator==4.2.6) (from versions: )
No matching distribution found for certifi>=2017.1.23 (from elasticsearch-curator==4.2.6)

5、根据官网来分别安装

pip install urllib3-1.12.tar.gz

pip install click-6.3.tar.gz

pip install certifi-2017.1.23.tar.gz 

pip install PyYAML-3.11.tar.gz

pip install voluptuous-0.9.3.tar.gz 

pip install  elasticsearch-py-2.4.1.tar.gz  

#最后安装curator

pip install elasticsearch-curator-4.2.6.tar.gz 

5、检查是否安装成功

curator --help

如果出现正确帮助信息,即为成功安装


Usage: curator [OPTIONS] ACTION_FILE

  Curator for Elasticsearch indices.

  See http://elastic.co/guide/en/elasticsearch/client/curator/current

Options:
  --config PATH  Path to configuration file. Default: ~/.curator/curator.yml
  --dry-run      Do not perform any changes.
  --version      Show the version and exit.
  --help         Show this message and exit.

curator使用

curator --config ./curatory.yml action.yml

curator.yml配置文件

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: True

logging:
  loglevel: INFO
  logfile: ../logs/es-curator.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

1、hosts

hosts: [ "10.0.0.1", "10.0.0.2" ]

or 直接指定端口号

hosts:
  - 10.0.0.1:9200
  - 10.0.0.2:9201

2、master_only

当设置为True时,不需要在集群各结点运行curator;并且hosts只能设置主结点的ip+port

action.yml

例子:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: ACTION1
    description: OPTIONAL DESCRIPTION
    options:
      option1: value1
      ...
      optionN: valueN
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: *first*
      filter_element1: value1
      ...
      filter_elementN: valueN
    - filtertype: *second*
      filter_element1: value1
      ...
      filter_elementN: valueN
  2:
    action: ACTION2
    description: OPTIONAL DESCRIPTION
    options:
      option1: value1
      ...
      optionN: valueN
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: *first*
      filter_element1: value1
      ...
      filter_elementN: valueN
    - filtertype: *second*
      filter_element1: value1
      ...
      filter_elementN: valueN
  3:
    action: ACTION3
    ...
  4:
    action: ACTION4
    ...

1、disable_action 是否生效

2、ignore_empty_list

如果设置为True时,filter出的indices如果为空时,curator 正常退出,日志打印出INFO信息

如果设置为False时,filter出的indices如果为空时,curator 异常退出,返回1, 日志打印出ERROR信息

3、continue_if_exception

如果设置为True时,忽略异常报错,跳到下一个action,正常执行下去

4、filtertype

针对index的过滤器有以下多个,分别为:

age ## 根据时间来过滤indices
alias
count
pattern  ## 正则匹配
space
kibana
forcemerged
closed
opened
allocated
none
  • age的其他配置项

1) source 三个选择:name, creation_date, or field_stats

name -- 根据时间字串 timestring来

creation_date -- 根据index创建时间(creation time)来

field_stats -- 使用timestamp的字段来匹配

2)direction 两个选择 : older or younger

根据另外参数unit or unit_count两个相结合的, 匹配出比当前时间要旧或者新的一个时间的indices

3)unit and unit_count

unit 单位

unit_count 数值

4)timestring 指定时间字串的格式

由% 与Y,y,m,W,d,H,M,S,j 组合

5)exclude

如果为True,即屏蔽掉匹配的项; False即保留匹配的项

6)stats_result

当source设置为field_stats时,些项才生效; 值为 either min_value or max_value

  • pattern

如:

  filters:
  - filtertype: pattern
    kind: regex
    value: '^(alpha-|bravo-|charlie-).*$'

1)kind ; pattern必备

有:prefix, suffix, timestring, and regex

2)删除过期indices的数据

action.yml

# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: fx_stream_event-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

运行方法:

curator --config curator.yml action.yml 

把任务放到crontab定期删除

1、编写脚本curator-delete-index.sh

#!/bin/bash
/usr/local/bin/curator --config /data/elastic/elasticsearch-5.2.0/config/curator.yml  /data/elastic/elasticsearch-5.2.0/config/action.yml
echo "delete index success"

2、放于crontab

30 05 * * * /data/elastic/elasticsearch-5.2.0/curator-delete-index.sh