[PYTHON] Weight loss Elasticsearch with Curator

Elasticsearch is heavy. The memory of the PC is poor. I haven't seen this data (index) anymore.

To you.

Environment to prepare in advance

Elasticsearch 5.0.1 (same for 5.0.0) Python 2.7.10 (same for 2.7.6)

things to do

First of all, if you don't have pip,

$ curl -kL https://bootstrap.pypa.io/get-pip.py | sudo python

Install Curator

$ sudo pip install elasticsearch-curator
$ curator --version
curator, version 4.2.4

If Elasticsearch is 5.x, curator needs to be 4.x or higher.

Files required for Curator

There are two Configuration file

~/.curator/curator.yml

client:
  hosts:
    - localhost
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
# if x-pack is installed, set "http_auth: elastic:changeme"
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

ACTION_FILE

actions:
  1:
    action: delete_indices
    description: >-
      (custommessage)Delete indices older than 5 days (based on index name), for .monitoring-es-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.(custom message)
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: .monitoring-es-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 5
      exclude:
  2:
    action: close
    description: >-
      (custom message)Close indices older than 4 days (based on index name), for .monitoring-es-
      prefixed indices.(custom message)
    options:
      ignore_empty_list: True
      delete_aliases: False
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: .monitoring-es-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 4
      exclude:

This closes the index more than 4 days ago and deletes the index more than 5 days ago. ʻIgnore_empty_list: True` You can pass through the error when the corresponding index is not found. (If you do not attach it, it will stop with an error when you execute it twice or more in a row.) Although it is unit_count, if it is 4, the data 4 days ago will also be deleted. % d of index seems to be the standard.

Run

When the Configuration file is created with the default path (~ / .curator / curator.yml)

$ curator ${action_file}

If not

$ curator --config ${configuration_file} ${action_file}

Then run it manually when you feel like it, or let Uncle jenkins at home do it daily.

• I only do simple things. There are more features, so please go to Official Document.