I am looking for a Process to Create Daily Indices . That can Generate Logs report On daily basis

Question

I am Using Kibana 6.4.3 and I want to Retain few month Logs but Untill I setup daily indices. Need a process to setup auto generation of Daily Indices.

Answer 1

You need to implement Rollover Index.

I'd suggest you to refer to the link to see how it can be done. Basically you can simply execute the below queries to implement and test this Rollover Index feature

Step 1: Create Initial Index

PUT /logs-000001
{
  "aliases": {
    "logs_write": {}
  }
}

Step 2: Execute Rollover API

POST /logs_write/_rollover
{
  "conditions" : {
    "max_age": "1d",
    "max_docs": 1000,
    "max_size": "5gb"
  },
  "settings": {
    "index.number_of_shards": 2
  }
}

What would happen is elasticsearch would go ahead and create index logs-000002 after a day and so on and on.

The API accepts a single alias name and a list of conditions. The alias must point to a single index only. If the index satisfies the specified conditions then a new index is created and the alias is switched to point to the new index.

You can actually test the roll over using dry_run as shown in below query.

Dry-Run Query

POST /logs_write/_rollover?dry_run
{
  "conditions" : {
    "max_age": "1d",
    "max_docs": 1000,
    "max_size": "5gb"
  }
}

When you run this test query, notice the response in the below format. It would show what would happen if the rollover happens after a day. Note that when you execute dry run, it doesn't create logs-000002.

Dry Run Response

{
  "acknowledged": false,
  "shards_acknowledged": false,
  "old_index": "logs-000001",
  "new_index": "logs-000002",
  "rolled_over": false,
  "dry_run": true,
  "conditions": {
    "[max_age: 1d]": false,
    "[max_docs: 1000]": false,
    "[max_size: 5gb]": false
  }
}

Important Note:

Rollover doesn't happen automatically. It has to be done manually using crontab or such scheduling tools and keep checking the conditions to execute rollover API programmatically.

Refer to this LINK for more info on this

Hope it helps!