ES Kibana rollover and having the app use the new index

26 Views Asked by At

I understand the title isn't very clear so I'll do my best to explain it.

My app has its log configuration set up like this:

"logConfiguration": {
      "logDriver":"awsfirelens",
      "options": {
        "Name": "es",
        "Host": "${logs_endpoint}",
        "Port": "443",
        "Index": "default-logs-000001",
        "Type": "_doc",
        "Aws_Auth": "On",
        "Aws_Region": "${region}",
        "tls": "On"
      }
    }

Basically, its streaming logs to AWS ElasticSearch Kibana index called default-logs-000001. This is ok and expected. The problem is when this index gets too big, the search becomes slow and filled with errors. So to fix that, I figured I should create a rollover policy that would automatically create default-logs-000002 and so on, whenever the original log is older than 7 days or bigger than 3gb, and the policy looks like this:

{
    "policy_id": "default-logs-policy",
    "description": "Default Logs Policy",
    "last_updated_time": 1690200226833,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "3gb",
                        "min_index_age": "7d"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "warm_state"
                }
            ]
        },
        {
            "name": "warm_state",
            "actions": [],
            "transitions": [
                {
                    "state_name": "delete_state",
                    "conditions": {
                        "min_index_age": "60d"
                    }
                }
            ]
        },
        {
            "name": "delete_state",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": [
        {
            "index_patterns": [
                "default-logs-*"
            ],
            "priority": 0,
            "last_updated_time": 167700000
        }
    ]
}

Now, there are problems with this policy. If I run it manually, it works as expected, and it creates a new index with incremented number. Otherwise its stuck in warm state despite meeting transition criteria. But that's not my issue. My issue is, that my app, doesn't switch to the new rolledover index. Which, to be honest, makes sense. I mean, even if Kibana creates a new log to ease the burden on the original, how is my app supposed to know to switch? It still has default-logs-000001 hardcoded into it. That's what I'd like to know. How to have it automatically switch, without me making commits to the log configuration every time a new index pops up. Original idea was to use default-logs-$date, but that was bad because we'd have a lot of releases, which led to a ton of unused indices clogging up Kibana, so we'd have to manually delete those. I just need a way for the app to send logs to one index, and as soon as it gets big, switch to a new one. How would I do that? Thank you.

0

There are 0 best solutions below