In my es cluster, I've been getting the below error continuously in the past few days and seeing a significant performance issue with the elasticsearch cluster.
[2024-02-07T12:29:58,861][ERROR][o.e.x.w.Watcher ] [node-1] watch history could not be written [WATCH_32_ALERT_5_c048-6e01-43b7-b1aa-2024-02-07T12:29:58.108385427Z], failure [java.lang.IllegalArgumentException: Limit of total fields [1000] has been exceeded]
Is there any relation between this error and the performance issue of the cluster? If yes, why? and how to eliminate it?
Note: The fix I tried is deleting the .watcher-history- indices that are auto-generated and restarting the cluster, surprisingly, post the restart, the cluster performance has been improving for some time but after some time the error is being started happening again, also the performance issues. Performance issues i.e. significant search latency observed, dashboard are not loading with data, high CPU usage, etc.
ELK Stack Details -
ELK Version: 7.11.1
ES Node: 3
Dedicated servers for Kibana and logstash
JVM Heap: 48 GB
Indices: 316
Total Shard: 632
Documents: 1,341,422,500
Data: 1.1 TB
Kibana Users: 154
Watcher Scripts: 140
Need your help to understand the root cause.