how elastic store indexes

Question

The elastic indexes are getting bigger and bigger and then some days the indexes are small. The days that indexes are small no machine is down; everything is the same as in the days the indexes are big. I noticed that elasticsearch still store documents in the indexes from days before.

Is it possible that elastic pilling up the days before in the current day? How elastic stores the documents on indexes?

We had to decrease the days the indexes are stored since some days one index is 2x the size of another.

Thanks

Answer 1

@maryf there can be 2 possibilities here if there are date based indexes:

1. log shipper is not persisting the registry which contains info about which log files have ben harvested and upto what offset.
2. index is defined to use incorrect timestamp field for timelines.

In first case, whenever your log shipper restarts, it will start reading log files from beginning and you can see duplicate records in your index. While in the second case, logs are stored to the index based on the timestamp field being used. If the timestamp is from older date, it will be stored in older index matching the date.