Manage daily, weekly and monthly backups in AWS S3

Question

I am asked to design a solution to manage backups on AWS S3. Currently, I am uploading daily backups to S3. Only 30 backups are stored and older backups are deleted as new backups are created.

Now I need to manage these backups on a daily, weekly and monthly basis. For eg:

1-14 days  -  Daily backup (14 backups)

15-90 days - weekly backups (11 backups)

After 90 days till 6 months - Monthly backups (3 backups)

This way we'll have to store 28 backups.

Currently, all I have in my mind is to create three folders inside the S3 bucket (daily, weekly, and monthly). Then create a bash script to move around the backups between these folders. And finally, daily trigger the bash script from Jenkins.

Note that these backups are created and uploaded to the S3 bucket using a third-party utility. So, there is nothing I can do at the time of uploading.

Is there any native solution provided by AWS to create such policies? Or if you have any better approach to solve this use case, please share.

Thanks in advance.

Answer 1

I would suggest triggering an AWS Lambda function after an upload, which can then figure out which backups to keep and which to remove.

For example, it could:

• Remove the Day 15 'daily backup' unless it represents the start of a new week
• Remove the oldest weekly backup unless it represents the start of a new month
• Remove the oldest monthly backup if it is more than 6 months old

No files need to be 'moved' -- you just need to selectively delete some backups.