Cleanup may not delete snapshots older than the defined retention for the policy
CPM removes the snapshots in accordance with defined retention policies.
However, only successful backups are counted towards the retention.
It is crucial to understand what a "successful backup" is.
A "successful backup" is a backup that has both completed backup and replication of all its snapshots without issues.
If a single snapshot in a policy that backups 100 volumes fails, this backup won't count.
If this continues, you may end up with a larger than expected amount of snapshots in your account and a heavier bill from AWS as a result.
By default, only backups with a status "Backup Successful" are counted towards retention:
You can change this behavior by changing the "backup is successful when" setting in the "more options" screen of a policy,:
If you'll select the second option, backups that are "Partially Successful" will now count as well:
Please keep in mind, that if you choose to ignore the output of scripts/VSS, you may end up with snapshots that are only crash consistent instead of fully consistent.
It is recommended to resolve your problems with scripts/VSS instead of ignoring them.
Here's an actual example that will explain the Generations to save better:
However, only successful backups are counted towards the retention.
It is crucial to understand what a "successful backup" is.
A "successful backup" is a backup that has both completed backup and replication of all its snapshots without issues.
If a single snapshot in a policy that backups 100 volumes fails, this backup won't count.
If this continues, you may end up with a larger than expected amount of snapshots in your account and a heavier bill from AWS as a result.
By default, only backups with a status "Backup Successful" are counted towards retention:
You can change this behavior by changing the "backup is successful when" setting in the "more options" screen of a policy,:
If you'll select the second option, backups that are "Partially Successful" will now count as well:
Please keep in mind, that if you choose to ignore the output of scripts/VSS, you may end up with snapshots that are only crash consistent instead of fully consistent.
It is recommended to resolve your problems with scripts/VSS instead of ignoring them.
Sometimes Partially Successful backups may be caused by CPM trying to backup instances that do not exist anymore: https://support.n2ws.com/portal/kb/articles/a-backup-may-be-marked-partially-successful-due-to-the-instance-probably-doesn-t-exist-anymore-error
CPM will cleanup out of retention snapshots (successful or failed) only when it has enough successful backups as you configured in your policy's retention (Generations to save).
Let's assume you have a daily policy that runs on 2 instances and is supposed to save 3 generations per instance so you expect to have 6 snapshots on EC2 after it ran 3 days and on, right?
On day 1 it runs and succeeds on both instances = 2 snapshots. Successful backup
On day 2 it runs but only the first instance is successfully backed up so now you have 3 snapshots. This is not considered as a successful backup.
On day 3 it runs again but only the first instance is successfully backed up so now you have 4 snapshots. This is not considered as a successful backup.
On day 4 it runs and succeeds = 6 snapshots (none is deleted because you only have 2 successful backups, not 3). Successful backup
On day 5 it runs and succeeds = 8 snapshots (none is deleted because you only have 3 successful backups) - on this day you'll pay for extra 2 snapshots. Successful backup
On day 1 it runs and succeeds on both instances = 2 snapshots. Successful backup
On day 2 it runs but only the first instance is successfully backed up so now you have 3 snapshots. This is not considered as a successful backup.
On day 3 it runs again but only the first instance is successfully backed up so now you have 4 snapshots. This is not considered as a successful backup.
On day 4 it runs and succeeds = 6 snapshots (none is deleted because you only have 2 successful backups, not 3). Successful backup
On day 5 it runs and succeeds = 8 snapshots (none is deleted because you only have 3 successful backups) - on this day you'll pay for extra 2 snapshots. Successful backup
Cleanup process iwll still not clean the failed backups since they're still within the retention period.
On day 6 it runs and succeeds = 6 snapshots (day 1,2 and 3 snapshots are deleted). Here, cleanup will clean the older successful ones as well as the failed ones.
On day 6 it runs and succeeds = 6 snapshots (day 1,2 and 3 snapshots are deleted). Here, cleanup will clean the older successful ones as well as the failed ones.
The good backups don't necessarily have to be one after the other but as long as you have failed backups between successful backups, the failed ones will not get deleted.
Related Articles
How cleanup works and why it might not delete snapshots older than the defined retention setting.
Background This KB Article will explain how the cleanup process works and why sometimes cleanup might not delete old snapshots It includes following sections: About the cleanup process Why Cleanup might not delete old snapshots Examples About the ...Complete_VSS operation may fail if snapshot takes more than 45 minutes to complete
Complete_VSS operation may fail if snapshot takes more than 45 minutes to complete, due to exceeding the default VSS timeout: In order to verify this, you should check the Agent's logs - VSS script should fail with return code "3" immediately after ...Cleanup Log
This feature was added to allow CPM users to better understand whether Cleanup process ran successfully. It can be found in the “General Settings” tab. The Cleanup Interval may be set to run every 1 to 24 hours. Last cleanup timestamp will show the ...VSS pre-snapshot operations on an Agent may fail almost immediately with "vss script crashed or failed. return code: 1"
VSS pre-snapshot operations on an Agent may fail almost immediately with "vss script crashed or failed. return code: 1" listed in the Agent log: ERROR: start_vss_quiesce(remote_agent.pyo:909) policy ihpa_daily, vss failed (reason: vss script ...Error "Policy "Policy_Name" doesn't exist/attached to account" may be printed into CPM server log during the tag scanning operation
The following may be printed into the CPM server log during the tag scanning operation: ERROR: manage_account_tagged_resources(.\cpmserver\cpm\backup_tag.py:336) Policy_Name doesn't exist/attached to account To resolve: Find error in CPM server logs, ...