RDS Backup or DR fails with Reason: cannot create more than 100 manual snapshots

Issue:

Backup or DR may fail with the following errors:

ERROR: run_snapshots(.\agent.py:880) (database: "dbname") snapshot failed to start. Reason: cannot create more than 100 manual snapshots
ERROR: backup_function_internal(.\agent.py:1204) policy: "policyname". All Snapshots in policy failed!

This is happening as the number of snapshots reached the AWS limit

Solution:

Issue 1:

Customer must have exceeded the limit of 100 RDS snapshot per account per region: http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html

This could happened if the retention is set to keep to many snapshots or some of the 100 RDS snapshots may be present in other CPM policies or created outside CPM - please check the "Snapshots" view in the RDS Dashboard.

As a temporary workaround you can contact AWS support and ask to raise the limit.

You should either decrease the retention and run Cleanup, or ask AWS to increase this limit:

http://aws.amazon.com/contact-us/request-to-increase-the-amazon-rds-db-instance-limit/

Please note: decreasing retention will only affect new backups from that point on, so you probably will need to delete some manually (or ask AWS to increase the limit) before you'll be able to take new snapshots.

Issue 2: 

It could be that too many snapshots are accumlating due to cleanup issues.

As a workaround you can manually delete older backup records that should have been deleted to free up space - see screenshot below on how to delete backup record from UI

To resolve the cleanup issue contact support or read this KB: https://support.n2ws.com/portal/en/kb/articles/cleanup-may-not-delete-snapshots-older-than-the-defined-retention-for-the-policy

• Related Articles

• "SnapshotQuotaExceeded" error when trying to backup RDS

  Customer may receive the following error when trying to backup RDS: ERROR:  get_object(connection.py:1202)  400 Bad Request ERROR:  get_object(connection.py:1203)  <ErrorResponse xmlns="http://rds.amazonaws.com/doc/2013-05-15/">   <Error>     ...

• Cross-Region DR RDS backup may fail with "Copy failed due to an AWS limitation regarding default encryption key” error

  RDS DR backup of encrypted snapshots between different regions may fail with the following error in the Backup log: Error - RDS DR, copy snapshot failed (to DR account) due to an AWS limitation regarding default encryption key (source US East (Ohio), ...

• CPM may mark a successful RDS DR as failed

  On rare occasions, CPM may mark a successful RDS DR as failed in Backup Monitor. When this happens, you can see the following messages in the backup log: Error - RDS DR copy snapshot failed for region “RegionName”….Reason: Cannot create the snapshot ...

• DR process may time out due to exceeding the snapshot limit per region

  In case Backup finished successfully but DR failed after 24 hours due to timeout as seen in the “Backup Log” example below: Error - DR process failed: Timed out after 24 hours Please download logs and search the CPM logs for: RDS DR We reached the ...

• CPM may mark a successful RDS snapshot as failed

  CPM may mark a successful RDS snapshot as failed. In the backup log you may see the following error: Error - snapshot of database <DBname> failed. There may also be a "rate exceeded" error in the CPM server logs: ERROR: get_list(connection.py:1180) ...