N2WS 3.1.x - Warning Could not process CBT for volume snapshots
Issue summary
When running backup for EBS volumes on version 3.1 it might raise the following warning:
Issue description and troubleshooting
This warning can be caused by permission issues, communication or lack of EBS Direct API endpoint in target region.
In version 3.1 we have started to use the new EBS Direct API, this new API let us read the blocks directly from the snapshot and help us collect information about the snapshot size and block changes.
This new information can then be seen in the snapshot report:
When you run a backup, The server will try to collect CBT info(Changed Block Tracking) from the new EBS snapshots, if there is any access or permission issue it will raise this warning
Troubleshooting:
First you need to check that the new API is available in your target region Link - Endpoints for the EBS direct APIs, if it is not available then this warning is expected and can be ignored.
If the API is available for the region, then you need to check that you have the latest permission updated for the N2WS server role.
You can find latest permissions attached here:
You can also Check permissions in the UI:
If it is available and you updated the json permissions, then you can connect to the N2WS server via ssh(user is cpmuser & your private key) to test the connection manually.
If you don't know how to connect via SSH, You can see information from AWS on how to connect by clicking on the instance and then connect:
Once connected, you can test connectivity/permissions by running this curl command:
- aws ebs list-snapshot-blocks --snapshot-id snap-05cddc7aa3c379497 --region us-east-1
Note: You need to replace us-east-1 with the relevant region, the region to choose is the one where the snapshot is located
Note: If you run the command with snapshot id that does not exist, then the expected result is ‘snapshot not found’ - meaning command was able to reach the endpoint and had permission.
If it fails, it might timeout or give you permission issue:
You need to make sure that the API endpoint in the target region is accessible from N2WS server and that there is no permission issue.
Related Articles
N2WS 3.1.x - Warning 'Error verifying access to EBS API in region: us-east-1. CPM will not use read from snapshot for instances in this region'
Issue summary When running copy to S3 on version 3.1 it might raise the following warning. Issue description and troubleshooting This warning can be caused by permission issues, communication or lack of EBS Direct API endpoint in target region. In ...EBS/RDS DR & Recovery with KMS key
When copying cross account an EBS/RDS Volume encrypted with custom KMS, a KMS key should also be available in the other account. There are 2 ways that CPM uses for checking KMS key - Alias & Tag KMS Tag When using custom tag, you are telling CPM to ...Error "create multi-volume snapshots of instance i-123456789 failed" may occur during backup
Error: You might see this error in the cpm server logs or the UI Alerts when running backups in CPM 3.0 and above: create multi-volume snapshots of instance i-123456789 failed. Error: You are not authorized to perform this operation. Encoded ...How to fully encrypt CPM server root and data volumes in EBS
The steps below will help you properly encrypt the CPM server root and data EBS volumes for an existing CPM server instance. If encrypting only the data volume, it is still necessary to launch a new CPM server instance. Then during initial CPM ...Volume Snapshot may fail with error "The maximum per volume CreateSnapshot request rate has been exceeded"
Volume Snapshots may fail with the following errors in the log files. backup log 02-12-20 11:07:22 - Error - create snapshot of instance volume vol-xxxxxxxxxxxxx failed. 02-12-19 11:07:22 - Error - snapshot of instance ec2-xxxxxxxxx-xxxx-xx, volume ...