Retreive support logging from a failed S3 worker root volume
In some cases where an S3 copy fails the root volume of the worker is kept and logging is not automatically transferred to the CPM server.
order to troubleshoot the S3 copy issue, support may ask you to collect a copy of logging from the retained worker volume.
Error - Backup copy failed
Info - Root volume of failed worker
i-1234567890abcdef (vol-1234567890abcdef) has been retained
The retained worker volume will be automatically deleted once there is a successful S3 copy.
This KB will show how to copy worker logging for support.
Retrieving these logs will require mounting the volume to a Linux instance such as the CPM server.
The below steps use the CPM server, but nearly any distribution of Linux can be used.
Mounting to the CPM server will allow retrieving the logs using "download logs" in the CPM UI, please follow the below steps carefully.
Mounting the worker root volume to the CPM server
Attach the worker volume to an instance:
First, find the detached worker volume in EBS. The volume ID can be found in the backup log.
If the volume cannot be found, double check the session in Backup Manager is correct and the volume ID is not mistyped.
Next, attach this volume to the CPM server instance.
the CPM server instance cannot be found when attaching the volume,
check that region and availability zone of the worker volume match the CPM
Mounting the volume within Linux:
Once attached, the volume needs to be mounted within the instance. The below bash commands are specific to the CPM server
Using an SSH client of your choosing, make a connection and log into the CPM server. The CPM server uses key authentication (key selected in AWS when the CPM server was first launched) and user: cpmuser
Use the following command to find the volume:
The output will show all volume and partition names. The S3 worker partition will be "part" in the TYPE columnand will have a blank MOUNTPOINT,
The correct partition is "xvdg1" in this example.
Note: the partition name may be different on your CPM server. Be sure to check you have found the correct partition!
Create a mount point for the worker partition by running these commands:
cd / sudo mkdir /worker
Now mount the worker volume to the CPM server with the following command, replace "<partition name from step 3>" with the name of the partition found in step 3 above:
sudo mount /dev/<partition name from step 3> /worker
Confirm the mount by going to /workerand listing the directory. You should see the file system of the worker volume:
cd /worker ls
Getting worker logs from the mounted volume:
Compress the vmbapi folder
sudo tar -zcvf /tmp/workerlogpack.tar.gz /worker/var/log
Find workerlogpack.tar.gz in /tmp
cd /tmp ls
At this point, workerlogpack.tar.gz can be copied from the instance using FTP or another method, and this log package can be attached to the support case. See below for clean up.
Alternatively, if the worker volume has been mounted to the CPM server, its possible to copy this log pack to the CPM log directory, then retrieve the log package in the UI.
To copy the log package from /tmp to the CPM log directory run this command: