N2W Software

            Retreive support logging from a failed S3 worker root volume

            In some cases where an S3 copy fails the root volume of the worker is kept and logging is not automatically transferred to the CPM server.
            In order to troubleshoot the S3 copy issue, support may ask you to collect a copy of logging from the retained worker volume.

            • Error - Backup copy failed
            • Info - Root volume of failed worker i-1234567890abcdef (vol-1234567890abcdef) has been retained
            The retained worker volume will be automatically deleted once there is a successful S3 copy.

            This KB will show how to copy worker logging for support.

            Retrieving these logs will require mounting the volume to a Linux instance such as the CPM server.

            The below steps use the CPM server, but nearly any distribution of Linux can be used.

            Mounting to the CPM server will allow retrieving the logs using "download logs" in the CPM UI, please follow the below steps carefully.

            Mounting the worker root volume to the CPM server

            Attach the worker volume to an instance:

            First, find the detached worker volume in EBS. The volume ID can be found in the backup log.

            If the volume cannot be found, double check the session in Backup Manager is correct and the volume ID is not mistyped.

            Next, attach this volume to the CPM server instance.

            If the CPM server instance cannot be found when attaching the volume, check that region and availability zone of the worker volume match the CPM server.

            Mounting the volume within Linux:

            Once attached, the volume needs to be mounted within the instance.
            The below bash commands are specific to the CPM server
            1. Using an SSH client of your choosing, make a connection and log into the CPM server.
              The CPM server uses key authentication (key selected in AWS when the CPM server was first launched) and user: cpmuser

            2. Use the following command to find the volume:

                    sudo lsblk


            3. The output will show all volume and partition names.
              The S3 worker partition will be "part" in the TYPE column and will have a blank MOUNTPOINT,

                    The correct partition is "xvdg1" in this example.

              Note: the partition name may be different on your CPM server.
              Be sure to check you have found the correct partition!

            4. Create a mount point for the worker partition by running these commands:

                    cd /
                    sudo mkdir /worker


            5. Now mount the worker volume to the CPM server with the following command, replace "<partition name from step 3>" with the name of the partition found in step 3 above:

                    sudo mount /dev/<partition name from step 3> /worker


            6. Confirm the mount by going to /worker and listing the directory. You should see the file system of the worker volume:

                    cd /worker


            Getting worker logs from the mounted volume:

            1. Compress the vmbapi folder

                    sudo tar -zcvf /tmp/workerlogpack.tar.gz  /worker/var/log


            2. Find workerlogpack.tar.gz in /tmp

                    cd /tmp


            At this point, workerlogpack.tar.gz can be copied from the instance using FTP or another method, and this log package can be attached to the support case. See below for clean up.

            Alternatively, if the worker volume has been mounted to the CPM server, its possible to copy this log pack to the CPM log directory, then retrieve the log package in the UI.
            1. To copy the log package from /tmp to the CPM log directory run this command:

                     sudo cp /tmp/workerlogpack.tar.gz /var/log/cpm/workerlogpack.tar.gz


              The copy can be confirmed with this command:

                    ls /var/log/cpm


            2. Next, generate the log package from the CPM server by clicking "download logs"


            3. Finally, upload this CPM log package to the support case.

            The next steps show how to remove the mount and clean up created files.

            Clean up:

            1. Unmount the volume with these commands:
                    cd /
                    sudo umount /dev/xvdg1

            2. Remove the mount point folder with this command:
                    sudo rmdir /worker

            3. Remove the temporary log package with this command:
                    sudo rm /tmp/workerlogpack.tar.gz

            4. If logs were copied to the CPM server log directory, remove the log package with this command:
                    sudo rm /var/log/cpm/workerlogpack.tar.gz

            5. The worker root volume can now be detached from the instance in AWS. You can delete this volume, or CPM will automatically once the S3 copy is successful.



            Help us to make this article better
            0 0