N2W Software

            Worker did not establish connection / 0% S3 copy progress for extended period of time

            You may encounter the message "Worker did not establish connection" in the log of an S3 copy or S3 restore operation.


                  Day mm/dd/yyyy HH:MM:SS - Error - Worker i-1234567890abcdef did not establish connection - terminating operation

            Or you may see, the s3 copy task with the status "In progress (0%)" for an extended period of time (6+ hours), until the task fails by timeout.

            The first message indicates the S3 copy worker instance has failed to establish a network connection to the CPM server,

            Stuck in progress usually indicates the S3 copy worker could not connect to the S3 bucket.

            Check the following:

            1. The worker appliance security group settings, the following must be allowed for both backup and restore:

                       INBOUND HTTPS (port 443):  
              • To the CPM server private or public IP.
                       OUTBOUND HTTPS (port 443):     

                  Note 1: Amazon regularly rotates public gateway IPs, check current IP information here: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

                  Note 2: If restoring from S3 to a different region, select a security group at the target region which allows outbound connections to the source region S3 bucket. Please see 4. below as well.

            2. The CPM server security group settings:

                       INBOUND and OUTBOUND HTTPS (port 443):
              • To the subnet the worker is configured to use.

            3. If using an ACL, be sure the proper ports are open for bi-directional communication between the worker and S3: https://aws.amazon.com/premiumsupport/knowledge-center/connect-s3-vpc-endpoint/
             Also be sure access to the CPM server is allowed via port 443 where necessary.

            4. The worker will connect to S3 via internet, even if the CPM server and S3 bucket are in the same region.
            • The worker must have a public IP assigned by the subnet; -or- the subnet must be configured with a NAT gateway, for the worker to communicate bi-directionally with S3.

            • Only if the EBS backups and S3 bucket are in the same region, this can be worked around by configuring an S3 endpoint as shown here: https://n2ws.com/support/documentation/appendix-s3-configuration

            5. The worker must be able to resolve the DNS name of the region with your S3 bucket. (e.g. https://s3.us-west-2.amazonaws.com/). Please ensure the worker will be able to resolve DNS names of S3 endpoints.

            6. If an HTTP proxy is necessary to access public IPs, ensure this is properly configured in the worker settings and not blocking the transfer to s3. It may be advisable to test by opening internet to the worker without a proxy in some cases to be sure the proxy is not causing an issue.

            Help us to make this article better
            0 0