What steps can you take to mitigate crashes due to performance issues on the CPM Instance.

What steps can you take to mitigate crashes due to performance issues on the CPM Instance.

Background
The following contains steps that customers can enact to help performance issues like excess memory or CPU or crashes.
This KB article assumes that the CPM instance is sized correctly and has the correct volume type as per the Recommended instance sizes/volume types for CPM Server instances article.
These changes should be made at a time when there are no active CPM jobs, DR copy or S3 copy jobs are running.

Note: If you are running hourly backup jobs it is highly recommended to spread hourly backups into 4 smaller groups, that would run at different times within the hour for example - at hh:00, hh:15, hh:30 and hh:45. This will significantly reduce the load on CPM .

1. Please stop and start the CPM instance when no backups or DR/S3 copy jobs are active. This is done because the virtual hardware is refreshed and can help correct virtual hardware issues.
Note: A reboot of the CPM instance does not refresh the virtual hardware as does the stop and start.

 

2. There are some MySQL configuration changes that can change the way logging takes place and increasing the write buffer, 

  1. A.Connect to the CPM instance using ssh and the login cpmuser and the instances private key.
  1. B. sudo su -
  1. C. Please edit the file /etc/mysql/mysql.cnf using a text editor like nano or vim and add the following lines below any text in the file and then save the file.

 

Add at the end of the file:

[mysqld]

innodb_read_io_threads=4

innodb_write_io_threads=8 #To stress the double write buffer

innodb_log_file_size = 32M #Small log files, more page flush

innodb_log_files_in_group=2

innodb_file_per_table=1

innodb_log_buffer_size=8M

innodb_flush_method=O_DIRECT

innodb_flush_log_at_trx_commit=0

max_allowed_packet=128M

 

  1. D. For these changes to take effect you must recycle the MySQL process, so please run the command.
  1. sudo systemctl restart mysql
  2. E. Then verify that the service is active by typing:
  1. sudo systemctl status mysql

3. If you are facing memory issues you can create a swap file on the CPM instance.  Swap is a space on a disk that is used when the amount of physical RAM memory is full. (This is like a paging file on the Windows platform,).  When running CPM on an Ubuntu AWS instance a swap partition is not configured by default, and the only option is to create a swap file.
You can follow this N2WS KB article to create a swap file.

4. Performance issues can be caused by the cpmdata policy getting corrupted. The following steps show how you can "recreate" the cpmdata policy. When doing this please make sure that the application-consistent backup option is disabled.

  1. A. Disable the cpmdata policy ( do not delete it!) once the cpmdata policy is disabled, please go create a new policy and name it "mycpmdata". 
  1. B. Then in the policy list, click Backup targets next to this new policy and choose volumes - add the cpmdata volume to this policy (only add the cpmdata volume, do not add the instance) This will begin to back up the cpmdata volume in a different way but is still recoverable in the event of a complete recovery of the CPM instance is necessary.
  2. C. Once this is setup please wait until the CPM server has no active-backup or dr copies running, then stop/start the instance (do not use the reboot option in AWS)
5. If after taking these steps above and the performance issue still remains we would ask you to open an N2WS Support issue and to first gather the following information and logs to help speed the support ticket up. 
  1. A. Please gather a complete set of CPM logs and you can follow the directions in this KB article that details the necessary data for a support issue.
  2. B.  Please connect to the CPM instance via ssh and use the login cpmuser and the instances private key. Then run the following commands and send us the output files that will show us what process is using the most memory and or CPU.
  3. sudo su -
  4. free -h >free,txt  
  5. df -k >df.txt
  6. top >top.txt
  7. cat /etc/msql/mysql.cnf >/tmp/mysql.txt    Note: this command output file will be placed in /tmp so locate that file in that directory.