System Upgrade/Downgrade

You can upgrade/downgrade your CipherTrust Manager firmware to and from 2.22 by securely downloading and applying a new/older system archive file.

Specific upgrade steps vary depending on whether the CipherTrust Manager is clustered or unclustered.

Downgrade is only supported on unclustered appliances.

Upgrading a k160 Thales TCT Device

The k160 Thales Standard model TCT device has a different supported upgrade path than all other CipherTrust Manager models. For this device, the supported upgrade path are:

• 2.11.2-tct to 2.15.0 to 2.16.x to 2.20.x to 2.21.x to 2.22.x

• 2.11.2-tct to 2.15.0 to 2.16.x to 2.20.x to 2.22.x

• 2.11.2-tct to 2.15.0 to 2.20.x to 2.22.x

The High Assurance (HA) model is not supported with this release.

The supported upgrade methods are unclustered/standalone and cluster remove/rebuild.

System Upgrade for an Unclustered Appliance

Refer to Cluster Upgrade for details on upgrading a CipherTrust Manager which is part of a cluster of devices.

For 2.23.0, we tested upgrade from 2.11.8, 2.22.0, 2.21.0, and 2.20.0 for the CipherTrust Manager k570 and k470 physical appliances, and Virtual CipherTrust Manager k170v and k470v instances. For the CipherTrust Manager k160 small form factor appliance, consult the Upgrade Paths page for supported paths.

You require ksadmin level access with an SSH key.

1. Obtain the signed archive file for the upgrade from the Support Portal. The file has the format ks_upgrade_<major.minor.patch+build_number>.tar.gz.gpg.

2. On CipherTrust Manager create and download a backup with corresponding backup key, in case there are any problems.

   Note

   Upgrades keep all the data and may migrate the data and configuration. Therefore, as a precaution, it is recommended to take a backup before upgrading.

   Consult the backups page for details on forward compatibility for backups. Restoring a newer backup to an older version is never supported.

   Caution

   If local database audit logging is enabled by restoring a backup, system operation performance can degrade. We recommend disabling local database audit logging before taking a backup.

3. Transfer the archive file to the CipherTrust Manager. You require the private SSH key associated with the ksadmin account. The command and supported private key format depends on the operating system you are transferring the archive file from.

   Source Operating System	Example Command Syntax	Required Private Key Format
   Linux	scp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	OpenSSH format
   Windows	pscp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	PuTTY PPK format

4. ssh into the CipherTrust Manager as ksadmin and ensure there is sufficient disk space available. The required disk space is 5 times the size of the upgrade file. For example, if the upgrade file is 5 GB, 25 GB of free space is required. Use df -h / to view available space.

5. Run the following command:

   sudo /opt/keysecure/ks_upgrade.sh -f <archive_file_path>

   Here, <archive_file_path> specifies the CipherTrust Manager path to the signed archive file.

   The signature of the archive file is verified and the upgrade is applied.

6. Reboot the appliance when prompted. The following output indicates the upgrade has completed:

   Upgrade succeeded
   Reboot the instance to finish the upgrade
   

   Note

   If you have lost the SSH session, you can access upgrade logs in /home/ksadmin/. Upgrade log files have the prefix ks_upgrade. Search for "Upgrade succeeded" to confirm the upgrade completed, and that you must reboot. Take care to verify the upgrade has completed before rebooting. You must always reboot to fully apply the upgrade.

7. Ensure the CipherTrust Manager services have started. From the ksadmin session, run systemctl status keysecure. Alternatively, you can visit the CipherTrust Manager web console or attempt to connect with the ksctl CLI.

8. If you need to verify the new version, look for the version displayed in CipherTrust Manager banner in the SSH session on reboot, as shown in the screenshot. Alternatively, log in the web console UI and click the information icon at the top right to display version information.

   

9. Re-download and re-install ksctl to access commands and settings introduced in the new CipherTrust Manager software version.

   Note

   Previous versions of ksctl still work with new CipherTrust Manager software for existing features and settings.

System Downgrade

An unclustered CipherTrust Manager can be downgraded to the previous minor version. For release-specific upgrade/downgrade information, refer to the release notes for your release.

Downgrades perform a CipherTrust Manager reset, which wipes all CipherTrust Manager data except the backup files that already exist.

As well, the PCI HSM drivers on k570 models, and base operating system packages are not changed during downgrade.

To downgrade your CipherTrust Manager

1. SSH into the CipherTrust Manager as "ksadmin".

2. Downgrade the CipherTrust Manager:

   $ sudo /opt/keysecure/ks_downgrade.sh -f <~/filename>
   

Usage: ks_downgrade.sh -f <FILE> [-o]

*  `-f`: Path to the signed CipherTrust Manager installer file.

*  `-o`: Clustered node cannot be downgraded. Use this flag to override this behavior. -->

Cluster Upgrade

A cluster can be upgraded in three ways:

• Online In-place Cluster Upgrade

• Offline In-place Cluster Upgrade

• Using the Cluster Remove/Rebuild Method

The following table indicates the differences between the three ways:

Prerequisites

• You require ksadmin level access with an SSH key.

• Obtain the signed archive file for the upgrade from the Support Portal. The file has the format ks_upgrade_<major.minor.patch+build_number>.tar.gz.gpg.

Online In-place Cluster Upgrade

Be aware of the following considerations when performing an in-place cluster upgrade:

• You can only upgrade from 2.22.0 to 2.23.0 with this upgrade method. Due to better lifecycle management for system critical components introduced in 2.22.0, you cannot use in-place cluster upgrade to upgrade to 2.22.0. If you are starting at a lower version than 2.22.0, we recommend using cluster in-place upgrade to 2.21.0, and then using cluster remove/rebuild to upgrade from 2.21.0 to 2.23.0.

• The node being upgraded will be inaccessible during the upgrade. This may be as long at 10 or more minutes. Clients must be able to handle this outage.

• There will be a brief period of time (under 30 seconds) where the database will be locked while upgrading the first node. This affects all nodes at the same time, and some nodes may give error responses during this time.

• All nodes in the cluster should be upgraded as soon as possible - nodes running different version of the firmware will behave differently, potentially causing problems with applications.

• This type of upgrade is not supported for the CipherTrust Manager k160 appliance. Only cluster remove/rebuild is supported for k160 clusters.

To perform an online in-place cluster upgrade

1. Before doing any upgrade operation, ensure that you have a backup, and that you have downloaded the backup and associated backup key.

1. Ensure all nodes in the cluster are up and operating normally. Resolve any issues (like removing any obsolete nodes) before performing the upgrade.

2. Perform a system upgrade on each node, one at a time.

   1. Transfer the archive file to the CipherTrust Manager. You require the private SSH key associated with the ksadmin account. The command and supported private key format depends on the operating system you are transferring the archive file from.

      Source Operating System	Example Command Syntax	Required Private Key Format
      Linux	scp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	OpenSSH format
      Windows	pscp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	PuTTY PPK format

   2. SSH into the CipherTrust Manager as ksadmin and ensure there sufficient disk space available. The required disk space is 5 times the size of the upgrade file. For example, if the upgrade file is 5 GB, 25 GB of free space is required. Use df -h / to view available space.

   3. Run the following command to upgrade the device.

      $ sudo /opt/keysecure/ks_upgrade.sh -f <archive_file_path>
      

      Here, <archive_file_path> specifies the CipherTrust Manager path to the signed archive file.

      The signature of the archive file is verified and the upgrade is applied.

   4. Reboot the appliance as prompted. The following output indicates the upgrade has completed:

      Upgrade succeeded
      Reboot the instance to finish the upgrade
      

      Note

      If you have lost the SSH session, you can access upgrade logs in /home/ksadmin/. Upgrade log files have the prefix ks_upgrade. Search for "Upgrade succeeded" to confirm the upgrade completed, and that you must reboot. Take care to verify the upgrade has completed before rebooting. You must always reboot to fully apply the upgrade.

   5. Ensure the upgrade of each node is complete and that the node is operating normally, before proceeding to the next node.

      From the ksadmin session, run systemctl status keysecure to ensure that the CipherTrust Manager services have started. Alternatively, you can visit the CipherTrust Manager web console or attempt to connect with the ksctl CLI.

      Note

      When updating the first node in a cluster, the cluster nodes may briefly experience slower than usual response times. This occurs because the shared database schema for the cluster is updated with the first node.

   6. If you need to verify the new version, look for the version displayed in CipherTrust Manager banner in the SSH session on reboot, as shown in the screenshot. Alternatively, log in the web console UI and click the information icon at the top right to display version information.

      

1. Re-download and re-install ksctl to access commands and settings introduced in the new CipherTrust Manager software version.

   Note

   Previous versions of ksctl still work with new CipherTrust Manager software for existing features and settings.

Offline In-Place Cluster Upgrade

If your target CipherTrust Manager version is a Long Term Support (LTS) release, you can perform an offline upgrade. For 2.23.0, upgrade is supported from 2.11.8.

Prerequisites for Offline Cluster Upgrade

• Your CipherTrust Manager must be at version 2.11.8.

• You must have the 1.2 version of the cmupdate utility.

• You need to login as ksadmin with an SSH key from your client workstation to each node individually to accept the host fingerprint.

  Note

  For Windows, you need to be able to both use SSH and run the cmupdate tool in the Terminal application, as this application accepts special ANSI characters. You cannot use a separate SSH client utility such as PuTTY or other Windows command line environments.

• Ensure that you have a backup, and that you have downloaded the backup and associated backup key.

• Ensure all nodes in the cluster are up and operating normally. Resolve any issues (like removing any obsolete nodes) before performing the upgrade.

To perform an offline in-place cluster upgrade

1. Obtain the cmupdate utility for your client workstation operating system and install it.

   The following cmupdate utilities are available:

   • cmupdate-linux-amd64 for 64-bit Linux system

   • cmupdate-windows-amd64.exe for 64-bit Windows system.

     Note

     Windows 10 is the minimum supported version for the client workstation.

   • cmupdate-darwin-amd64 for 64-bit macOS system with Intel processor.

     Note

     For some versions of macOS, a prompt appears indicating that macOS cannot verify the developer. You need to dismiss the prompt and navigate into Security & Privacy settings to manually allow cmupdate.

2. Rename the utility to cmupdate, without the OS-specific suffix.

3. Run cmupdate in a terminal or command prompt program with full privileges to confirm it is set up properly.

   If successful, text similar to the following is displayed:

   Command line tool for upgrading clustered and standalone CipherTrust Managers
   
   Usage:
     cmupdate [command]
   
   Available Commands:
     help        Help about any command
     prepare     Prepare the CM(s) for a system upgrade
     upgrade     Perform a system upgrade on a standalone CM or multiple nodes in a cluster
     version     Output tool version
   
   Flags:
     -h, --help      help for cmupdate
     -v, --verbose   Provide verbose output while executing command (optional)
   
   Use "cmupdate [command] --help" for more information about a command.
   

4. Run the cmupdate prepare command to check if the cluster is ready for upgrade and to upload the upgrade file. You require the upgrade file and the SSH key(s) used for the CipherTrust Manager nodes. You also require network information for one or more nodes, depending on whether the cluster uses only one SSH key or multiple SSH keys.

   1. Run the command.

      One SSH key common to the cluster

      For this command you require only one node's hostname or IP address.

      cmupdate prepare --identities <common_ssh_private_key> --nodes <hostname_or_IP_address_of_a_node> --file <upgrade_file_path> --all
      

      Multiple SSH keys

      For this command, you require the IP address or hostname of every node in the cluster.

      cmupdate prepare --identities <ssh_private_key_1,ssh_private_key_2 ...> --nodes <hostname_or_IP_address_of_node1, hostname_or_IP_address_of_node2 ...> --file <upgrade_file_path>
      

      Caution

      You must provide the SSH key identities and their associated nodes in the same order. For example, the first specified SSH key should be associated with the first specified node, and the second specified SSH key should be associated with the second specified node.

      The prepare command outputs to a log file in your local directory.

      Note

      The upgrade file is uploaded to all nodes, which can take a long time depending on network speed. In a separate session, you can SSH as the ksadmin user and run the ls command in the /home/ksadmin directory to check the progress of the upgrade file upload.

   2. If the cmupdate prepare command output identifies any problems, correct them and repeat the cmupdate prepare command.

      These may include:

      • Insufficient disk space.

        • Check the retention setting for scheduled backups on each node to see if many backup files are being retained. You might need to manually transfer some backup files off the appliance.

        • Disable local database audit logging on one node. Local database logging can occupy a lot of disk space.

      • Cluster nodes not in active state.

        This message indicates that one or more nodes are not fully joined to the cluster, with a status of ready. Check the status of cluster nodes. If the status is joining, wait for the join operation to complete. If the status is nodes are in different states across databases, not clustered, killed, down, or unknown, attempt a rejoin operation.

      • Audit records are being written to the database.

        Disable local database audit logging on one node.

      • Cannot verify the signature of the upgrade file.

        Re-download the upgrade file from support portal. Contact customer support if the issue happens again.

      • Cluster nodes have different versions.

        Use one of the online in-place cluster upgrade or cluster remove/rebuild methods to bring up all the nodes to the same starting CipherTrust Manager version.

5. Stop as much CipherTrust Manager traffic as possible. The cluster will not be available for part of the ongoing upgrade.

6. Run the cmupdate upgrade command, which checks the uploaded upgrade file matches the specified upgrade file, disables the REST API for all nodes, and then upgrades each node and re-enables the REST API. During some of the upgrade, the cluster is unavailable to incoming traffic. You require the upgrade file and the SSH key(s) used for the CipherTrust Manager nodes. You also require network information for one or more nodes, depending on whether the cluster uses only one common SSH key or multiple SSH keys.

   One SSH key common to the cluster

   For this command you require only one node's hostname or IP address.

   cmupdate upgrade --identities <common_ssh_key> --nodes <hostname_or_IP_address_of_a_node> --file <upgrade_file_path> --all
   

   Multiple SSH keys

   For this command, you require the IP address or hostname of every node in the cluster.

   cmupdate upgrade --identities <ssh_private_key_1,ssh_private_key_2 ...> --nodes <hostname_or_IP_address_of_node1, hostname_or_IP_address_of_node2 ...> --file <upgrade_file_path>
   

   Caution

   You must provide the SSH key identities and their associated nodes in the same order. For example, the first specified SSH key should be associated with the first specified node, and the second specified SSH key should be associated with the second specified node.

7. Once the upgrade completes and the REST API is re-enabled, you can restart any client requests.

8. Re-download and re-install ksctl to access commands and settings introduced in the new CipherTrust Manager software version.

   Note

   Previous versions of ksctl still work with new CipherTrust Manager software for existing features and settings.

9. If you have an active Data Discovery and Classification (DDC) integration, you must re-assign the active DDC node as if you are performing a fresh installation. We recommend selecting the same node that was active before the upgrade.

   Note

   This DDC configuration requirement is specific to the 2.11.8 to 2.23.0 upgrade.

Cluster Remove/Rebuild Upgrade

Supported upgrade paths are the same as for system upgrade of an unclustered appliance.

Prerequisite for Cluster Remove/Rebuild Method

Plan for downtime when clients cannot make requests to the cluster and manually block client access to the CipherTrust Manager IP addresses/hostnames in your network before starting.

To Perform an Upgrade Using Cluster Remove/Rebuild Method

1. On one of the cluster nodes, create and download a backup with corresponding backup key, in case there are any problems.

2. Remove all nodes from the cluster except one.

3. Delete the cluster configuration on the remaining node.

   $ ksctl cluster delete
   

4. Perform the upgrade on that remaining node.

   1. Transfer the archive file to the CipherTrust Manager. You require the private SSH key associated with the ksadmin account. The command and supported private key format depends on the operating system you are transferring the archive file from.

      Source Operating System	Example Command Syntax	Required Private Key Format
      Linux	scp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	OpenSSH format
      Windows	pscp -i <path_to_private_SSH_key> <archive_file_name> ksadmin@<ip>:.	PuTTY PPK format

   2. SSH into the CipherTrust Manager as ksadmin and ensure there sufficient disk space available. The required disk space is 5 times the size of the upgrade file. For example, if the upgrade file is 5 GB, 25 GB of free space is required. Use df -h / to view available space.

   3. Run the following command to upgrade the device.

      $ sudo /opt/keysecure/ks_upgrade.sh -f <archive_file_path>
      

      Here, <archive_file_path> specifies the CipherTrust Manager path to the signed archive file.

      The signature of the archive file is verified and the upgrade is applied.

   4. Reboot the appliance as prompted. The following output indicates the upgrade has completed:

      Upgrade succeeded
      Reboot the instance to finish the upgrade
      

      Note

      If you have lost the SSH session, you can access upgrade logs in /home/ksadmin/. Upgrade log files have the prefix ks_upgrade. Search for "Upgrade succeeded" to confirm the upgrade completed, and that you must reboot. Take care to verify the upgrade has completed before rebooting. You must always reboot to fully apply the upgrade.

   5. From the ksadmin session, run systemctl status keysecure to ensure that the CipherTrust Manager services have started. Alternatively, you can visit the CipherTrust Manager web console or attempt to connect with the ksctl CLI.

   6. If you need to verify the new version, look for the version displayed in CipherTrust Manager banner in the SSH session on reboot, as shown in the screenshot. Alternatively, log in the web console UI and click the information icon at the top right to display version information.

      

5. After the appliance reboots, re-build the cluster by creating a new cluster on this node.

6. Perform the upgrade on all other removed nodes, as outlined in step 4.

   Note

   If a previously used node is to be re-used, the cluster must first be deleted from that system.

7. Join new instances to the cluster.

8. Re-establish client access to the cluster.

9. Re-download and re-install ksctl to access commands and settings introduced in the new CipherTrust Manager software version.

   Note

   Previous versions of ksctl still work with new CipherTrust Manager software for existing features and settings.

Upgrade Troubleshooting

The following section lists some commonly encountered errors during upgrade and their solutions.

Occasional Error for Devices Upgraded from NextGen KeySecure 1.6 or 1.7

CipherTrust Manager appliances that at one time ran NextGen KeySecure software versions 1.6 or 1.7 occasionally fail to upgrade on future versions. The error reported is E: Unable to locate package docker-compose-plugin.

Solution: Contact customer support to resolve this state.