Prometheus Metrics Endpoint

You can use the Prometheus metrics endpoint to connect the Prometheus monitoring system to CipherTrust Manager. You can set Prometheus to scrape the CipherTrust Manager continuously, providing metrics over time to help monitor overall system health, performance, and cryptographic activity.

A sample configuration with Prometheus and Grafana docker images is available on Github. The Grafana data visualization application provides graph visualizations of the Prometheus-collected metrics.

Prerequisites for Sample Configuration

• CipherTrust Manager 2.7.0 or later

• Docker

• Docker Compose (docker-compose)

Sample Configuration Setup

1. On your CipherTrust Manager, enable Prometheus metrics, either through a POST to the /v1/system/metrics/prometheus/enable endpoint, or with the ksctl metrics prometheus enable CLI command.

   A token is returned, which Prometheus needs to scrape CipherTrust Manager.

   Note

   This token does not expire, but can be manually renewed with ksctl metrics prometheus renew-token or a POST to /v1/system/metrics/prometheus/renew-token.

2. Get the token which Prometheus needs to scrape CipherTrust Manager, if needed. You can use GET with the /v1/system/metrics/prometheus/status endpoint or ksctl metrics prometheus status.

3. In the Prometheus Metrics directory, edit the prometheus.yml file.

   At minimum, you must provide the CipherTrust Manager hostname/IP in targets and the prometheus API token in bearer token. Prometheus can scrape multiple CipherTrust Managers, which might or might not share the API Token. This is an example configuration file with three CipherTrust Manager nodes, of which two share the same Prometheus API token:

   scrape_configs:
     - job_name: "CipherTrust Manager"
       scheme: "https"
       tls_config:
         #ca_file: "/trusted_cas/web-keysecure-local.pem"
         #server_name: "web.keysecure.local"
         insecure_skip_verify: true
       bearer_token: "1zplR4njZsRN5dNeWAFXhkL1x7MU9q4H"
       metrics_path: "/api/v1/system/metrics/prometheus"
       static_configs:
         - targets:
           - "1.1.1.1"
           - "1.1.1.2"
     - job_name: "CipherTrust Manager Staging"
       scheme: "https"
       tls_config:
         #ca_file: "/trusted_cas/web-keysecure-local.pem"
         #server_name: "web.keysecure.local"
         insecure_skip_verify: true
       bearer_token: "TnRHpdL9v8MnWv8DhN9xuAaKgPevMEZs"
       metrics_path: "/api/v1/system/metrics/prometheus"
       static_configs:
         - targets:
           - "1.1.1.3"
   

4. Set up TLS authentication. By default, the Prometheus configuration sets insecure_skip_verify: true which is not recommended for production deployments as it skips SSL/TLS certificate validation for the CipherTrust Manager server.

   1. On CipherTrust Manager, download the certificate associated with the web interface. Export to a pem format.

      ksctl interfaces certificate get --name web --icertfile <desired-filename>.pem
      

   2. Use openssl to retrieve the Common Name (CN) of the certificate, which will become the server_name value in Prometheus.

      openssl x509 -noout -subject -in <your-file>.pem
      

      Example response:

      subject=C = US, ST = MD, L = Belcamp, O = Gemalto, CN = web.keysecure.local
      

      The CN value, web.keysecure.local, is the value needed for Prometheus.

   3. Copy the certificate file to the trusted_cas folder in the Prometheus Metrics directory.

   4. Edit the prometheus.yaml file to include the ca_file path and server_name of the certificate, and disable the insecure_skip_verify parameter. For example:

      scrape_configs:
        - job_name: "CipherTrust Manager"
          scheme: "https"
          tls_config:
            ca_file: "/trusted_cas/web-keysecure-local.pem"
            server_name: "web.keysecure.local"
            #insecure_skip_verify: true
          bearer_token: "TnRHpdL9v8MnWv8DhN9xuAaKgPevMEZs"
          metrics_path: "/api/v1/system/metrics/prometheus"
          static_configs:
            - targets:
              - "1.1.1.1"
      

5. In the Prometheus directory run make up to start the stack.

   Note

   You can run make down to stop the stack and make clear to stop the stack and all persisted data.

6. Visit the Prometheus Dashboard in a browser at http://localhost:9090.

   1. Navigate to Status > Target to ensure that Prometheus is scraping CipherTrust Manager. The state should display as UP for each node, with no errors.

   2. If you detect a problem, verify the metrics endpoint on CipherTrust Manager with ksctl metrics prometheus get --api-token <api-token>, or curl -k 'https://<hostname>/api/v1/system/metrics/prometheus' -H 'Authorization: Bearer <api-token>' --compressed ). You can also use The Docker Compose logs to debug problem, with docker-compose logs -f.

7. Visit the Grafana Dashboard in a browser at http://localhost:3000.

   1. Login with the user admin and the password admin. Set a new password when prompted.

   2. Go to Dashboards -> Home to view the included dashboards.

Metrics Prefixes

The following high-level categories of metrics are returned from the endpoint:

Available Metrics Dashboards

The following dashboards are displayed in Grafana for CipherTrust Manager:

• CipherTrust Manager Developer - Metrics relevant to internal CipherTrust Manager developers to debug problems. This includes:

  • Average JWT processing time

  • Applications and Accounts Totals

  • Key Encryption Key (KEK)s Count

  • Authorization Policies Cache Hits pr Minute

  • Average Prometheus Metrics Scraping Response Time

• CipherTrust Manager Host - Metrics about the health of the CipherTrust Manager host, including CPU details, memory details, network details, network connections, and disk details.

• CipherTrust Manager HTTP Traffic - Metrics about HTTP traffic to the CipherTrust Manager. This includes:

  • Average HTTP Response Time Per Minute

  • HTTP Requests in the Last Minute

  • Average Network Latency Per Minute

  • Average CM HTTP Client Response Time Per Minute

  • HTTP 500 Errors in the Last Minute

• CipherTrust Manager NAE - Basic metrics about the performance of the NAE-XML cryptographic interface. This includes XML response time and XML processing time.

• CipherTrust Manager NAE Developer - More detailed metrics about operations and performance on the NAE-XML interface, intended for debugging. This includes:

  • Key Info Cache Misses Time Per Minute

  • Key Info Cache Hits Time Per Minute

  • XML Total Processing time

  • XML Parsing Time

  • XML Transmit Time

  • XML Receive Time

  • XML Execution Time

• CipherTrust Manager Resources - Metrics about creation and use of objects on CipherTrust Manager, such as audit records, keys, licenses, backup, and users. This includes:

  • Audit Records Created Per Second Over The Last Minute

  • Audit Records Created In Last Five Minutes

  • Total Number of Audit Records

  • Total Number of Keys By Algorithm

  • Crypto Operations Per Second Over The Last Minute

  • Total Number Of Connector Licenses Deployed

  • Number of License Units Consumed

  • License Unit Consumption by Percentage

  • Total Number Of Group Users in the System

  • Total Number Of Key Rotations

  • Key Rotations In Last Five Minutes

  • Time Taken To Create Backup

  • Number of Backups taken

  • Total number of Keys by States and Algorithm

• CipherTrust Manager Services - Metrics about the performance of individual microservices within CipherTrust Manager, intended for debug purposes. This includes:

  • CPU percentage

  • Memory usage

  • Network I/O (transmitting and receiving)

  • Disk I/O (reading and writing)

• CipherTrust Manager Node Metrics - Metrics of the nodes in a clustered system showing the node connection information. This includes:

  • Write, flush, and replay lags

  • Sent, write, flush, and replay lag sizes

  • Whether replication is blocked

  • Whether node is connected

  • Connect time for a node

  • Apply rate

  • Catchup interval

  • Uptime for a connection

• CipherTrust Manager CTE Resources - More detailed metrics about operations and performance on the CTE resources, intended for debugging. This includes:

  • CTE Clients

  • CTE Clients Health Status

  • CTE Groups

  • CTE Guard Points State

• CipherTrust Manager Secrets - Metrics for the various secrets (Akeyless) gateway operations. This includes:

  • CPU utilization metrics

  • Disk I/O metrics

  • CPU load metrics

  • Memory utilization metrics

  • Network interface I/O metrics & TCP connection metrics

  • Current transaction number

  • Total transaction by an admin client

  • Total transaction limit per hour

  • Status of HTTP response

  • Total number of requests

  For more details, click here.