Prometheus

Scrape VM host capacity into Prometheus. One job, one target, every 5 minutes.

This is the fleet snapshot from VM host capacity, rendered as Prometheus text. It is host CPU, memory, storage, traffic, and VM count. It is not per-VM guest metrics and it is not RRD history.

Scrape job

Add one job. The target is the API host. The path is a separate setting and must start with /. The resource name is devices (plural).

scrape_configs:
  - job_name: devicevmhost
    scheme: https
    metrics_path: /api/rest/v2.10/devices/vmhost/prometheus
    scrape_interval: 5m
    scrape_timeout: 30s
    basic_auth:
      username: API_UUID
      password_file: /etc/prometheus/rackcorp_api_secret
    static_configs:
      - targets:
          - api.rackcorp.net

password_file is the file that contains the API secret and nothing else (no newline is required; a trailing newline is fine). Mode 0640, owner root, group prometheus. password: with the secret inline also works. Keep that file out of git.

Setting Value
scheme https
metrics_path /api/rest/v2.10/devices/vmhost/prometheus
targets api.rackcorp.net only. Do not put the path or a URL scheme in the target.
scrape_interval 5m. Put it on this job even when the global interval is 15s or 1m.
scrape_timeout 30s. The first scrape of a 5-minute window runs the query. Later scrapes in that window read a cache.
TLS Public certificate on api.rackcorp.net. Leave insecure_skip_verify unset.

Prometheus Operator

On kube-prometheus-stack, use a ScrapeConfig. That CRD shipped in Prometheus Operator 0.65. The secret keys are the API UUID and the API secret.

apiVersion: v1
kind: Secret
metadata:
  name: rackcorp-api
  namespace: monitoring
type: Opaque
stringData:
  username: API_UUID
  password: API_SECRET
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
  name: rackcorp-devicevmhost
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  staticConfigs:
    - targets:
        - api.rackcorp.net
  scheme: HTTPS
  metricsPath: /api/rest/v2.10/devices/vmhost/prometheus
  scrapeInterval: 5m
  scrapeTimeout: 30s
  basicAuth:
    username:
      name: rackcorp-api
      key: username
    password:
      name: rackcorp-api
      key: password

The release label has to match the label your Prometheus selects on ScrapeConfig objects. kube-prometheus-stack uses the Helm release name.

Before reload

promtool check config /etc/prometheus/prometheus.yml
curl -fsS -X POST http://127.0.0.1:9090/-/reload

/-/reload needs --web.enable-lifecycle. Otherwise restart the Prometheus process after promtool check config succeeds.

API key

HTTP Basic username is the API UUID. Password is the API secret. Use an API key, not a JWT. A JWT expires and Prometheus cannot refresh it, so the target goes down on a timer.

The key needs device access that includes device.vmhost.prometheus.getall, device.vmhost.availability.getall, or device.*. The series are the VM hosts in that key's customer tree. One key is one job. A second customer is a second job with its own key and its own job name.

Check the scrape

Confirm the body before Prometheus polls it. A good response is text/plain; version=0.0.4 and the first lines are # HELP / # TYPE.

curl -fsS -D - -o /tmp/rackcorp.prom \
  -u "$API_UUID:$API_SECRET" \
  https://api.rackcorp.net/api/rest/v2.10/devices/vmhost/prometheus

promtool check metrics < /tmp/rackcorp.prom

X-Rackcorp-Scrape-Cache: miss means this response ran the query and was stored. hit means Prometheus is being served the body from the last 5 minutes. Cache-Control tells caches not to share it. Prometheus itself always performs the GET.

A JSON body means the path is wrong or the key is not allowed to call this command. # authentication required means the key did not resolve to a customer. Fix that before looking at Grafana. In Prometheus, the target should be UP under Status, Targets, job devicevmhost.

One sample per host

Each numeric series is emitted once per device_id. A line looks like this:

rackcorp_device_vm_total{device_id="24657"} 80

80 on rackcorp_device_vm_total is the planning cap, not the number of running guests. Running guests are rackcorp_device_vm_used.

Prometheus rejects a scrape that contains the same metric name and label set twice (collected metric was collected before with the same name and label values). The endpoint keeps the first row for a device_id and drops a later row, so that line appears once in the body. Confirm with:

awk '!/^#/ && NF {print}' /tmp/rackcorp.prom | sort | uniq -d

No output means the body has no repeated sample.

If a graph still shows the same host twice, the second series was scraped by another job. The job and instance labels differ:

count by (device_id, job, instance) (rackcorp_device_vm_total)

count by (device_id) (rackcorp_device_vm_total) > 1

A single job named devicevmhost and a single instance api.rackcorp.net:443 is the healthy result. The second query returns nothing. A row there means two jobs, the same target listed under two job names, or a second Prometheus remote-writing into this TSDB. Remove the extra job. Leave honor_labels at its default (false). These metrics do not carry a job or instance label of their own.

Interval

Hosts report about once a minute. The API reuses the rendered body for 300 seconds per customer. A scrape faster than 5 minutes does not return newer numbers and does not run the query again. Set scrape_interval: 5m on this job.

rackcorp_exporter_scrape_duration_seconds is the time taken to render that body. On a cache hit the value is the original render, so it stays flat until the next miss. rackcorp_exporter_build_info is always 1. Its version label is the exporter build.

Metrics

Every host series has a device_id label. Names and datacentre live only on rackcorp_device_info, which is always 1. Join it when a legend needs a hostname. Status gauges use 0 OK, 1 WARNING, 2 CRITICAL or ERROR, -1 for anything else.

Metric Labels Meaning
rackcorp_device_info device_id, device_name, dc_id, dc_name, dc_short, svc_info, svc_description, vmh_version Always 1. The labels are the host name and location.
rackcorp_device_cpu_total_cores device_id Logical CPU count.
rackcorp_device_cpu_free_cores device_id Idle cores, from the 5-minute load average.
rackcorp_device_cpu_used_cores device_id Busy cores.
rackcorp_device_cpu_utilization_ratio device_id used / total, from 0 to 1. Idle ratio is free / total.
rackcorp_device_cpu_status_code device_id WARNING below 30% idle, CRITICAL below 20% idle.
rackcorp_device_memory_total_bytes device_id Reported RAM.
rackcorp_device_memory_used_bytes device_id Reported RAM in use.
rackcorp_device_memory_free_bytes device_id Headroom after projected allocations and a 32 GB reserve, floored at 0. This is not total - used.
rackcorp_device_memory_status_code device_id Follows swap oversubscribe when swap is known: OK at or under 10% of RAM, WARNING above 10%, CRITICAL above 40%.
rackcorp_device_storage_total_bytes device_id Host data volume size.
rackcorp_device_storage_used_bytes device_id Host data volume used.
rackcorp_device_storage_free_bytes device_id Reported free space plus projected allocation. Returned as 0 when that sum is under 64 GB.
rackcorp_device_storage_status_code device_id WARNING under 500 GB free, CRITICAL under 100 GB free.
rackcorp_device_raid_healthy device_id 1 healthy, 0 not.
rackcorp_device_traffic_capacity_bits_per_second device_id, direction Link speed. direction is inbound or outbound.
rackcorp_device_traffic_used_bits_per_second device_id, direction Average bits/second since the previous sample (about 60 seconds).
rackcorp_device_traffic_free_bits_per_second device_id, direction Capacity minus used, floored at 0.
rackcorp_device_traffic_status_code device_id, direction WARNING under 200 Mbit/s free, CRITICAL under 100 Mbit/s free.
rackcorp_device_vm_used device_id Running guests. This is the VM count to graph.
rackcorp_device_vm_total device_id Planning soft cap of 80. Not a scheduler limit and not the running count.
rackcorp_device_vm_free device_id 80 - used.
rackcorp_device_vm_status_code device_id WARNING at 70 running guests, CRITICAL at 80.
rackcorp_device_out_of_date device_id 1 when the host has not checked in for 20 minutes. Leave that host out of capacity decisions until it returns to 0.
rackcorp_device_incident device_id 1 when the host row is flagged with an incident.

Divide bytes by 1024^3 for GiB. Divide bits/second by 1000^2 for Mbit/s. Field-level notes that are easy to code wrong are in vmhost-capacity.md.

PromQL

Running guests, with the hostname and datacentre on the legend:

rackcorp_device_vm_used
  * on (device_id) group_left(device_name, dc_short)
  rackcorp_device_info

Idle CPU ratio. Drop hosts that have stopped checking in:

(rackcorp_device_cpu_free_cores / rackcorp_device_cpu_total_cores)
  and on (device_id) (rackcorp_device_out_of_date == 0)

Free disk in GiB, same filter:

(rackcorp_device_storage_free_bytes / 1024 / 1024 / 1024)
  and on (device_id) (rackcorp_device_out_of_date == 0)

Outbound Mbit/s:

rackcorp_device_traffic_used_bits_per_second{direction="outbound"} / 1000 / 1000

Alerts

Page on a host that stays silent, on disk, and on the VM planning cap. for: 15m is longer than one scrape so a single slow poll does not fire.

groups:
  - name: rackcorp-vmhost
    rules:
      - alert: RackcorpVmhostStale
        expr: |
          (rackcorp_device_out_of_date == 1)
            * on (device_id) group_left(device_name, dc_short)
            rackcorp_device_info
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.device_name }} in {{ $labels.dc_short }} has not checked in"

      - alert: RackcorpVmhostDiskCritical
        expr: |
          (rackcorp_device_storage_status_code == 2)
            * on (device_id) group_left(device_name, dc_short)
            (rackcorp_device_info and on (device_id) (rackcorp_device_out_of_date == 0))
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.device_name }} has under 100 GB free"

      - alert: RackcorpVmhostVmCap
        expr: |
          (rackcorp_device_vm_status_code == 2)
            * on (device_id) group_left(device_name, dc_short)
            rackcorp_device_info
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.device_name }} is at the 80 VM planning cap"

Check the rules the same way as the scrape config: promtool check rules rackcorp-vmhost.yml.

What this endpoint is not

  • Per-VM CPU, memory, or disk. Those are VMs on one host and latest guest metrics, on demand.
  • History. Host free CPU, RAM, and disk are not stored. Prometheus is the history, because it keeps each scrape.
  • monitoring.data.get. That reads RRD and is for a chart a person opened. Do not scrape it.