Prometheus
Scrape VM host capacity into Prometheus. One job, one target, every 5 minutes.
Scrape job
Add one job. The target is the API host. The path is a separate setting and must start with /.
The resource name is devices (plural).
scrape_configs:
- job_name: devicevmhost
scheme: https
metrics_path: /api/rest/v2.10/devices/vmhost/prometheus
scrape_interval: 5m
scrape_timeout: 30s
basic_auth:
username: API_UUID
password_file: /etc/prometheus/rackcorp_api_secret
static_configs:
- targets:
- api.rackcorp.net
password_file is the file that contains the API secret and nothing else (no newline is required; a trailing newline is fine).
Mode 0640, owner root, group prometheus.
password: with the secret inline also works. Keep that file out of git.
| Setting | Value |
|---|---|
scheme |
https |
metrics_path |
/api/rest/v2.10/devices/vmhost/prometheus |
targets |
api.rackcorp.net only. Do not put the path or a URL scheme in the target. |
scrape_interval |
5m. Put it on this job even when the global interval is 15s or 1m. |
scrape_timeout |
30s. The first scrape of a 5-minute window runs the query. Later scrapes in that window read a cache. |
| TLS | Public certificate on api.rackcorp.net. Leave insecure_skip_verify unset. |
Prometheus Operator
On kube-prometheus-stack, use a ScrapeConfig. That CRD shipped in Prometheus Operator 0.65. The secret keys are the API UUID and the API secret.
apiVersion: v1
kind: Secret
metadata:
name: rackcorp-api
namespace: monitoring
type: Opaque
stringData:
username: API_UUID
password: API_SECRET
---
apiVersion: monitoring.coreos.com/v1alpha1
kind: ScrapeConfig
metadata:
name: rackcorp-devicevmhost
namespace: monitoring
labels:
release: kube-prometheus-stack
spec:
staticConfigs:
- targets:
- api.rackcorp.net
scheme: HTTPS
metricsPath: /api/rest/v2.10/devices/vmhost/prometheus
scrapeInterval: 5m
scrapeTimeout: 30s
basicAuth:
username:
name: rackcorp-api
key: username
password:
name: rackcorp-api
key: password
The release label has to match the label your Prometheus selects on ScrapeConfig objects. kube-prometheus-stack uses the Helm release name.
Before reload
promtool check config /etc/prometheus/prometheus.yml curl -fsS -X POST http://127.0.0.1:9090/-/reload
/-/reload needs --web.enable-lifecycle. Otherwise restart the Prometheus process after promtool check config succeeds.
API key
HTTP Basic username is the API UUID. Password is the API secret. Use an API key, not a JWT. A JWT expires and Prometheus cannot refresh it, so the target goes down on a timer.
The key needs device access that includes device.vmhost.prometheus.getall, device.vmhost.availability.getall, or device.*.
The series are the VM hosts in that key's customer tree. One key is one job.
A second customer is a second job with its own key and its own job name.
Check the scrape
Confirm the body before Prometheus polls it. A good response is text/plain; version=0.0.4 and the first lines are # HELP / # TYPE.
curl -fsS -D - -o /tmp/rackcorp.prom \ -u "$API_UUID:$API_SECRET" \ https://api.rackcorp.net/api/rest/v2.10/devices/vmhost/prometheus promtool check metrics < /tmp/rackcorp.prom
X-Rackcorp-Scrape-Cache: miss means this response ran the query and was stored.
hit means Prometheus is being served the body from the last 5 minutes.
Cache-Control tells caches not to share it. Prometheus itself always performs the GET.
A JSON body means the path is wrong or the key is not allowed to call this command.
# authentication required means the key did not resolve to a customer.
Fix that before looking at Grafana. In Prometheus, the target should be UP under Status, Targets, job devicevmhost.
One sample per host
Each numeric series is emitted once per device_id. A line looks like this:
rackcorp_device_vm_total{device_id="24657"} 80
80 on rackcorp_device_vm_total is the planning cap, not the number of running guests. Running guests are rackcorp_device_vm_used.
Prometheus rejects a scrape that contains the same metric name and label set twice
(collected metric was collected before with the same name and label values).
The endpoint keeps the first row for a device_id and drops a later row, so that line appears once in the body.
Confirm with:
awk '!/^#/ && NF {print}' /tmp/rackcorp.prom | sort | uniq -dNo output means the body has no repeated sample.
If a graph still shows the same host twice, the second series was scraped by another job. The job and instance labels differ:
count by (device_id, job, instance) (rackcorp_device_vm_total) count by (device_id) (rackcorp_device_vm_total) > 1
A single job named devicevmhost and a single instance api.rackcorp.net:443 is the healthy result.
The second query returns nothing. A row there means two jobs, the same target listed under two job names, or a second Prometheus remote-writing into this TSDB.
Remove the extra job. Leave honor_labels at its default (false). These metrics do not carry a job or instance label of their own.
Interval
Hosts report about once a minute. The API reuses the rendered body for 300 seconds per customer.
A scrape faster than 5 minutes does not return newer numbers and does not run the query again.
Set scrape_interval: 5m on this job.
rackcorp_exporter_scrape_duration_seconds is the time taken to render that body.
On a cache hit the value is the original render, so it stays flat until the next miss.
rackcorp_exporter_build_info is always 1. Its version label is the exporter build.
Metrics
Every host series has a device_id label. Names and datacentre live only on rackcorp_device_info, which is always 1.
Join it when a legend needs a hostname. Status gauges use 0 OK, 1 WARNING, 2 CRITICAL or ERROR, -1 for anything else.
| Metric | Labels | Meaning |
|---|---|---|
rackcorp_device_info |
device_id, device_name, dc_id, dc_name, dc_short, svc_info, svc_description, vmh_version |
Always 1. The labels are the host name and location. |
rackcorp_device_cpu_total_cores |
device_id |
Logical CPU count. |
rackcorp_device_cpu_free_cores |
device_id |
Idle cores, from the 5-minute load average. |
rackcorp_device_cpu_used_cores |
device_id |
Busy cores. |
rackcorp_device_cpu_utilization_ratio |
device_id |
used / total, from 0 to 1. Idle ratio is free / total. |
rackcorp_device_cpu_status_code |
device_id |
WARNING below 30% idle, CRITICAL below 20% idle. |
rackcorp_device_memory_total_bytes |
device_id |
Reported RAM. |
rackcorp_device_memory_used_bytes |
device_id |
Reported RAM in use. |
rackcorp_device_memory_free_bytes |
device_id |
Headroom after projected allocations and a 32 GB reserve, floored at 0. This is not total - used. |
rackcorp_device_memory_status_code |
device_id |
Follows swap oversubscribe when swap is known: OK at or under 10% of RAM, WARNING above 10%, CRITICAL above 40%. |
rackcorp_device_storage_total_bytes |
device_id |
Host data volume size. |
rackcorp_device_storage_used_bytes |
device_id |
Host data volume used. |
rackcorp_device_storage_free_bytes |
device_id |
Reported free space plus projected allocation. Returned as 0 when that sum is under 64 GB. |
rackcorp_device_storage_status_code |
device_id |
WARNING under 500 GB free, CRITICAL under 100 GB free. |
rackcorp_device_raid_healthy |
device_id |
1 healthy, 0 not. |
rackcorp_device_traffic_capacity_bits_per_second |
device_id, direction |
Link speed. direction is inbound or outbound. |
rackcorp_device_traffic_used_bits_per_second |
device_id, direction |
Average bits/second since the previous sample (about 60 seconds). |
rackcorp_device_traffic_free_bits_per_second |
device_id, direction |
Capacity minus used, floored at 0. |
rackcorp_device_traffic_status_code |
device_id, direction |
WARNING under 200 Mbit/s free, CRITICAL under 100 Mbit/s free. |
rackcorp_device_vm_used |
device_id |
Running guests. This is the VM count to graph. |
rackcorp_device_vm_total |
device_id |
Planning soft cap of 80. Not a scheduler limit and not the running count. |
rackcorp_device_vm_free |
device_id |
80 - used. |
rackcorp_device_vm_status_code |
device_id |
WARNING at 70 running guests, CRITICAL at 80. |
rackcorp_device_out_of_date |
device_id |
1 when the host has not checked in for 20 minutes. Leave that host out of capacity decisions until it returns to 0. |
rackcorp_device_incident |
device_id |
1 when the host row is flagged with an incident. |
Divide bytes by 1024^3 for GiB. Divide bits/second by 1000^2 for Mbit/s.
Field-level notes that are easy to code wrong are in vmhost-capacity.md.
PromQL
Running guests, with the hostname and datacentre on the legend:
rackcorp_device_vm_used * on (device_id) group_left(device_name, dc_short) rackcorp_device_info
Idle CPU ratio. Drop hosts that have stopped checking in:
(rackcorp_device_cpu_free_cores / rackcorp_device_cpu_total_cores) and on (device_id) (rackcorp_device_out_of_date == 0)
Free disk in GiB, same filter:
(rackcorp_device_storage_free_bytes / 1024 / 1024 / 1024) and on (device_id) (rackcorp_device_out_of_date == 0)
Outbound Mbit/s:
rackcorp_device_traffic_used_bits_per_second{direction="outbound"} / 1000 / 1000Alerts
Page on a host that stays silent, on disk, and on the VM planning cap.
for: 15m is longer than one scrape so a single slow poll does not fire.
groups:
- name: rackcorp-vmhost
rules:
- alert: RackcorpVmhostStale
expr: |
(rackcorp_device_out_of_date == 1)
* on (device_id) group_left(device_name, dc_short)
rackcorp_device_info
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.device_name }} in {{ $labels.dc_short }} has not checked in"
- alert: RackcorpVmhostDiskCritical
expr: |
(rackcorp_device_storage_status_code == 2)
* on (device_id) group_left(device_name, dc_short)
(rackcorp_device_info and on (device_id) (rackcorp_device_out_of_date == 0))
for: 15m
labels:
severity: critical
annotations:
summary: "{{ $labels.device_name }} has under 100 GB free"
- alert: RackcorpVmhostVmCap
expr: |
(rackcorp_device_vm_status_code == 2)
* on (device_id) group_left(device_name, dc_short)
rackcorp_device_info
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.device_name }} is at the 80 VM planning cap"
Check the rules the same way as the scrape config: promtool check rules rackcorp-vmhost.yml.
What this endpoint is not
- Per-VM CPU, memory, or disk. Those are VMs on one host and latest guest metrics, on demand.
- History. Host free CPU, RAM, and disk are not stored. Prometheus is the history, because it keeps each scrape.
monitoring.data.get. That reads RRD and is for a chart a person opened. Do not scrape it.