LLM API

Hosted model instances, access tokens, and usage. Command API v2.10 (the LLM module).

Overview

An LLM instance is a hosted model in one datacentre. Tokens are how clients call it. Results are under results, next to code. List and get include instances you own and instances a parent has cascaded to you (customer_cascade). owned is false when you can use the instance but do not own it; stdname is blank in that case. Call llm.reference.getall before create. It has no parameters and returns the backends, versions, models, and the backend_env keys each model requires.

Instances

CommandParameters
llm.instance.createname, dc_id, backend, backend_version, model. Optional customer_id (default the caller), customer_cascade (boolean, default false), backend_args (array of strings), backend_env (object of strings; include every required key from the reference), enable_for_integrated_services (boolean). clone_llm_id copies an instance you own and ignores every other field. A clone does not copy tokens, the server binding, or the load balancer.
llm.instance.updateReplaces the instance. Required: id, customer_id, name, backend, backend_version, model. Same optional fields as create except dc_id, which cannot be sent.
llm.instance.getid. Returns the instance, including decrypted backend_env, plus llmservers (llmserver_id, operationalstatus, comment).
llm.instance.getallNo parameters. Returns instances you own or that are cascaded to you.
llm.instance.getintegratedOptional customer_id. Returns results.items and results.count. Each item is an instance with integrated services enabled, plus endpoint (https://<hostname>/v1).
llm.instance.deleteid. Soft-deletes the instance.
{
  "cmd": "llm.instance.create",
  "name": "support-model",
  "dc_id": 4,
  "backend": "vllm",
  "backend_version": "0.6.0",
  "model": "meta-llama/Llama-3.1-8B-Instruct",
  "backend_args": [],
  "backend_env": {"HF_TOKEN": "..."}
}

Create and update return the instance under results, including a generated hostname on create. Backend, version, and model must be a combination from llm.reference.getall.

Tokens

CommandParameters
llm.token.createname, llm_id. Optional customer_id (defaults to the instance owner). You must own the instance, or it must be cascaded to you. Returns results.token once. Store it then. Later calls only show a prefix.
llm.token.getallOptional llm_id. Each row has id, llm_id, customer_id, name, token_prefix, date_created, date_last_used.
llm.token.deleteid. Sets the token status to DELETED.

Usage

llm.usage.getall needs exactly one of llm_id or token_id, plus date_from_utc and date_to_utc (unix timestamps). results is hourly buckets: timestamp, request_count, prompt_tokens, completion_tokens.