LLM API
Hosted model instances, access tokens, and usage. Command API v2.10 (the LLM module).
Overview
An LLM instance is a hosted model in one datacentre. Tokens are how clients call it.
Results are under results, next to code.
List and get include instances you own and instances a parent has cascaded to you (customer_cascade).
owned is false when you can use the instance but do not own it; stdname is blank in that case.
Call llm.reference.getall before create. It has no parameters and returns the backends, versions, models, and the backend_env keys each model requires.
Instances
| Command | Parameters |
|---|---|
llm.instance.create | name, dc_id, backend, backend_version, model. Optional customer_id (default the caller), customer_cascade (boolean, default false), backend_args (array of strings), backend_env (object of strings; include every required key from the reference), enable_for_integrated_services (boolean). clone_llm_id copies an instance you own and ignores every other field. A clone does not copy tokens, the server binding, or the load balancer. |
llm.instance.update | Replaces the instance. Required: id, customer_id, name, backend, backend_version, model. Same optional fields as create except dc_id, which cannot be sent. |
llm.instance.get | id. Returns the instance, including decrypted backend_env, plus llmservers (llmserver_id, operationalstatus, comment). |
llm.instance.getall | No parameters. Returns instances you own or that are cascaded to you. |
llm.instance.getintegrated | Optional customer_id. Returns results.items and results.count. Each item is an instance with integrated services enabled, plus endpoint (https://<hostname>/v1). |
llm.instance.delete | id. Soft-deletes the instance. |
{
"cmd": "llm.instance.create",
"name": "support-model",
"dc_id": 4,
"backend": "vllm",
"backend_version": "0.6.0",
"model": "meta-llama/Llama-3.1-8B-Instruct",
"backend_args": [],
"backend_env": {"HF_TOKEN": "..."}
}Create and update return the instance under results, including a generated hostname on create. Backend, version, and model must be a combination from llm.reference.getall.
Tokens
| Command | Parameters |
|---|---|
llm.token.create | name, llm_id. Optional customer_id (defaults to the instance owner). You must own the instance, or it must be cascaded to you. Returns results.token once. Store it then. Later calls only show a prefix. |
llm.token.getall | Optional llm_id. Each row has id, llm_id, customer_id, name, token_prefix, date_created, date_last_used. |
llm.token.delete | id. Sets the token status to DELETED. |
Usage
llm.usage.getall needs exactly one of llm_id or token_id, plus date_from_utc and date_to_utc (unix timestamps).
results is hourly buckets: timestamp, request_count, prompt_tokens, completion_tokens.