Chat Completions

The chat completions API is ScaiGrid's flagship inference endpoint. Send a conversation, get an assistant response back. Supports streaming, tool calls, multimodal input (text + images + audio), and all the parameters you expect.

Endpoint: POST /v1/inference/chat

Basic request#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

curl -X POST https://scaigrid.scailabs.ai/v1/inference/chat \
  -H "Authorization: Bearer $SCAIGRID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "scailabs/poolnoodle-omni",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is the capital of Switzerland?"}
    ],
    "max_tokens": 50
  }'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import httpx, os

resp = httpx.post(
    "https://scaigrid.scailabs.ai/v1/inference/chat",
    headers={"Authorization": f"Bearer {os.environ['SCAIGRID_API_KEY']}"},
    json={
        "model": "scailabs/poolnoodle-omni",
        "messages": [
            {"role": "system", "content": "You are a concise assistant."},
            {"role": "user", "content": "What is the capital of Switzerland?"},
        ],
        "max_tokens": 50,
    },
)
data = resp.json()["data"]
print(data["choices"][0]["message"]["content"])

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

const resp = await fetch("https://scaigrid.scailabs.ai/v1/inference/chat", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCAIGRID_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "scailabs/poolnoodle-omni",
    messages: [
      { role: "system", content: "You are a concise assistant." },
      { role: "user", content: "What is the capital of Switzerland?" },
    ],
    max_tokens: 50,
  }),
});
const { data } = await resp.json();
console.log(data.choices[0].message.content);

Request parameters#

Messages#

Each message has a role and content:

• system — behavioral instructions. Usually the first message.
• user — human or upstream-system input.
• assistant — prior assistant responses (for multi-turn).
• tool — result of a tool call. Requires tool_call_id.

Multi-turn conversations#

1
2
3
4
5
6

messages = [
    {"role": "system", "content": "You translate to French."},
    {"role": "user", "content": "Hello"},
    {"role": "assistant", "content": "Bonjour"},
    {"role": "user", "content": "Good night"},
]

The full history is sent every time — ScaiGrid is stateless. For long-running conversations, use Sessions to store history server-side.

Multimodal content#

For vision or audio models, content can be an array of typed parts:

1
2
3
4
5
6
7

{
  "role": "user",
  "content": [
    {"type": "text", "text": "What's in this image?"},
    {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}
  ]
}

Supported part types:

• text — plain text
• image_url — {"url": "...", "detail": "auto" | "low" | "high"}
• image_base64 — {"data": "<base64>", "media_type": "image/png"}
• audio_url — for audio input on models like GPT-4o audio
• audio_base64 — {"data": "<base64>", "media_type": "audio/wav"}

ScaiGrid rewrites base64 images to proxy URLs on the fly, so you can send them inline without bloating your request.

Streaming#

Set "stream": true to receive tokens as they arrive. The response is Server-Sent Events (Content-Type: text/event-stream).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

import httpx, json, os

with httpx.stream(
    "POST",
    "https://scaigrid.scailabs.ai/v1/inference/chat",
    headers={"Authorization": f"Bearer {os.environ['SCAIGRID_API_KEY']}"},
    json={
        "model": "scailabs/poolnoodle-omni",
        "messages": [{"role": "user", "content": "Tell me a story."}],
        "stream": True,
    },
    timeout=600,
) as r:
    for line in r.iter_lines():
        if line.startswith("data: "):
            payload = line[6:]
            if payload == "[DONE]":
                break
            chunk = json.loads(payload)
            print(chunk["choices"][0]["delta"].get("content", ""), end="", flush=True)
        elif line.startswith("event: error"):
            # Next data: line carries the error payload
            pass

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

const resp = await fetch("https://scaigrid.scailabs.ai/v1/inference/chat", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCAIGRID_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "scailabs/poolnoodle-omni",
    messages: [{ role: "user", content: "Tell me a story." }],
    stream: true,
  }),
});

const reader = resp.body!.getReader();
const decoder = new TextDecoder();
let buf = "";
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buf += decoder.decode(value, { stream: true });
  const lines = buf.split("\n");
  buf = lines.pop()!;
  for (const line of lines) {
    if (!line.startsWith("data: ")) continue;
    const payload = line.slice(6);
    if (payload === "[DONE]") return;
    const chunk = JSON.parse(payload);
    process.stdout.write(chunk.choices[0].delta.content || "");
  }
}

SSE event types#

• data: {...} — a chat completion chunk. choices[0].delta.content accumulates. Reasoning-mode requests may also carry choices[0].delta.reasoning_content on chunks before the visible answer starts; see Reasoning mode.
• data: [DONE] — stream end. Close your reader.
• event: error\ndata: {...} — an error occurred mid-stream. Payload has {code, message}.

Any stream always ends with data: [DONE] whether it completed normally or errored.

Tool calls#

Declare tools the model can invoke:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                },
                "required": ["location"],
            },
        },
    },
]

resp = httpx.post(
    "https://scaigrid.scailabs.ai/v1/inference/chat",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "model": "scailabs/poolnoodle-omni",
        "messages": [{"role": "user", "content": "What's the weather in Zürich?"}],
        "tools": tools,
    },
).json()["data"]

msg = resp["choices"][0]["message"]
if msg.get("tool_calls"):
    tc = msg["tool_calls"][0]
    print(f"Tool: {tc['function']['name']}")
    print(f"Args: {tc['function']['arguments']}")  # JSON string

When the model wants to call a tool, the response has tool_calls instead of content. Execute the tool, add both the assistant turn and the tool result to your history, then call again:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

history = [
    {"role": "user", "content": "What's the weather in Zürich?"},
    msg,  # assistant turn with tool_calls
    {
        "role": "tool",
        "tool_call_id": msg["tool_calls"][0]["id"],
        "content": '{"temp": 18, "unit": "celsius"}',
    },
]

final = httpx.post(... json={"model": ..., "messages": history, "tools": tools}, ...)

The model's second call produces a plain text response using the tool result.

Reasoning mode#

Modern reasoning models — Claude Opus 4.x, GPT-5, o3, Gemini 2.5, DeepSeek-R1 — spend an internal chain-of-thought before answering. ScaiGrid exposes a single provider-neutral toggle so you can enable, budget, and surface that reasoning from any client.

Send thinking on any chat request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

{
  "model": "anthropic/claude-opus-4-8",
  "messages": [
    {"role": "user", "content": "Prove that √2 is irrational."}
  ],
  "thinking": {
    "enabled": true,
    "budget_tokens": 8192,
    "effort": "high",
    "include_thoughts": false
  }
}

Omit the thinking field entirely and behavior is identical to any pre-reasoning request.

Which models support it#

Query GET /v1/models and filter on capabilities. Any slug tagged with thinking accepts the toggle.

1
2
3
4
5
6
7
8
9

resp = httpx.get(
    "https://scaigrid.scailabs.ai/v1/models",
    headers={"Authorization": f"Bearer {API_KEY}"},
).json()

thinking_capable = [
    m for m in resp["data"]
    if "thinking" in (m.get("capabilities") or [])
]

At time of writing, 41+ slugs are tagged (all Gemini 2.5+ and 3.x variants, all GPT-5.x, o1/o3/o4-mini, Qwen QwQ, Qwen QVQ). Claude Opus 4.x / Sonnet 5 and self-hosted DeepSeek-R1 slugs pick up the tag as they're registered.

Sending thinking to a non-supporting model#

ScaiGrid doesn't reject the request — the field is silently ignored on providers that don't understand it. Two things happen instead:

1. usage.reasoning_tokens on the response is 0.
2. The response carries X-Scaigrid-Thinking-Unsupported: true as an HTTP header. Surface it in your UI as a soft warning if you want end-users to know their reasoning intent was dropped.

Budget vs effort#

Providers disagree on the shape. Send whichever you have; ScaiGrid derives the other:

When only budget_tokens is sent to an effort-based provider (OpenAI o-series), ScaiGrid maps it back:

The response#

Non-streaming responses gain a nested reasoning count on usage (present but 0 when thinking was off):

1
2
3
4
5
6
7
8

{
  "usage": {
    "prompt_tokens": 1240,
    "completion_tokens": 340,
    "total_tokens": 7790,
    "reasoning_tokens": 6210
  }
}

Reasoning tokens are billed at the same per-token price as completion tokens. A "deep-think" answer can spend 20k+ reasoning tokens; surface the split in your UI so users understand where the cost went.

When include_thoughts=true the response also carries an ordered list of the raw reasoning blocks:

1
2
3
4
5
6

{
  "choices": [{"message": {"role": "assistant", "content": "…"}, "finish_reason": "stop"}],
  "thinking": [
    {"text": "Let me consider…", "redacted": false, "signature": "…"}
  ]
}

redacted: true marks blocks the provider returned in opaque form (Anthropic occasionally does this for safety-adjacent chains). signature is populated for Anthropic-signed blocks and is required for tool-use continuity (see next section).

Streaming reasoning#

When you set stream: true and thinking.include_thoughts=true, reasoning chunks arrive on choices[0].delta.reasoning_content before the visible answer starts. Ordinary content deltas continue as usual:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

for line in r.iter_lines():
    if not line.startswith("data: "):
        continue
    payload = line[6:]
    if payload == "[DONE]":
        break
    chunk = json.loads(payload)
    delta = chunk["choices"][0]["delta"]
    if delta.get("reasoning_content"):
        # Show a "thinking…" indicator or stream to a hidden pane
        print(f"[think] {delta['reasoning_content']}", end="", flush=True)
    if delta.get("content"):
        print(delta["content"], end="", flush=True)

If include_thoughts is false (or omitted), reasoning still happens on the provider side but no reasoning_content chunks are sent — the visible content stream is identical to a non-thinking request.

Multi-turn tool-use (Anthropic only, opt-in)#

To continue a tool-use conversation from an assistant turn that included thinking, the raw thinking block MUST be echoed back verbatim on the next assistant-role message. Anthropic verifies the signature and errors otherwise.

Only relevant if: the model is anthropic/* AND your flow uses tool-use AND you want thinking on those turns.

Pattern:

1. Send the initial request with thinking.include_thoughts=true.
2. Save response.thinking alongside response.content in your conversation log.
3. On the next request, attach thinking_blocks to the assistant-role message when replaying history:

1
2
3
4
5
6

{
  "role": "assistant",
  "content": "Let me use the calculator…",
  "tool_calls": [{"id":"tc_1","type":"function","function":{"name":"calc","arguments":"…"}}],
  "thinking_blocks": [{"text": "…", "signature": "…"}]
}

ScaiGrid round-trips thinking_blocks to Anthropic verbatim. Non-Anthropic providers ignore the field, so this only matters when the flow can hit Claude.

Provider-specific notes#

• Anthropic requires temperature = 1.0 when thinking is enabled. ScaiGrid silently overrides any other value you sent, logging the override so it shows up in your request trace.
• Google Gemini 2.5+ bundles reasoning tokens inside completion_tokens in some responses. You'll see reasoning_tokens=0 there — nothing is wrong; the count is folded into completion.
• OpenAI o-series / GPT-5 doesn't stream reasoning content in real time (as of writing). include_thoughts=true produces reasoning blocks on the final response but no reasoning_content deltas mid-stream.
• Self-hosted reasoning models (via ScaiInfer, e.g. DeepSeek-R1) will support the same shape when deployed. If a self-hosted node hasn't wired reasoning yet, X-Scaigrid-Thinking-Unsupported: true fires and the request runs without reasoning.

Cost UX guidance#

• Default to off. Enabling thinking silently in a paid product is a support-ticket generator.
• Show the split. "Answer: 340 tokens · Reasoning: 6.2k tokens" makes the cost visible.
• Guardrail large budgets. If your product has tenant-level budget caps, warn before a big reasoning spend: "Reasoning could cost up to $X extra."

Response shape#

Non-streaming response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

{
  "status": "ok",
  "data": {
    "id": "chatcmpl-abc",
    "model": "scailabs/poolnoodle-omni",
    "created": 1713888000,
    "choices": [{
      "index": 0,
      "message": {"role": "assistant", "content": "Bern."},
      "finish_reason": "stop"
    }],
    "usage": {"prompt_tokens": 24, "completion_tokens": 3, "total_tokens": 27, "reasoning_tokens": 0},
    "_meta": {"request_id": "req_xyz", "latency_ms": 310}
  }
}

usage.reasoning_tokens is present on every response and is 0 unless the caller enabled thinking on a supporting model. See Reasoning mode.

finish_reason values:

• stop — model finished naturally
• length — hit max_tokens
• tool_calls — model wants to call a tool
• content_filter — upstream safety filter triggered

What's next#

• Embeddings — vectorize text for search.
• OpenAI Compatibility — drop-in replacement for OpenAI SDK.
• Sessions and Rooms — server-side conversation storage.
• Errors — handling failures.