---
title: Agent + server
path: deployment/agent-and-server
status: published
---

# Agent + server

The distributed shape puts a central `scailog-server` at the middle and a
`scailog-agent` on every host that emits logs. The SDK writes to its local
agent over a Unix socket; the agent encrypts PI immediately, caches to a local
SQLite file, and ships batches to the server over HTTP. This is the production
shape for a fleet — the agent keeps working, and keeps PI encrypted, even while
the server is briefly unreachable.

## Install the two components

The server and agent are separate extras of the same package. Install each where
it runs:

```bash
# On the central host
pip install 'scailog[server]'

# On every host that emits logs
pip install 'scailog[agent]'
```

Services that only emit logs need nothing but the base SDK (`pip install
scailog`); they talk to the local agent socket.

## Run the central server

Initialize the data directory once to mint the bootstrap key, then run the
server bound to an address your agents can reach:

```bash
scailog-server --init --data-dir ./data     # prints the bootstrap key once

SCAILOG_KEYSTORE_PASSPHRASE=... \
  scailog-server --data-dir ./data --host 0.0.0.0 --port 8080
```

Do **not** pass `--standalone` here — in the distributed shape the agents open
the sockets, not the server. Use the bootstrap key to mint a `log_writer` API key
per agent (see [Authentication & API keys](/docs/scailog/deployment/auth-and-keys)).

## Run a per-host agent

Each agent needs the server URL, an agent API key, the tenant it serves, and a
socket path the local SDKs will connect to. The key can come from `--key` or the
`SCAILOG_AGENT_KEY` environment variable.

```bash
SCAILOG_AGENT_KEY=slk_agent_... \
  scailog-agent \
    --server http://central-host:8080 \
    --tenant default \
    --socket /run/scailog/agent.sock \
    --cache ./agent.db
```

Flags accepted by `scailog-agent` (each also has an environment default):

| Flag | Env | Default | Meaning |
| --- | --- | --- | --- |
| `--socket` | `SCAILOG_SOCKET` | `/run/scailog/agent.sock` | Local Unix socket the SDK connects to. |
| `--cache` | `SCAILOG_AGENT_CACHE` | `./agent.db` | Local SQLite cache for encrypted, unshipped entries. |
| `--server` | `SCAILOG_SERVER` | `http://127.0.0.1:8080` | Central server URL. |
| `--key` | `SCAILOG_AGENT_KEY` | — | Agent API key (bearer). |
| `--tenant` | `SCAILOG_TENANT` | `default` | Tenant this agent serves. |
| `--host` | `HOSTNAME` | `localhost` | Host label stamped on entries. |
| `--policy-file` | — | built-in default | Local policy YAML for agent-side handling. |
| `--fail-open` | — | off | **Dev only.** Cache plaintext PI if no key is available (see below). |

## What the agent does with a log

The agent is where PI is protected at the earliest possible moment. For each
`emit` frame arriving on the socket, it:

1. **Validates the entry** the same way the server would — rejecting reserved
   field names (`RESERVED_FIELD`), PI markers without a subject
   (`SUBJECT_REQUIRED`), and unknown or malformed PI types (`PI_TYPE_UNKNOWN`) —
   so a mistyped marker can't slip through the agent path.
2. **Applies the policy engine** to each PI field to pick an action —
   `encrypt`, `redact`, `drop`, or `keep`.
3. **Encrypts immediately.** For `encrypt` fields it fetches the per-subject DEK
   (wrapped) and encrypts the value in memory, so only ciphertext is ever written
   to the local cache. `redact` replaces the value with `[REDACTED:<pi_type>]`;
   `drop` discards it.
4. **Caches** the processed entry in the local SQLite file and returns an ack to
   the SDK.

### Enrollment fetches the tenant KEK

On startup the agent enrolls: it calls `POST /v1/agent/enroll` with its API key
and receives the unwrapped tenant KEK into memory. When it first sees a new
subject it calls `POST /v1/agent/dek` to fetch that subject's wrapped DEK, which
it unwraps in memory. An erased subject's DEK is gone, so the server returns
HTTP 410 and the agent has no key to encrypt with.

### Panic mode when the server is unreachable

If the agent cannot enroll or cannot fetch a subject's DEK — the server is down,
or the subject was erased — it enters panic mode rather than write plaintext PI
to disk. It **refuses to cache the plaintext value**, replaces the field with a
`[PANIC_DROPPED:<pi_type>]` marker in the entry metadata, increments a
`panic_drops` counter, and emits a non-silent warning under the reserved call
site `SCAILOG-AGENT-PANIC-DROP-001`. The drop is observable, never silent, and no
plaintext PI is ever persisted. (The dev-only `--fail-open` flag disables this and
caches the plaintext instead — never use it in production; the agent prints a
loud warning when it starts with fail-open on.)

### Batch shipping with dead-lettering

A background shipper drains the cache to `POST /v1/ingest` as zstd-compressed
NDJSON batches with idempotency keys. It retries transient failures (5xx,
network, 429) and auth failures (401/403) with exponential backoff and jitter (1s
up to a 5-minute cap), so nothing is lost while you fix a cause. A **poison
batch** — one the server will never accept (400/413/422) — is bisected to isolate
the single offending entry, which is **dead-lettered** so the queue keeps
advancing instead of head-of-line blocking forever. After a restart the shipper
simply re-batches unshipped rows.

## How the SDK reaches the agent

Your service uses the SDK exactly as in standalone mode — it just points at the
local agent's socket. Set `SCAILOG_SOCKET` (default `/run/scailog/agent.sock`) to
the path the agent is listening on. The SDK queues entries in-process and sends
newline-delimited JSON over the socket; if the socket is unavailable or the queue
is full, it falls back to stderr in the same shape with PI **values redacted**,
so a down agent degrades safely instead of leaking or silently dropping. See
[SDKs](/docs/scailog/sdk).