Agent + server

The distributed shape puts a central scailog-server at the middle and a scailog-agent on every host that emits logs. The SDK writes to its local agent over a Unix socket; the agent encrypts PI immediately, caches to a local SQLite file, and ships batches to the server over HTTP. This is the production shape for a fleet — the agent keeps working, and keeps PI encrypted, even while the server is briefly unreachable.

Install the two components#

The server and agent are separate extras of the same package. Install each where it runs:

1
2
3
4
5

# On the central host
pip install 'scailog[server]'

# On every host that emits logs
pip install 'scailog[agent]'

Services that only emit logs need nothing but the base SDK (pip install scailog); they talk to the local agent socket.

Run the central server#

Initialize the data directory once to mint the bootstrap key, then run the server bound to an address your agents can reach:

1
2
3
4

scailog-server --init --data-dir ./data     # prints the bootstrap key once

SCAILOG_KEYSTORE_PASSPHRASE=... \
  scailog-server --data-dir ./data --host 0.0.0.0 --port 8080

Do not pass --standalone here — in the distributed shape the agents open the sockets, not the server. Use the bootstrap key to mint a log_writer API key per agent (see Authentication & API keys).

Run a per-host agent#

Each agent needs the server URL, an agent API key, the tenant it serves, and a socket path the local SDKs will connect to. The key can come from --key or the SCAILOG_AGENT_KEY environment variable.

1
2
3
4
5
6

SCAILOG_AGENT_KEY=slk_agent_... \
  scailog-agent \
    --server http://central-host:8080 \
    --tenant default \
    --socket /run/scailog/agent.sock \
    --cache ./agent.db

Flags accepted by scailog-agent (each also has an environment default):

What the agent does with a log#

The agent is where PI is protected at the earliest possible moment. For each emit frame arriving on the socket, it:

1. Validates the entry the same way the server would — rejecting reserved field names (RESERVED_FIELD), PI markers without a subject (SUBJECT_REQUIRED), and unknown or malformed PI types (PI_TYPE_UNKNOWN) — so a mistyped marker can't slip through the agent path.
2. Applies the policy engine to each PI field to pick an action — encrypt, redact, drop, or keep.
3. Encrypts immediately. For encrypt fields it fetches the per-subject DEK (wrapped) and encrypts the value in memory, so only ciphertext is ever written to the local cache. redact replaces the value with [REDACTED:<pi_type>]; drop discards it.
4. Caches the processed entry in the local SQLite file and returns an ack to the SDK.

Enrollment fetches the tenant KEK#

On startup the agent enrolls: it calls POST /v1/agent/enroll with its API key and receives the unwrapped tenant KEK into memory. When it first sees a new subject it calls POST /v1/agent/dek to fetch that subject's wrapped DEK, which it unwraps in memory. An erased subject's DEK is gone, so the server returns HTTP 410 and the agent has no key to encrypt with.

Panic mode when the server is unreachable#

If the agent cannot enroll or cannot fetch a subject's DEK — the server is down, or the subject was erased — it enters panic mode rather than write plaintext PI to disk. It refuses to cache the plaintext value, replaces the field with a [PANIC_DROPPED:<pi_type>] marker in the entry metadata, increments a panic_drops counter, and emits a non-silent warning under the reserved call site SCAILOG-AGENT-PANIC-DROP-001. The drop is observable, never silent, and no plaintext PI is ever persisted. (The dev-only --fail-open flag disables this and caches the plaintext instead — never use it in production; the agent prints a loud warning when it starts with fail-open on.)

Batch shipping with dead-lettering#

A background shipper drains the cache to POST /v1/ingest as zstd-compressed NDJSON batches with idempotency keys. It retries transient failures (5xx, network, 429) and auth failures (401/403) with exponential backoff and jitter (1s up to a 5-minute cap), so nothing is lost while you fix a cause. A poison batch — one the server will never accept (400/413/422) — is bisected to isolate the single offending entry, which is dead-lettered so the queue keeps advancing instead of head-of-line blocking forever. After a restart the shipper simply re-batches unshipped rows.

How the SDK reaches the agent#

Your service uses the SDK exactly as in standalone mode — it just points at the local agent's socket. Set SCAILOG_SOCKET (default /run/scailog/agent.sock) to the path the agent is listening on. The SDK queues entries in-process and sends newline-delimited JSON over the socket; if the socket is unavailable or the queue is full, it falls back to stderr in the same shape with PI values redacted, so a down agent degrades safely instead of leaking or silently dropping. See SDKs.