Architecture

ScaiLog is built from a handful of small components that pass structured log entries down one path: your application emits an entry, an optional per-host agent encrypts any PI immediately, and a central FastAPI server stores it in SQLite. This page walks that path, names each component, and states the design invariants that shape the whole system.

The components#

There are four runtime components plus one offline job. Each is a separate package so you install only what a given host needs.

The SDK deliberately imports no heavy dependency: a service that only emits logs does pip install scailog and gets from scailog import log, pi with nothing pulled in behind it. Running components (server, agent, cli, audit) each install their own extra.

How a log entry flows#

You call the SDK. It builds an emit frame — a small JSON object carrying the call_site_id, level, message, plain fields, and any pi markers — and writes it as one NDJSON line to the agent's local socket. From there:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

your service                 scailog-agent (per host)              scailog-server
  ------------                 ----------------------                --------------
  log.info(...)                receive emit frame
    |  pi(value, type)   --->  encrypt PI *immediately*  ----.
    |  emit frame (NDJSON)     (subject DEK, AES-256-GCM)     |
    |                          cache to agent.db (SQLite)     |
    |                          pi_fields = ciphertext         |
    |                          message stays plaintext        |
    |                                    |                     |
    |                          batch (zstd NDJSON) -----------> POST /v1/ingest
    |                                                          verify checksum
    |                                                          store in scailog.db
    |                                                          FTS5 indexes message
    |                                                          + plaintext fields only

The crucial property is where encryption happens. In the agent topology the agent runs the ingest pipeline (process_emit) the moment it receives an entry, so PI is already ciphertext before it is ever written to the agent's on-disk cache. The server then stores the already-processed entry. PI plaintext never lands on disk anywhere.

The transformation applied at that point is the same whether the agent or the server runs it: for each pi marker, the policy engine resolves an action (encrypt, redact, drop, or keep), and encrypted fields become an opaque pi_fields blob with a pi_meta record describing what happened. See encryption & cryptographic erasure for the key hierarchy.

Panic mode#

If the agent cannot reach the server and has no encryption key in memory, it enters panic mode: rather than cache plaintext PI, it replaces each encrypt-action PI value with a [PANIC_DROPPED:<pi_type>] marker. The entry is still recorded; the PI is dropped rather than leaked.

Direct SDK to server#

The agent is optional. An SDK can also emit straight to the server, in which case the server runs the same process_emit pipeline at ingest. The topology is a deployment choice, not a change to the wire format or the stored shape.

Standalone mode#

The smallest supported deployment is one binary and one SQLite file. Running scailog-server --standalone embeds the agent's encryption path in-process, so a single command gives you emit, PI encryption, storage, query, and erasure with no agent and no external services:

1
2
3

scailog-server --init --data-dir ./data          # prints the bootstrap key once
SCAILOG_KEYSTORE_PASSPHRASE=dev SCAILOG_STANDALONE=true \
  scailog-server --standalone --data-dir ./data --port 8080

Scaling up (multiple hosts with agents, replication, sharding) is a deployment decision layered on the same SQLite storage — not an architectural rewrite.

Design invariants#

Three invariants (from the spec) drive the shape above and are hard constraints, not preferences.

• P1 — zero Scai* runtime dependencies. ScaiLog is the foundational logger every other platform service logs into, so it cannot depend at runtime on any Scai* service; that would create an unbootstrappable cycle. Key management and auth are built in. ScaiKey (OIDC) and ScaiVault (key custody) appear only as optional plugins that must degrade to the built-in mechanisms.
• P2 — SQLite everywhere. One storage engine at both tiers — the agent cache and the central store are both SQLite with FTS5 and WAL. There is no second storage format.
• P6 — smallest deployment first. The one-binary, one-file standalone mode is a first-class supported deployment, not a toy.

The store's structure enforces a further guarantee: the FTS5 full-text index covers only message and the plaintext fields JSON — never the pi_fields ciphertext. Full-text search therefore cannot leak PI, and erasing a subject never requires rebuilding the index.