---
audience: all ScaiLabs product teams
summary: "The repeatable path to adopt ScaiLog central logging, derived from the completed\
  \ ScaiKey reference implementation \u2014 including the traps not in ScaiLog's own\
  \ docs."
title: ScaiLog adoption playbook
path: tutorials/scailog-adoption-playbook
status: published
---

# ScaiLog adoption playbook

**For:** every ScaiLabs product team adopting central logging.
**Status:** ScaiKey is the completed reference implementation (100 call sites,
live and ingesting). This playbook is the distillation of doing it end to end —
including the traps that are **not** in ScaiLog's own docs, because we only found
them by shipping.
**Baseline versions:** ScaiLog **1.0.2** (SDK + agent, one wheel). Do not use
1.0.0 (agent flattens tenants) or 1.0.1 (`[agent]` extra missing PyYAML).

---

## 0. The one-paragraph model

ScaiLog is GDPR-native central logging. Your service emits structured entries
over a local Unix socket to a per-host **agent**; the agent encrypts each PI
field under a per-subject key, caches locally, and ships to the central server.
Every entry carries a developer-assigned **`call_site_id`** (so remediation is
"fix one call site") and a **tenant** (the customer the event concerns, or
`_global` for platform work). Erasing a person is destroying their key — no
distributed delete. PII in your logs becomes encrypted, attributable, and
erasable; the message stays plaintext-searchable and must therefore be PI-free.

---

## 1. Read this before you touch code — the model that bites

Four contracts the SDK enforces at emit, the server re-checks at ingest, and
`scailog-ci` checks at build. Violating any is a `ValueError` at emit (which,
inside a `try/except`, looks identical to the entry silently never being
written) or a server-side rejection.

1. **`call_site_id` must be a string LITERAL, well-formed, unique.**
   `^[A-Z0-9]+(-[A-Z0-9]+){2,}$` — ≥3 uppercase-hyphen segments. Reserved
   prefixes `SCAILOG-` / `AUTO-` are forbidden.
   **Trap we hit:** a *named constant* — even one defined in the same module —
   is rejected as `SITE_NON_CONSTANT`, and the run reports "0 named" rather than
   erroring. So a registry of `SITE = "KEY-..."` constants referenced at call
   sites makes every site **invisible to the gate**. Write the id inline; keep a
   registry module for review, cross-checked by a test (see §4).

2. **The message must be a constant.** No f-string, `%`, `.format()`.
   Interpolation is how PII reaches the plaintext, full-text-indexed `msg`
   column — outside the crypto-shredding envelope, so an Article 17 erasure will
   **not** remove it. Variable data goes in `fields`.

3. **Every `pi()` field requires `subject=`.** The subject is the erasure/DSAR
   anchor. `pi()` values must be scalars or flat lists.
   **Pattern worth internalising:** the PI-requires-subject rule pushes a log to
   the point where identity is known. Where a callback logs *before* resolving
   the local user id, **move the log below the resolution** rather than reaching
   for the external subject — that keeps one subject namespace and DSAR-by-user
   working.

4. **Field names must not collide with the envelope.** Reserved:
   `event_id, tenant, service, env, site, level, ts, msg, fields, pi, subject,
   trace_id, request_id`. Using one raises at emit.
   **Trap we hit:** `event_id` as a field name. Namespace yours
   (`scaikey_event_id`). Guard it statically (§4).

---

## 2. Prerequisites the platform provides (per service)

Do not start until these exist:

| Prerequisite | Notes |
|---|---|
| **SDK 1.0.2+** | `pip install "scailog[agent]==1.0.2"` (or the pinned wheel URL + `#sha256=`; no index carries it yet). One wheel = SDK **and** agent; the agent is a console script, not a separate artifact. |
| **A `log_writer` key** | `ingest`-only, `service_scope: ["<your service>"]`. See §7 for the tenant-scope tradeoff and why the key never goes near a transcript. |
| **An agent on each backend host** | §6. Frontend/static hosts get none. |
| **Your tenant naming decided** | §5. This is a real decision, not a default. |

---

## 3. Phasing — the order is not optional

We did it in three phases and the order is load-bearing:

- **Phase 1 — close the PI leaks first.** Find every `logger.*` that
  interpolates personal data and drop it (log the pseudonymous `usr_` id, or a
  non-identifying substitute like an email *domain*). Ship this alone; it is a
  security fix with no ScaiLog dependency.
- **Phase 2 — logging foundation.** Central logging config, request-correlation
  middleware, make `LOG_LEVEL` actually live.
  **Why the order:** turning on a root handler *un-silences* every INFO site
  that was previously discarded. If Phase 1 hasn't run, Phase 2 is itself a
  PI-leak-introducing change. Gate Phase 2 on the Phase-1 lint being green.
- **Phase 3 — migrate to ScaiLog.** Convert call sites to constant messages +
  structured fields + `pi()`/`subject`, bind tenants (§5), wire the CI gate.

**A single-choke-point assumption will burn you.** We assumed all audit rows
went through one writer; there were three. Grep for *every* path that writes
your audit/log stream before wiring anything that must cover all of them.

---

## 4. The code — call sites, registry, guards

**Emit** (Python; the other SDKs are wire-identical):
```python
from scailog import log
from scailog.constants import SERVICE_SUBJECT   # "_service"

log.info(
    "KEY-MAIL-SENT",                 # literal, never a constant reference
    "transactional email sent",      # constant message
    subject=SERVICE_SUBJECT,         # or the usr_ id where identity is known
    fields={"template": template, "to_domain": email_domain(to_email)},
)
```

**Levels:** `trace debug info warn error fatal`. There is **no `exception`
level and no traceback.** Convert `logger.exception(...)` to `log.error(...)`
with `error_type=type(e).__name__`; where a traceback genuinely aids triage
(e.g. the unhandled-exception handler), keep a stdlib `logger.exception`
alongside the ScaiLog entry.

**Never interpolate `str(exc)`** — many libraries' exceptions echo the offending
input (an email, a bind DN, a token). `error_type` is the safe field.

**PI type mapping** (drives encrypt/keep/drop at the policy engine):

| Value | `pi_type` | Default action |
|---|---|---|
| email, UPN, sAMAccountName, DN, CN, display/given/family name, SAML NameID, IdP `sub` | `direct_identifier` | encrypt |
| IP address, user-agent, cookie/device id | `online_identifier` | encrypt |
| `usr_*` `sess_*` `grp_*` `tnt_*` `app_*` ids | `pseudonymous` | **keep** — these are the join keys; encrypting them destroys the log's usefulness for no privacy gain |
| password, client_secret, any token, code_verifier, TOTP/backup code, private key | `credentials` | **drop** — never pass at all; the drop is a net, not a licence |
| free-text user content, SAML attribute values, `details` blobs | `freeform_user_content` | encrypt |

**Call-site registry.** Keep `obs/sites.py` listing every id (documentation +
grep target), but the values are **written inline at call sites** (§1.1). A
local test scans the source for the literals actually used and fails if one is
missing from the registry, or if a declared id is used nowhere.

**Two durable local guards** (they run in the ordinary suite — no SDK, no agent,
no network — which is exactly where `scailog-ci` *cannot* run, e.g. TS/Go/.NET
repos and every dev machine before install):

- `test_no_pi_in_logs.py` — AST-walks the package, fails if a bare reference
  whose name is a known PI/secret field is interpolated into any log call.
  Make it receiver-aware (`idp.name` is an org name, not a person; a bare
  `name` is flagged).
- `test_log_sites.py` — mirrors `scailog-ci`'s SITE_INVALID / SITE_DUPLICATE /
  SITE_RESERVED, **plus** reserved-field-name detection and registry-drift.

---

## 5. Tenancy — the piece that changed under us mid-programme

ScaiLog's **tenant is per-entry, mandatory, and is the slug of the tenant that
caused the event.** Reserved `_global` for platform work that belongs to no
tenant (bootstrap, workers, system failures). An unbound emit raises
`TENANT_REQUIRED` — there is no silent fallback.

**A GLOBAL/multi-tenant service does not pick one tenant.** It binds the tenant
per request/operation. In ScaiKey:

- **Request paths:** middleware binds the slug from the URL
  (`/api/v1/auth/tenants/{slug}/...`), else `_global`. One binding covers every
  site below it.
- **Workers / boot:** explicit `tenant=GLOBAL_TENANT`.
- **Flows that hold a `tnt_` id, not a slug** (LDAP sync, webhook delivery,
  event publish): resolve id → slug via a small cached lookup and bind it around
  the operation. **Never send the raw `tnt_` id** — ScaiLog has no tenant
  registry, so a bad string silently mints a phantom tenant.

Precedence is `tenant=` arg → bound context → Logger default. Leaving the Logger
default unset keeps any unbound path failing loud, which surfaces bugs; set it
to `_global` only if you'd rather never risk a `TENANT_REQUIRED` in a hot path.

**Erasure benefit, confirmed:** if your user ids are globally unique across your
tenants (ScaiKey's are — 43 users, 43 distinct ids), one
`POST /v1/erase {tenant, subject}` reaches a person regardless of tenant. If
they're not, a DSAR becomes a cross-tenant fan-out — decide your subject-id
scheme **before** wave 2, not after.

---

## 6. Deployment — the agent, and the trap that cost us a debugging session

**Install** (one wheel, isolated venv, per host):
```bash
useradd --system scailog
python3 -m venv /opt/scailog-agent/venv
/opt/scailog-agent/venv/bin/pip install "scailog[agent]==1.0.2"   # or wheel URL + #sha256=
```

**Agent systemd unit** — key points from ScaiKey's working unit
(`deploy/standalone/scailog-agent.service`):
- `User=scailog`, `RuntimeDirectory=scailog` (creates `/run/scailog` 0750),
  `StateDirectory=scailog-agent` (persistent cache — **not** `/run`, it must
  survive reboots), socket at `/run/scailog/agent.sock` (0660 scailog:scailog).
- **No `--tenant` flag.** The per-entry model means the agent must pass the
  frame's tenant through; a fixed `--tenant` would flatten everything to it
  (the 1.0.0 bug). 1.0.2 threads `frame["tenant"]` correctly — verify on your
  version with a two-tenant probe before trusting it.
- Agent is `Wants=`, **never `Requires=`**, of your service — a logging agent
  must not gate an IAM (or any) service. The stderr fallback exists for exactly
  the agent-down case, and it strips PI **values** (keeps field name + type), so
  a missing agent degrades safely.

**Your service's unit needs a drop-in:**
```ini
[Service]
SupplementaryGroups=scailog          # to reach the 0660 socket
ReadWritePaths=/run/scailog          # if you use ProtectSystem=strict
Environment=SCAILOG_SERVICE=<your service>
Environment=SCAILOG_ENV=production
Environment=SCAILOG_SOCKET=/run/scailog/agent.sock
```

> **THE TRAP.** `SCAILOG_*` must be **systemd `Environment=` lines, not in
> `.env`.** pydantic-settings reads `.env` but does **not** export into
> `os.environ`, which the SDK reads directly. Miss `SCAILOG_SERVICE` and the SDK
> emits `service="unknown"`; your `service`-scoped writer key then rejects
> **every** entry as `SERVICE_FORBIDDEN`. This looks exactly like "logging is
> broken" and the agent log is silent about it — the rejection is server-side.
> It cost us a full debug loop. Set `SCAILOG_SERVICE` first, verify with a
> direct `/v1/ingest` probe if in doubt.

Two more, briefly: `sudo -u <svc> test -w /run/scailog/agent.sock` is a **false
negative** (a fresh login doesn't inherit systemd's `SupplementaryGroups`;
check `/proc/<MainPID>/status` `Groups:` instead). And `/run/scailog` at 0750 is
unreadable to your own admin user — inspect the socket as root.

---

## 7. Keys — custody

- **Agent key:** `log_writer`, `ingest`-only, scoped to your service. It may
  need `tenant_scope: *` (multi-tenant services attribute per entry). Know the
  cost: `/v1/agent/enroll` returns the **unwrapped tenant KEK** to any
  `ingest`-capable key, so a `*`-scoped writer key can pull every tenant's KEK.
  Mitigate with an **`ip_allowlist`** pinned to the host and a `rate_limit`.
- **Never let the plaintext transit a transcript, ticket, or chat.** Mint it and
  paste it directly into the root shell at install
  (`SCAILOG_AGENT_KEY=... ./install...`), or into `/etc/scailog/agent.env`
  (0640 root:scailog). A key that has been pasted into a session is compromised
  and must be rotated — this happened to us and is a standing debt.
- **CI/manifest push:** a `log_provisioner` (`provision`-only) or `log_admin`
  key. Push from a controlled release job, not a PR runner, and note manifest
  push has historically defaulted to the wrong tenant — pass it explicitly or
  use a version that made the endpoint service-scoped.

---

## 8. The CI gate

Add to the lint job, **blocking from day one**:
```yaml
- run: pip install "scailog[agent]==1.0.2"
- run: scailog-ci check src
```
A repo with zero call sites exits 0, so adding it estate-wide now is free until
the first site is written. Blocking (not warn-then-block) is correct here: the
violations it catches — interpolated messages, PI without subject — are the
*unrecoverable* ones. A warn phase is a phase spent generating exactly the
records you can never take back.

**`scailog-ci` is Python-only** (it `ast`-parses `*.py`). TS/Go/.NET services
get the two local guards from §4 plus **server-side drift detection**: a nightly
`GET /v1/sites?tenant=...&auto_only=true` / `&pi_only=true` compared to the
pushed manifest flags any observed-but-undeclared or `AUTO-*` site, in every
language.

---

## 9. Verification — prove it, don't trust the 201

The discipline that caught two real bugs (the tenant flatten, the missing
PyYAML) is **install-and-run, then check the wire**, not trust a success code.

- **PI closure (Phase 1):** drive the identity flows, then
  `journalctl -u <svc> | grep -Ei '<email>|"(access|refresh|id)_token"'` must be
  empty. Keep the AST guard as the durable regression control.
- **Correlation (Phase 2):** one request with `X-Request-Id` must echo back,
  appear in the log, and land in the audit row — one assertion covers the chain.
  Then 40 concurrent distinct ids must map 1:1 (catches contextvar leakage).
- **Ingest (Phase 3):** capture `ingest_accepted` on `/v1/status`, drive N
  requests, confirm it climbs by N and `ingest_rejected` stays flat. Confirm
  **zero stderr-fallback lines** in your journal (proves the socket path is
  used). Confirm `panic_drops: 0`.
- **Attribution:** read the agent cache
  (`sqlite3 /var/lib/scailog-agent/agent.db 'SELECT tenant_id, call_site_id ...'`)
  — tenant-path entries under their slug, platform under `_global`, all
  `shipped`. (Your `ingest`-only key can't query the server; the cache is the
  authoritative local record.)
- **Erasure (the real acceptance test):** create a throwaway subject, generate
  activity, `POST /v1/erase`, confirm the PI no longer decrypts. Record any
  copy that erasure does **not** reach (see §11).

---

## 10. Sequencing across the estate

Tiers, from ScaiLog's zero-Scai-dependency invariant and each service's fan-in:

- **Tier 0 — ScaiLog itself.** Deployed. Must never acquire a runtime dependency
  on a service that logs into it (do **not** install its `oidc`/`vault`/`audit`
  extras on the server — they pull SDKs and create a bootstrap cycle).
- **Tier 1 — ScaiKey.** Done. Everything authenticates through it, so it was the
  right reference and second adopter.
- **Tier 2 — the rest (~39).** Order by PI density first, infrastructure last
  (its blast radius is the whole estate). Add the free `scailog-ci check` to
  every Python repo now, while it's still a no-op.

**Never** put an agent on a static frontend host — no server-side app logs.

---

## 11. Known limits — state these, don't discover them

- **Retention deletes whole journal/agent files, not entries.** A per-entry
  window is enforced going *forward*; a pre-existing backlog co-mingled with
  recent data ages out over the window, it can't be surgically purged.
- **Access logs are out of scope by design.** High-volume request logs (client
  IPs, `login_hint` emails) stay local with a real rotation policy — for ScaiKey,
  a 14-day journald cap on the backend matching the 14-day nginx logrotate on
  the proxy. Centralising them would swamp signal for little compliance gain.
  Just make sure the local rotation is *configured*, not aspirational.
- **Mirroring an existing plaintext store (e.g. an `audit_logs` table) into
  ScaiLog does not make the original erasable.** Dual-write improves DSAR
  *reporting* and adds an encrypted copy, but the plaintext original stays
  outside erasure's reach until you either make ScaiLog authoritative for that
  PI or encrypt the columns. Do not record "we mirror to ScaiLog" as closing an
  audit-PI finding.
- **Agent server-outage behaviour:** entries cache and ship on recovery, but PI
  for a subject whose key isn't already resident is **panic-dropped**
  permanently, and an agent restart during an outage loses all resident keys.
  Treat `panic_drops > 0` as a page; never auto-restart an agent mid-incident.

---

## Appendix — the ScaiKey reference

- 100 call sites, `scailog-ci` clean; `obs/{redaction,logging_config,context,
  sites,tenants}.py`; guards `test_no_pi_in_logs.py`, `test_log_sites.py`.
- Live and ingesting from `scaikey-be1`; per-tenant attribution verified on the
  wire.
- Findings raised upstream this programme:
  `2026-08-25-notice-scailog-agent-tenant-flattening.md` (the 1.0.0 flatten;
  fixed in 1.0.2). The RP-logout series is unrelated to ScaiLog but shows the
  same "verify against a live instance" discipline.