ScaiLog adoption playbook

For: every ScaiLabs product team adopting central logging. Status: ScaiKey is the completed reference implementation (100 call sites, live and ingesting). This playbook is the distillation of doing it end to end — including the traps that are not in ScaiLog's own docs, because we only found them by shipping. Baseline versions: ScaiLog 1.0.2 (SDK + agent, one wheel). Do not use 1.0.0 (agent flattens tenants) or 1.0.1 ([agent] extra missing PyYAML).

0. The one-paragraph model#

ScaiLog is GDPR-native central logging. Your service emits structured entries over a local Unix socket to a per-host agent; the agent encrypts each PI field under a per-subject key, caches locally, and ships to the central server. Every entry carries a developer-assigned call_site_id (so remediation is "fix one call site") and a tenant (the customer the event concerns, or _global for platform work). Erasing a person is destroying their key — no distributed delete. PII in your logs becomes encrypted, attributable, and erasable; the message stays plaintext-searchable and must therefore be PI-free.

1. Read this before you touch code — the model that bites#

Four contracts the SDK enforces at emit, the server re-checks at ingest, and scailog-ci checks at build. Violating any is a ValueError at emit (which, inside a try/except, looks identical to the entry silently never being written) or a server-side rejection.

1. call_site_id must be a string LITERAL, well-formed, unique. ^[A-Z0-9]+(-[A-Z0-9]+){2,}$ — ≥3 uppercase-hyphen segments. Reserved prefixes SCAILOG- / AUTO- are forbidden. Trap we hit: a named constant — even one defined in the same module — is rejected as SITE_NON_CONSTANT, and the run reports "0 named" rather than erroring. So a registry of SITE = "KEY-..." constants referenced at call sites makes every site invisible to the gate. Write the id inline; keep a registry module for review, cross-checked by a test (see §4).

2. The message must be a constant. No f-string, %, .format(). Interpolation is how PII reaches the plaintext, full-text-indexed msg column — outside the crypto-shredding envelope, so an Article 17 erasure will not remove it. Variable data goes in fields.

3. Every pi() field requires subject=. The subject is the erasure/DSAR anchor. pi() values must be scalars or flat lists. Pattern worth internalising: the PI-requires-subject rule pushes a log to the point where identity is known. Where a callback logs before resolving the local user id, move the log below the resolution rather than reaching for the external subject — that keeps one subject namespace and DSAR-by-user working.

4. Field names must not collide with the envelope. Reserved: event_id, tenant, service, env, site, level, ts, msg, fields, pi, subject, trace_id, request_id. Using one raises at emit. Trap we hit: event_id as a field name. Namespace yours (scaikey_event_id). Guard it statically (§4).

2. Prerequisites the platform provides (per service)#

Do not start until these exist:

3. Phasing — the order is not optional#

We did it in three phases and the order is load-bearing:

• Phase 1 — close the PI leaks first. Find every logger.* that interpolates personal data and drop it (log the pseudonymous usr_ id, or a non-identifying substitute like an email domain). Ship this alone; it is a security fix with no ScaiLog dependency.
• Phase 2 — logging foundation. Central logging config, request-correlation middleware, make LOG_LEVEL actually live. Why the order: turning on a root handler un-silences every INFO site that was previously discarded. If Phase 1 hasn't run, Phase 2 is itself a PI-leak-introducing change. Gate Phase 2 on the Phase-1 lint being green.
• Phase 3 — migrate to ScaiLog. Convert call sites to constant messages + structured fields + pi()/subject, bind tenants (§5), wire the CI gate.

A single-choke-point assumption will burn you. We assumed all audit rows went through one writer; there were three. Grep for every path that writes your audit/log stream before wiring anything that must cover all of them.

4. The code — call sites, registry, guards#

Emit (Python; the other SDKs are wire-identical):

1
2
3
4
5
6
7
8
9

from scailog import log
from scailog.constants import SERVICE_SUBJECT   # "_service"

log.info(
    "KEY-MAIL-SENT",                 # literal, never a constant reference
    "transactional email sent",      # constant message
    subject=SERVICE_SUBJECT,         # or the usr_ id where identity is known
    fields={"template": template, "to_domain": email_domain(to_email)},
)

Levels: trace debug info warn error fatal. There is no exception level and no traceback. Convert logger.exception(...) to log.error(...) with error_type=type(e).__name__; where a traceback genuinely aids triage (e.g. the unhandled-exception handler), keep a stdlib logger.exception alongside the ScaiLog entry.

Never interpolate str(exc) — many libraries' exceptions echo the offending input (an email, a bind DN, a token). error_type is the safe field.

PI type mapping (drives encrypt/keep/drop at the policy engine):

Call-site registry. Keep obs/sites.py listing every id (documentation + grep target), but the values are written inline at call sites (§1.1). A local test scans the source for the literals actually used and fails if one is missing from the registry, or if a declared id is used nowhere.

Two durable local guards (they run in the ordinary suite — no SDK, no agent, no network — which is exactly where scailog-ci cannot run, e.g. TS/Go/.NET repos and every dev machine before install):

• test_no_pi_in_logs.py — AST-walks the package, fails if a bare reference whose name is a known PI/secret field is interpolated into any log call. Make it receiver-aware (idp.name is an org name, not a person; a bare name is flagged).
• test_log_sites.py — mirrors scailog-ci's SITE_INVALID / SITE_DUPLICATE / SITE_RESERVED, plus reserved-field-name detection and registry-drift.

5. Tenancy — the piece that changed under us mid-programme#

ScaiLog's tenant is per-entry, mandatory, and is the slug of the tenant that caused the event. Reserved _global for platform work that belongs to no tenant (bootstrap, workers, system failures). An unbound emit raises TENANT_REQUIRED — there is no silent fallback.

A GLOBAL/multi-tenant service does not pick one tenant. It binds the tenant per request/operation. In ScaiKey:

• Request paths: middleware binds the slug from the URL (/api/v1/auth/tenants/{slug}/...), else _global. One binding covers every site below it.
• Workers / boot: explicit tenant=GLOBAL_TENANT.
• Flows that hold a tnt_ id, not a slug (LDAP sync, webhook delivery, event publish): resolve id → slug via a small cached lookup and bind it around the operation. Never send the raw tnt_ id — ScaiLog has no tenant registry, so a bad string silently mints a phantom tenant.

Precedence is tenant= arg → bound context → Logger default. Leaving the Logger default unset keeps any unbound path failing loud, which surfaces bugs; set it to _global only if you'd rather never risk a TENANT_REQUIRED in a hot path.

Erasure benefit, confirmed: if your user ids are globally unique across your tenants (ScaiKey's are — 43 users, 43 distinct ids), one POST /v1/erase {tenant, subject} reaches a person regardless of tenant. If they're not, a DSAR becomes a cross-tenant fan-out — decide your subject-id scheme before wave 2, not after.

6. Deployment — the agent, and the trap that cost us a debugging session#

Install (one wheel, isolated venv, per host):

1
2
3

useradd --system scailog
python3 -m venv /opt/scailog-agent/venv
/opt/scailog-agent/venv/bin/pip install "scailog[agent]==1.0.2"   # or wheel URL + #sha256=

Agent systemd unit — key points from ScaiKey's working unit (deploy/standalone/scailog-agent.service):

• User=scailog, RuntimeDirectory=scailog (creates /run/scailog 0750), StateDirectory=scailog-agent (persistent cache — not /run, it must survive reboots), socket at /run/scailog/agent.sock (0660 scailog:scailog).
• No --tenant flag. The per-entry model means the agent must pass the frame's tenant through; a fixed --tenant would flatten everything to it (the 1.0.0 bug). 1.0.2 threads frame["tenant"] correctly — verify on your version with a two-tenant probe before trusting it.
• Agent is Wants=, never Requires=, of your service — a logging agent must not gate an IAM (or any) service. The stderr fallback exists for exactly the agent-down case, and it strips PI values (keeps field name + type), so a missing agent degrades safely.

Your service's unit needs a drop-in:

1
2
3
4
5
6

[Service]
SupplementaryGroups=scailog          # to reach the 0660 socket
ReadWritePaths=/run/scailog          # if you use ProtectSystem=strict
Environment=SCAILOG_SERVICE=<your service>
Environment=SCAILOG_ENV=production
Environment=SCAILOG_SOCKET=/run/scailog/agent.sock

THE TRAP. SCAILOG_* must be systemd Environment= lines, not in .env. pydantic-settings reads .env but does not export into os.environ, which the SDK reads directly. Miss SCAILOG_SERVICE and the SDK emits service="unknown"; your service-scoped writer key then rejects every entry as SERVICE_FORBIDDEN. This looks exactly like "logging is broken" and the agent log is silent about it — the rejection is server-side. It cost us a full debug loop. Set SCAILOG_SERVICE first, verify with a direct /v1/ingest probe if in doubt.

Two more, briefly: sudo -u <svc> test -w /run/scailog/agent.sock is a false negative (a fresh login doesn't inherit systemd's SupplementaryGroups; check /proc/<MainPID>/status Groups: instead). And /run/scailog at 0750 is unreadable to your own admin user — inspect the socket as root.

7. Keys — custody#

• Agent key: log_writer, ingest-only, scoped to your service. It may need tenant_scope: * (multi-tenant services attribute per entry). Know the cost: /v1/agent/enroll returns the unwrapped tenant KEK to any ingest-capable key, so a *-scoped writer key can pull every tenant's KEK. Mitigate with an ip_allowlist pinned to the host and a rate_limit.
• Never let the plaintext transit a transcript, ticket, or chat. Mint it and paste it directly into the root shell at install (SCAILOG_AGENT_KEY=... ./install...), or into /etc/scailog/agent.env (0640 root:scailog). A key that has been pasted into a session is compromised and must be rotated — this happened to us and is a standing debt.
• CI/manifest push: a log_provisioner (provision-only) or log_admin key. Push from a controlled release job, not a PR runner, and note manifest push has historically defaulted to the wrong tenant — pass it explicitly or use a version that made the endpoint service-scoped.

8. The CI gate#

Add to the lint job, blocking from day one:

1
2

- run: pip install "scailog[agent]==1.0.2"
- run: scailog-ci check src

A repo with zero call sites exits 0, so adding it estate-wide now is free until the first site is written. Blocking (not warn-then-block) is correct here: the violations it catches — interpolated messages, PI without subject — are the unrecoverable ones. A warn phase is a phase spent generating exactly the records you can never take back.

scailog-ci is Python-only (it ast-parses *.py). TS/Go/.NET services get the two local guards from §4 plus server-side drift detection: a nightly GET /v1/sites?tenant=...&auto_only=true / &pi_only=true compared to the pushed manifest flags any observed-but-undeclared or AUTO-* site, in every language.

9. Verification — prove it, don't trust the 201#

The discipline that caught two real bugs (the tenant flatten, the missing PyYAML) is install-and-run, then check the wire, not trust a success code.

• PI closure (Phase 1): drive the identity flows, then journalctl -u <svc> | grep -Ei '<email>|"(access|refresh|id)_token"' must be empty. Keep the AST guard as the durable regression control.
• Correlation (Phase 2): one request with X-Request-Id must echo back, appear in the log, and land in the audit row — one assertion covers the chain. Then 40 concurrent distinct ids must map 1:1 (catches contextvar leakage).
• Ingest (Phase 3): capture ingest_accepted on /v1/status, drive N requests, confirm it climbs by N and ingest_rejected stays flat. Confirm zero stderr-fallback lines in your journal (proves the socket path is used). Confirm panic_drops: 0.
• Attribution: read the agent cache (sqlite3 /var/lib/scailog-agent/agent.db 'SELECT tenant_id, call_site_id ...') — tenant-path entries under their slug, platform under _global, all shipped. (Your ingest-only key can't query the server; the cache is the authoritative local record.)
• Erasure (the real acceptance test): create a throwaway subject, generate activity, POST /v1/erase, confirm the PI no longer decrypts. Record any copy that erasure does not reach (see §11).

10. Sequencing across the estate#

Tiers, from ScaiLog's zero-Scai-dependency invariant and each service's fan-in:

• Tier 0 — ScaiLog itself. Deployed. Must never acquire a runtime dependency on a service that logs into it (do not install its oidc/vault/audit extras on the server — they pull SDKs and create a bootstrap cycle).
• Tier 1 — ScaiKey. Done. Everything authenticates through it, so it was the right reference and second adopter.
• Tier 2 — the rest (~39). Order by PI density first, infrastructure last (its blast radius is the whole estate). Add the free scailog-ci check to every Python repo now, while it's still a no-op.

Never put an agent on a static frontend host — no server-side app logs.

11. Known limits — state these, don't discover them#

• Retention deletes whole journal/agent files, not entries. A per-entry window is enforced going forward; a pre-existing backlog co-mingled with recent data ages out over the window, it can't be surgically purged.
• Access logs are out of scope by design. High-volume request logs (client IPs, login_hint emails) stay local with a real rotation policy — for ScaiKey, a 14-day journald cap on the backend matching the 14-day nginx logrotate on the proxy. Centralising them would swamp signal for little compliance gain. Just make sure the local rotation is configured, not aspirational.
• Mirroring an existing plaintext store (e.g. an audit_logs table) into ScaiLog does not make the original erasable. Dual-write improves DSAR reporting and adds an encrypted copy, but the plaintext original stays outside erasure's reach until you either make ScaiLog authoritative for that PI or encrypt the columns. Do not record "we mirror to ScaiLog" as closing an audit-PI finding.
• Agent server-outage behaviour: entries cache and ship on recovery, but PI for a subject whose key isn't already resident is panic-dropped permanently, and an agent restart during an outage loses all resident keys. Treat panic_drops > 0 as a page; never auto-restart an agent mid-incident.

Appendix — the ScaiKey reference#

• 100 call sites, scailog-ci clean; obs/{redaction,logging_config,context, sites,tenants}.py; guards test_no_pi_in_logs.py, test_log_sites.py.
• Live and ingesting from scaikey-be1; per-tenant attribution verified on the wire.
• Findings raised upstream this programme: 2026-08-25-notice-scailog-agent-tenant-flattening.md (the 1.0.0 flatten; fixed in 1.0.2). The RP-logout series is unrelated to ScaiLog but shows the same "verify against a live instance" discipline.