Skip to content

ADR 0007: Tamper-evident audit log via per-record hash chaining

  • Status: Accepted
  • Date: 2026-06-01

Context

ADR 0002 makes the service account, not an internal sandbox, the trust boundary, and names the append-only audit log as the first compensating control. ADR 0003 classifies each call; the audit record is where that classification, the redacted arguments, and the SHA-256 of the output are committed to disk. The log is the most security-load-bearing artifact the project produces: the forensic record of everything a persuaded model (or a compromised client) did through the relay.

Until now its integrity rested on two controls, both outside the record:

  1. chattr +a (append-only) on the on-disk file, preserved across rotation by deploy/logrotate/relay-shell.
  2. Off-host shipping to a SIEM (docs/audit-shipper.md), so the authoritative copy lives somewhere the relay host cannot reach.

Both are necessary and both are kept. But SECURITY.md §"Residual risk" states that a compromise of the client or transport yields the capabilities of the service account, and in privileged posture, root. A root-equivalent attacker can chattr -a the file (root holds CAP_LINUX_IMMUTABLE), rewrite or excise lines, and restore the attribute in the window before the next shipper flush. Nothing in the on-disk record lets a downstream consumer detect that line N was edited, that a run of lines was deleted, or that two records were reordered. The hash-not-body invariant protects output confidentiality, not record-stream integrity. That is the gap this ADR closes.

The OWASP Logging Cheat Sheet (a trusted reference in CLAUDE.md) calls for log integrity protection for this threat. The canonical construction is a hash chain: each record commits to the hash of the one before it, so any edit, insertion, reorder, or interior deletion is detectable by recomputation, even from the shipped copy, after the fact, without trusting the relay host. Truncation at the file's head or tail is the chain's known boundary; see "What the single-file chain proves" below.

Decision

Add an opt-in, additive per-record hash chain to the audit log.

  • Opt-in, default off. A new RELAY_SHELL_AUDIT_CHAIN setting (default false) turns it on. When off, the record is byte-identical to v0.1: no new fields, no new code on the write path, no lock taken.
  • Additive record shape. When on, each record gains three trailing fields and nothing else changes:
  • seq: a monotonic integer, 0 at genesis.
  • prev: the previous record's chain (the 64-zero genesis anchor for seq 0).
  • chain: SHA-256(prev || canonical(record-without-chain)), where the canonical form is the record serialized with sorted keys and compact separators so the value is independent of dict insertion order and of the on-disk formatter. The existing fields (ts, tool, tier, denied, args, output_sha256, output_len, exit_code, request_id, client_id) keep their meaning and position. Off-host parsers built against the current shape keep working; they simply see three extra keys.
  • jsonl only. The chain is resumed across restarts and rotation by re-parsing the last on-disk record, which is only well-defined for the canonical jsonl format. RELAY_SHELL_AUDIT_CHAIN=true with RELAY_SHELL_AUDIT_FORMAT=cef|leef is rejected at startup (a config validation error). CEF/LEEF target a SIEM that owns integrity on its side.
  • Restart- and rotation-safe. On construction the logger reads the last record and resumes from its seq + 1 and chain. If the active file is empty after rotation, it reads the newest dateext- or numeric-named rotated sibling (plain or gzip-compressed) and resumes from that tail. While the process keeps running, the in-memory anchor follows the file (WatchedFileHandler reopens after a rename; copytruncate keeps the same fd), so the new file continues the same seq/chain. A missing, unchained, or unparseable active tail starts a fresh chain at genesis: a visible seam (a seq reset) that a verifier surfaces, never a silent gap. A non-empty malformed active tail is never bypassed in favor of older rotated history. The off-host stream remains authoritative for detecting missing tails and validating cross-file seams.
  • Ordering invariant under concurrency. The seq/prev read-modify-write and the line emit are taken under one lock, so the chain stays monotonic and correctly linked even if a caller records from another thread (e.g. the ADR 0006 seccomp-notify supervisor). With seccomp-notify off every write runs on the event-loop thread and the lock is uncontended; with it on, the lock serializes the supervisor thread's syscall_notify lines against the tool-call path. It is engaged only when chaining is on.
  • Offline verification, no new tool, fail-closed. A new CLI verb relay-shell --verify-audit [--audit-path PATH] [--segment] [--json] walks a file, mirroring --check-config and --verify-deploy. It is deliberately not an MCP tool: verifying the audit trail is an operator/forensic action, not something the audited model should drive (this also leaves the 21-tool contract, tests/test_server.py::_EXPECTED, untouched). The library audit.verify_chain is structural: it reports ok (no in-region break), records, present, and whether the region is genesis-anchored. The CLI applies a fail-closed policy: exit 0 only when the file exists, carries a chained record, verifies clean, and is genesis-anchored; a missing / empty / unchained log, a broken chain, or a non-genesis start (head-truncation) exits 2. --segment relaxes only the genesis-anchor requirement, for verifying a mid-stream rotation segment that legitimately starts at seq > 0. The default refuses to bless an absent or front-excised trail.

What the single-file chain proves (and what it does not)

A keyless single-file chain proves the records from its first surviving one to its last are unaltered, contiguous, and correctly ordered. By recomputation it detects an edit, insertion, reorder, or interior deletion anywhere in that range. It does not, from one file in isolation, prove the boundaries:

  • Head-truncation (excising leading records, including seq 0): the remaining records still form a valid sub-chain. Caught by the genesis anchor: a log built from genesis but no longer starting at seq 0 / genesis prev has had leading records removed. The CLI fails this by default (ChainResult.anchored exposes it programmatically); --segment opts out for a mid-stream rotation segment that legitimately starts at seq > 0. The ambiguous case (head-truncation vs rotation segment) resolves to "refuse" unless the operator asserts the segment.
  • Tail-truncation (dropping the newest records): leaves a shorter but valid prefix and is not detectable from the file alone. The defense is the off-host copy, which holds the later records, the same off-host shipping this ADR already designates as the durability/truncation control. An on-host high-water-mark would not change this: the residual-risk attacker who can truncate the log can clear an on-host checkpoint too (the same reason chattr +a is not sufficient). Tail-truncation is out of scope for the in-file chain by the same architectural choice that rejected external anchoring below.

The user-facing claims (SECURITY.md, README, --verify-audit help, docs/deployment.md §6a, docs/runbook.md §2.3) are scoped to exactly this: edits / insertions / reorders / interior deletions + head-truncation in the file; tail-truncation and cross-file/durability off-host.

Consequences

  • The audit-record schema grows three optional fields under RELAY_SHELL_AUDIT_CHAIN=true. Documented in docs/architecture.md §"Request lifecycle" (step 5) and docs/runbook.md §2.3. The default-off record is unchanged, so this is purely additive, the same compatibility promise ADR 0006 makes for its syscall_notify events.
  • server_info().audit reports format and chain, so the live integrity posture is visible without re-deriving it from env names.
  • The runbook §2.3 "audit-the-audit" step gains a chain-verification check (relay-shell --verify-audit) when chaining is enabled. The §3.3 security-sensitive checklist gains the chain fields alongside the existing audit-record-field check.
  • docs/audit-shipper.md is unchanged: the chained record is still one JSONL line per call on the stream the three recipes already tail. A SIEM that re-verifies the chain gains end-to-end tamper-evidence from the relay host to the sink.
  • Operational guidance: enable chaining on a freshly rotated log so the chain runs from genesis. Verify the live log (or a shipped copy) with --verify-audit --audit-path; the fail-closed default suits a log that should be complete from genesis. Add --segment only for a mid-stream rotation segment. When the process ran across a rotation, or restarted with an empty active file, cross-rotation continuity is an equality check on the seam (prev of file N+1's first record == chain of file N's last record), which the verifier prints as the start anchor. Unexpected genesis segments remain verification failures unless an operator explicitly records and reviews the reset as an incident.

Rejected alternatives

  • HMAC-with-a-secret-key instead of a plain hash chain. An HMAC keyed by a secret the relay does not store on the same host would also defend against an attacker forging a fresh consistent chain after truncating the log (a plain chain lets a root attacker recompute a clean chain over doctored records). But it requires a key-management story (where the key lives, rotation; a host that holds the key to sign yields it to the same compromise) out of proportion to an opt-in integrity aid, and it changes the verification trust model from "anyone with the file" to "anyone with the file and the key". The plain chain already defeats the dominant threat, silent in-place edits and excisions that the off-host shipper has not yet captured, because the shipped prefix pins every hash the attacker would have to remain consistent with. Tamper-evidence against a host that holds no signing secret is the goal; tamper-proofing against a key-holding root is the off-host shipper's job. Recorded so a future HMAC/Merkle-anchor extension has a starting point.
  • External anchoring (periodic Merkle root to a notary / transparency log). Strongest integrity, but adds a network dependency and a second service on the audit hot path, the coupling the single-process architecture (ADR 0001) avoids. The hash chain is the in-band primitive an external anchor would build on; ship the primitive first.
  • Rely on chattr +a and the off-host shipper alone (status quo). Both are kept, but neither makes a single altered record detectable from the file, and the shipper has a flush window. The chain is the missing in-record evidence.
  • A second audit.chain sidecar file of hashes. Splitting the chain from the records makes off-host shipping and forensic correlation harder and adds a second file to keep append-only and rotate in lockstep. One stream with the hash inline (as resource-read events, tool="resource:<name>", already discriminate on a field rather than a file) ships on the existing pipeline.
  • Chain in CEF/LEEF too. The chain math is format-independent, but resuming it across a restart means re-parsing the last record, which is clean for jsonl and lossy for the SIEM text formats. Rather than emit a chain that cannot be reliably resumed (producing genesis seams on every restart), the combination is refused at startup. SIEM integrity is the aggregator's responsibility in those deployments.

Validation outcome (2026-06-01)

Implemented and validated in the same PR (the ADR 0005 four-step pass; full record in docs/adr/0005-codebase-validation.md §"Validation outcome (2026-06-01)" and audit/2026-06-01-engagement.md):

  • ruff check, ruff format --check, mypy --strict clean. pytest -q: 277 passed, 13 deselected (up from 250; +27 chain/config/CLI tests). pytest -m fuzz: 13 invariants pass. coverage: 92% with subprocess collection (floor 90%); config.py 99%, audit.py 95%.
  • 21 MCP tools and 3 resources unchanged: the chain adds a CLI verb, not a tool, so tests/test_server.py::_EXPECTED is untouched.
  • Behavior validation: a chained log emits seq/prev/chain with a genesis anchor and correct linkage; resumes seq across a simulated restart; verify_chain returns intact on a clean log and detects edit, chain-field forgery, interior deletion, reorder, a garbage line, a non-genesis seq-0 anchor, and a legacy line inside the region; reports anchored=false for a head-truncated log and present=false for a missing file; default-off records carry none of the three fields (byte-identical to v0.1); and audit_chain=true with a non-jsonl format is rejected at startup. The --verify-audit CLI is fail-closed: exit 0 on a clean genesis chain; exit 2 on an edited body, a head-truncated log (no --segment), a missing log, or an unchained log; exit 0 on a head-truncated log with --segment.

PR-review hardening

Automated review (Copilot + Codex) plus the bundled /security-review converged on one point: the initial draft overclaimed detection and was too lenient by default. A keyless single-file chain cannot detect boundary truncation, and a verifier must not bless an absent or front-excised log. The fix kept the architecture (no sidecar, no external anchor, both rejected above) and (a) made --verify-audit fail-closed: a missing / empty / unchained log, a broken chain, or a non-genesis start (head-truncation) all exit 2 by default, with --segment the explicit opt-out for a rotation segment; (b) added ChainResult.anchored / present so head-truncation and absence are surfaced; (c) scoped tail-truncation and cross-file durability to the off-host stream in every user-facing claim. A 2026-07-28 operational review subsequently closed the rotation-then-restart gap: an empty active file now resumes from the newest rotated sibling, while a non-empty malformed tail still fails visibly instead of falling back. See "What the single-file chain proves". Regression tests pin head-truncation (anchored=false), tail-truncation (valid-prefix, the documented limitation), and the fail-closed CLI exit codes for missing, unchained, head-truncated (with and without --segment), and genesis logs.

No change to policy admission, tier semantics, the no-sandbox posture, or any tool's response shape. This pass hardened a compensating control (ADR 0002's first one); it did not move the trust boundary.