Shipping the audit log off-host¶

relay-shell writes one JSON object per tool call to a local audit file (/var/log/relay-shell/audit.jsonl by default). On its own that file is evidence only until the host is compromised: an attacker with sufficient privilege can rotate it, mask it, or block writes. The audit guarantee survives only if the records are shipped to a system the operator controls separately from the relay host. The local hash chain cannot show that the newest records were truncated; only an off-host copy can.

This guide gives one worked example each for Vector, Fluent Bit and journalctl → systemd-journal-remote. Pick one; they overlap. The choice is operational (what is already in your stack, how the SIEM ingests).

This page is canonical for streaming the live log off-host. The hourly chain verifier, the anchor and the rsync mirror (including the rollback check, retention pruning and the approval ledger) are canonical in deploy/evidence/README.md and complement, not replace, the shippers here.

Cross-references:

The audit-file contract and chattr +a posture: SECURITY.md and docs/deployment.md §6.
The runbook's "audit-the-audit" procedure: docs/runbook.md §2.3.

0. Common requirements¶

A shipper that meets the project's audit posture must:

Preserve append-only. The local file has chattr +a set (and logrotate drops/restores the attribute across rotation; see deploy/logrotate/relay-shell). The shipper reads, it does not rewrite. None of the three examples below modify the file.
Preserve content. No JSON re-encoding that drops fields, no field-level redaction beyond what redaction.py already did, no truncation. Records are already bounded by the relay (output_sha256 + output_len, never the body).
Survive log rotation. The shipper must reopen the file after logrotate moves it. Vector and Fluent Bit use inode tracking (the source of truth across a rotation); the tail -F-based journal forwarder follows by name and reopens on rotation, which is acceptable here because the bundled logrotate config uses create (the new file is created at rotation, with no records buffered past the close of the old fd). See "Rotation gap risk" in §3 for the residual race.
Be observable. Drops, retries, and back-pressure events must reach the operator. If the shipper silently buffers for hours, you have lost the property you were paying for.
Send over TLS to a remote collector. Examples below use TLS for transport; the listener-side configuration is out of scope (it is your SIEM / log-aggregator's responsibility).

The relay's installer creates /var/log/relay-shell/audit.jsonl as 0600 relay-shell:relay-shell (see deploy/install.sh and deploy/logrotate/relay-shell). The shipper needs read access. Two documented approaches:

Dedicated shipper user + POSIX ACL (recommended). Keep the shipper under its own service account and grant explicit read with setfacl -m u:<shipper>:r /var/log/relay-shell/audit.jsonl plus a setfacl -m u:<shipper>:rx /var/log/relay-shell. The audit file is append-only (chattr +a, set by deploy/install.sh and kept by the bundled logrotate config), and Linux refuses an ACL change on an append-only file (Operation not permitted), so clear the flag for the change and set it again, and re-apply the grant after every rotation (see "Re-applying the ACL after rotation" below).
Run the shipper as the relay-shell user. This avoids ACL management but weakens separation of duties: the same uid that can execute relay commands can also stop or tamper with shipper process state. Use only if your threat model explicitly accepts that tradeoff.

Re-applying the ACL after rotation¶

Logrotate creates each new audit.jsonl as 0600 relay-shell:relay-shell without the ACL, and the bundled config sets +a on it before it finishes, so the grant has to be re-applied after every rotation run. Do it outside /etc/logrotate.d/relay-shell: that file is managed by deploy/install.sh (which replaces it on the next run), and an edit shows as drift under relay-shell --verify-deploy. Where logrotate runs from logrotate.timer (current Debian and Ubuntu), a drop-in on its service does it after each run, whether or not the run succeeded. Replace <shipper> with the shipper's user (vector, fluent-bit, ...), one setfacl per user:

sudo mkdir -p /etc/systemd/system/logrotate.service.d
sudo tee /etc/systemd/system/logrotate.service.d/relay-shell-audit-acl.conf >/dev/null <<'EOF'
[Service]
ExecStopPost=/bin/sh -c '/usr/bin/chattr -a /var/log/relay-shell/audit.jsonl && /usr/bin/setfacl -m u:<shipper>:r /var/log/relay-shell/audit.jsonl; /usr/bin/chattr +a /var/log/relay-shell/audit.jsonl'
EOF
sudo systemctl daemon-reload

The file is not append-only for the moment the ACL is written, and the shipper cannot open a newly rotated file until the drop-in has run. Where logrotate runs from cron instead, run the same three commands after it.

Do not weaken the file mode to grant group read; the 0600 default is part of the on-host posture. The shipper writes nowhere under the relay state directory in either approach.

1. Vector (recommended for most operators)¶

Vector (https://vector.dev) is a single static binary with a strict config and end-to-end metrics. Pick it when one tool should own the pipeline and expose back-pressure as Prometheus metrics.

Install¶

Use the official Timber-maintained apt repository with an explicit keyring, not the curl ... | bash one-liner, so the steps are auditable.

# 1. Fetch and verify the signing key. The fingerprint is published
#    at https://vector.dev/download/ - confirm it before importing.
sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://repositories.timber.io/public/vector/gpg.3543DA2B.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/vector.gpg
sudo chmod 0644 /etc/apt/keyrings/vector.gpg

# 2. Add the apt source pinned to the keyring.
echo "deb [signed-by=/etc/apt/keyrings/vector.gpg] \
  https://repositories.timber.io/public/vector/deb/ubuntu $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/vector.list

# 3. Install.
sudo apt-get update && sudo apt-get install -y vector

Config (`/etc/vector/vector.yaml`)¶

# /etc/vector/vector.yaml
# Read the relay-shell audit log line-by-line, parse each record as
# JSON, and ship to a remote Vector / OTel / SIEM endpoint over TLS.

data_dir: /var/lib/vector

sources:
  relay_shell_audit:
    type: file
    include:
      - /var/log/relay-shell/audit.jsonl
    # Track by inode so rotation does not lose position. Vector
    # remembers checkpoints under data_dir.
    fingerprint:
      strategy: device_and_inode
    # Restart from beginning if the relay host was offline during a
    # rotation; checkpoint dedupes already-shipped records.
    read_from: beginning
    # Don't follow truncations - the file is append-only on disk.
    ignore_older_secs: 0
    # Bound memory.
    max_line_bytes: 1048576

transforms:
  parse:
    type: remap
    inputs: [relay_shell_audit]
    # One JSON object per line. If a line ever fails to parse,
    # forward it as-is with an .error field so the alert fires
    # rather than the record being dropped.
    source: |
      parsed, err = parse_json(.message)
      if err == null {
        . = merge(., parsed)
      } else {
        .parse_error = err
      }
      .host = get_hostname!()
      .service = "relay-shell"

sinks:
  remote_collector:
    type: vector
    inputs: [parse]
    address: collector.example.org:6000
    compression: true
    tls:
      enabled: true
      verify_certificate: true
      ca_file: /etc/vector/ca.pem
    healthcheck:
      enabled: true
    # Back-pressure: buffer to disk so an upstream outage doesn't
    # drop records. Alert when usage > 80%.
    buffer:
      type: disk
      max_size: 268435456  # 256 MiB
      when_full: block

  # Local Prometheus exporter so the shipper itself is observable.
  internal_metrics:
    type: prometheus_exporter
    inputs: [_internal_metrics_]
    address: 127.0.0.1:9598

Run as a dedicated Vector user (recommended)¶

Keep Vector as its own user and grant read access to the audit file with ACLs:

sudo id -u vector >/dev/null 2>&1 || sudo useradd --system --home /var/lib/vector --shell /usr/sbin/nologin vector
sudo install -d -m 0755 -o vector -g vector /var/lib/vector
sudo setfacl -m u:vector:rx /var/log/relay-shell
sudo chattr -a /var/log/relay-shell/audit.jsonl
sudo setfacl -m u:vector:r /var/log/relay-shell/audit.jsonl
sudo chattr +a /var/log/relay-shell/audit.jsonl

sudo mkdir -p /etc/systemd/system/vector.service.d
sudo tee /etc/systemd/system/vector.service.d/override.conf >/dev/null <<'EOF'
[Service]
User=vector
Group=vector
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now vector

If you instead run Vector as relay-shell, treat that as an explicit risk acceptance (see §0).

Verify¶

# Vector ran without config errors:
sudo systemctl status vector --no-pager
sudo journalctl -u vector -n 100 --no-pager | grep -iE "error|warn" || true

# Drive one tool call through the relay and confirm it lands at the
# remote side. From the relay host (Settings(audit_path=...) wins
# over the env var, so pass the path explicitly here):
python -c "
import asyncio
from relay_shell.config import Settings
from relay_shell.server import build_server
m = build_server(Settings(audit_path='/var/log/relay-shell/audit.jsonl'))
asyncio.run(m.call_tool('server_info', {}))
"

# Internal metrics: events received, retries, buffer usage.
curl -s http://127.0.0.1:9598/metrics | grep -E '^vector_(events|errors|buffer)'

Troubleshoot¶

"permission denied" on the audit file. Check ls -l /var/log/relay-shell/audit.jsonl: the file is created 0600 relay-shell:relay-shell by deploy/install.sh and the bundled logrotate config preserves that mode. Grant the vector user a POSIX ACL as above (with chattr -a / chattr +a around it, and re-applied after rotation; see "Re-applying the ACL after rotation"). Do not chmod the file.
Records duplicated after restart. Confirm data_dir is persistent and writable by the user running Vector (vector in this recipe); the checkpoint lives there.
Records lost after rotation. logrotate should be the bundled config (deploy/logrotate/relay-shell), which uses create so the inode changes only at rotate time. Vector's device_and_inode fingerprint follows the rename.

2. Fluent Bit (lightweight, broad output plugin set)¶

Fluent Bit (https://fluentbit.io) fits if your aggregator speaks Forward, Loki, OpenSearch, S3 or a cloud-native sink natively, or if you want a smaller footprint than Vector. The pipeline below is a thin file → JSON → output chain.

Install¶

Use the official Fluent Bit apt repository with an explicit keyring, not the curl ... | sh one-liner.

# 1. Fetch and verify the signing key. The fingerprint is published
#    at https://docs.fluentbit.io/manual/installation/linux/ubuntu -
#    confirm it before importing.
sudo install -d -m 0755 /etc/apt/keyrings
curl -fsSL https://packages.fluentbit.io/fluentbit.key \
  | sudo gpg --dearmor -o /etc/apt/keyrings/fluentbit.gpg
sudo chmod 0644 /etc/apt/keyrings/fluentbit.gpg

# 2. Add the apt source pinned to the keyring.
echo "deb [signed-by=/etc/apt/keyrings/fluentbit.gpg] \
  https://packages.fluentbit.io/ubuntu/$(lsb_release -cs) $(lsb_release -cs) main" \
  | sudo tee /etc/apt/sources.list.d/fluent-bit.list

# 3. Install.
sudo apt-get update && sudo apt-get install -y fluent-bit

Config (`/etc/fluent-bit/fluent-bit.conf`)¶

# /etc/fluent-bit/fluent-bit.conf
# Tail the relay-shell audit log, parse JSON, ship over Forward (TLS).

[SERVICE]
    Flush         5
    Daemon        Off
    Log_Level     info
    Parsers_File  parsers.conf
    HTTP_Server   On
    HTTP_Listen   127.0.0.1
    HTTP_Port     2020
    storage.path  /var/lib/fluent-bit/storage
    storage.sync  normal

[INPUT]
    Name              tail
    Path              /var/log/relay-shell/audit.jsonl
    # Inode-based tracking; rotation is transparent.
    Inotify_Watcher   true
    Refresh_Interval  5
    Rotate_Wait       30
    DB                /var/lib/fluent-bit/relay-shell-audit.db
    DB.Sync           normal
    Tag               relay_shell.audit
    Parser            relay_shell_audit
    # Skip lines longer than Buffer_Max_Size instead of halting the file; alert on drops via the metrics endpoint.
    Skip_Long_Lines   On
    storage.type      filesystem

[FILTER]
    Name              modify
    Match             relay_shell.audit
    Add               host ${HOSTNAME}
    Add               service relay-shell

[OUTPUT]
    Name              forward
    Match             relay_shell.audit
    Host              collector.example.org
    Port              24224
    # Mandatory in production.
    tls               On
    tls.verify        On
    tls.ca_file       /etc/fluent-bit/ca.pem
    # Buffer to disk to survive upstream outages.
    storage.total_limit_size 256M
    Retry_Limit       False
    # Compression on the wire.
    Compress          gzip

And the parser (/etc/fluent-bit/parsers.conf):

[PARSER]
    Name        relay_shell_audit
    Format      json
    # Deliberately no Time_Key / Time_Format. The relay emits "ts" as
    # an ISO-8601 string with a `+00:00` offset (see relay_shell.util
    # `now_iso()` and tests/test_util.py::test_now_iso_has_offset);
    # Fluent Bit's strptime varies in colon-offset support across
    # libc versions. The `ts` field is preserved in the record for
    # the receiving SIEM to index as the event time; Fluent Bit's
    # default per-record arrival timestamp covers the agent side.

Run as a dedicated Fluent Bit user (recommended)¶

Keep Fluent Bit as its own user and grant read access with ACLs:

sudo id -u fluent-bit >/dev/null 2>&1 || sudo useradd --system --home /var/lib/fluent-bit --shell /usr/sbin/nologin fluent-bit
sudo install -d -m 0755 -o fluent-bit -g fluent-bit /var/lib/fluent-bit/storage
sudo setfacl -m u:fluent-bit:rx /var/log/relay-shell
sudo chattr -a /var/log/relay-shell/audit.jsonl
sudo setfacl -m u:fluent-bit:r /var/log/relay-shell/audit.jsonl
sudo chattr +a /var/log/relay-shell/audit.jsonl

sudo mkdir -p /etc/systemd/system/fluent-bit.service.d
sudo tee /etc/systemd/system/fluent-bit.service.d/override.conf >/dev/null <<'EOF'
[Service]
User=fluent-bit
Group=fluent-bit
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now fluent-bit

If you instead run Fluent Bit as relay-shell, treat that as an explicit risk acceptance (see §0).

Verify¶

sudo systemctl status fluent-bit --no-pager
sudo journalctl -u fluent-bit -n 100 --no-pager | grep -iE 'error|fail' || true

# HTTP metrics: events ingested, dropped, retried.
curl -s http://127.0.0.1:2020/api/v1/metrics/prometheus | grep -E '^fluentbit_(input|output)'

# End-to-end: drive a tool call, then watch the relay side and the
# remote receiver. On the receiver, you should see one record per
# call with the original fields plus `host`, `service`.

Troubleshoot¶

"file rotation event lost": confirm Rotate_Wait is at least the time logrotate holds the old file before unlinking; 30 s is the value in the config above, not something the bundled logrotate config sets.
Records replayed after restart: DB must be on a persistent path with DB.Sync normal (not off); the position is otherwise lost on crash.

3. journalctl → `systemd-journal-remote`¶

Use this when the relay host already runs systemd-journald and the ops standard is "journal everything, ship the journal". It is the lowest-friction option without a third-party agent on the host, but the audit file must also be delivered to the journal (it is not, by default). The recipe has two parts:

A tiny forwarder unit tail -Fs the audit file and pipes each line to journald. Each line is stored verbatim as the journal record's MESSAGE field; journald does not parse JSON into structured fields automatically; the JSON travels as a string and the receiving SIEM is responsible for re-parsing it. Pair with a SYSLOG_IDENTIFIER so the records are easy to query.
systemd-journal-upload.service runs on the relay host and pushes journal entries over HTTPS to a collector. The collector side runs systemd-journal-remote.service to receive. The two service names are easy to confuse, but they sit on different hosts.

Forward the audit log into the journal¶

/etc/systemd/system/relay-shell-audit-tail.service:

[Unit]
Description=Forward relay-shell audit.jsonl into the systemd journal
After=relay-shell.service
Wants=relay-shell.service

[Service]
Type=simple
# tail -F follows rotation by name and reopens when logrotate's
# `create` directive lands a new file. SyslogIdentifier sets a stable
# tag in the journal; each MESSAGE is the original JSONL line verbatim
# - the receiving SIEM is responsible for JSON-parsing it.
ExecStart=/bin/sh -c 'exec /usr/bin/tail -n0 -F /var/log/relay-shell/audit.jsonl'
StandardOutput=journal
SyslogIdentifier=relay-shell-audit
User=relay-shell-audit-tail
Group=relay-shell-audit-tail
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

sudo id -u relay-shell-audit-tail >/dev/null 2>&1 || sudo useradd --system --home /nonexistent --shell /usr/sbin/nologin relay-shell-audit-tail
sudo setfacl -m u:relay-shell-audit-tail:rx /var/log/relay-shell
sudo chattr -a /var/log/relay-shell/audit.jsonl
sudo setfacl -m u:relay-shell-audit-tail:r /var/log/relay-shell/audit.jsonl
sudo chattr +a /var/log/relay-shell/audit.jsonl
sudo systemctl daemon-reload
sudo systemctl enable --now relay-shell-audit-tail.service

Confirm in the journal:

journalctl -t relay-shell-audit -o cat -n 5
# Each line is the original JSONL record verbatim.

Ship the journal to a remote collector¶

On the relay host, install and configure systemd-journal-upload:

sudo apt-get install -y systemd-journal-remote   # provides upload too

/etc/systemd/journal-upload.conf:

[Upload]
URL=https://journal-collector.example.org:19532
# Mutual TLS - the upload side authenticates with a client cert.
ServerKeyFile=/etc/ssl/private/journal-upload.key
ServerCertificateFile=/etc/ssl/journal-upload.crt
TrustedCertificateFile=/etc/ssl/journal-ca.crt

sudo systemctl enable --now systemd-journal-upload.service

On the collector host (out of scope for this repo; documented here for completeness), systemd-journal-remote.service listens on 19532/tcp with a peer cert and writes to a local journal namespace that the SIEM ingests.

Verify¶

# 1. Forwarder is reading the file and emitting to the journal:
journalctl -u relay-shell-audit-tail --no-pager -n 5

# 2. Upload service is connected and not erroring:
systemctl status systemd-journal-upload --no-pager
journalctl -u systemd-journal-upload -n 100 --no-pager | grep -iE 'error|fail' || true

# 3. End-to-end: drive a relay tool call, then on the collector,
# confirm the matching record appears under SYSLOG_IDENTIFIER=relay-shell-audit.

Rotation gap risk¶

tail -F (--follow=name --retry) tracks the file by name and reopens when logrotate lands a new file at the canonical path. This is acceptable here because the bundled deploy/logrotate/ relay-shell uses create (the new file is created at rotation), but it is not inode-safe in the strict sense Vector and Fluent Bit provide. A small race exists: if the relay writes the very last byte to the old inode after tail -F has switched to the new path, that byte ships when the rotated/compressed old file is ingested by your aggregator (it is still on disk under audit.jsonl-YYYYMMDD[.gz]), not through this forwarder. For an audit pipeline that is the documented price for having no third-party agent on the host; if a zero-gap guarantee is required, prefer the Vector recipe.

Troubleshoot¶

No records in the journal. Confirm the forwarder unit is running and reading the file: journalctl -u relay-shell-audit-tail -n 5. With the bundled logrotate config (create 0600 relay-shell relay-shell) tail -F reopens the new file as soon as it appears.
Upload backs off without recovering. Check the collector cert and that 19532/tcp is reachable; systemd-journal-upload's only remediation on connection error is exponential back-off with no on-disk spool.

4. Picking one¶

Scenario	Pick
Standalone shipper, one tool owns the pipeline, Prom metrics	Vector
Fluent / Loki / OpenSearch / cloud-native sink, small footprint	Fluent Bit
journald-centric ops org, no third-party agent on the host	journal-remote
You want the audit record AND the journal both shipped	journal-remote + a copy of the JSONL via Vector or Fluent Bit

Each preserves the same property: a remote copy of every audit record, written without modifying the on-host append-only file, with back-pressure that surfaces rather than silently dropping. The relay's audit guarantee ends at the filesystem; the shipper extends it to the rest of your stack. For chain verification and a verified rsync mirror of the rotated segments, see deploy/evidence/README.md.

Shipping the audit log off-host¶

0. Common requirements¶

Re-applying the ACL after rotation¶

1. Vector (recommended for most operators)¶

Install¶

Config (/etc/vector/vector.yaml)¶

Run as a dedicated Vector user (recommended)¶

Verify¶

Troubleshoot¶

2. Fluent Bit (lightweight, broad output plugin set)¶

Install¶

Config (/etc/fluent-bit/fluent-bit.conf)¶

Run as a dedicated Fluent Bit user (recommended)¶

Verify¶

Troubleshoot¶

3. journalctl → systemd-journal-remote¶

Forward the audit log into the journal¶

Ship the journal to a remote collector¶

Verify¶

Rotation gap risk¶

Troubleshoot¶

4. Picking one¶

Config (`/etc/vector/vector.yaml`)¶

Config (`/etc/fluent-bit/fluent-bit.conf`)¶

3. journalctl → `systemd-journal-remote`¶