Skip to content

In-depth systems review (2026-05-21)

Scope and method

This review inspects architecture, runtime behavior, safety controls, security posture, test coverage, and simulation realism. Findings are based on repository evidence in code and documentation, with prioritized recommendations.

Executive summary

The project has a strong control spine for a simulator at this maturity stage: explicit policy tiers, per-call audit records, typed modules, and documented boundaries. The largest gaps are realism completeness, assurance depth for regulated contexts, and mismatch risk between documented architecture and current implementation state.

Top priority improvements:

  1. Close architecture-to-implementation drift by wiring or demoting currently stubbed subsystems and tools.
  2. Add deterministic replay and calibration harnesses to convert realism claims into measurable evidence.
  3. Harden deployment and supply chain posture for regulated operations, especially around auto-deploy and cloud inference pathways.
  4. Expand observability and negative-path testing for policy, audit degradation, and scenario reproducibility.

Prioritized findings

1) Correctness defects and behavior gaps

  1. Architecture promises broader subsystem orchestration than the active engine loop.
  2. docs/architecture.md describes tick orchestration across all subsystems and self-model refresh each tick.
  3. src/nous/engine.py currently wires power and APU estimators only, with defaulted load and ambient values.
  4. Risk: consumers over-trust device health and capability claims while major subsystem couplings are not active.

  5. Improvement: introduce explicit capability_matrix and fidelity_level outputs in device_info and device_health so downstream logic can gate behavior by implemented fidelity.

  6. Profile loading has permissive fallback behavior.

  7. _load_profile returns a generic fallback dict if profile file is missing or invalid shape.
  8. Risk: silent simulation under unrealistic defaults can invalidate scenario outcomes.

  9. Improvement: fail fast on missing profile in non-test mode, with an explicit strictness setting for CI/dev.

  10. Thermal and other subsystem stubs can imply false precision.

  11. src/nous/subsystems/thermal.py returns fixed values with additive time progression only.
  12. Risk: capability decisions can appear physically grounded while being static placeholders.
  13. Improvement: expose model_fidelity: stub|parametric|validated per subsystem in tool output.

2) Security and control-plane findings

  1. Admission control is clear but coarse for high-impact tool families.
  2. Tiering exists and runner enforcement is centralized, which is strong.
  3. Risk: regex-based allow/deny can be brittle and hard to audit across changing tool names.

  4. Improvement: add signed, versioned policy bundles with explicit tool IDs and optional argument constraints.

  5. Auto-deploy-to-production cadence is operationally aggressive.

  6. STATUS.md states production tracks main with frequent polling and auto-restart.
  7. Risk: narrow rollback windows and accidental propagation of quality regressions.

  8. Improvement: add staged promotion (main -> canary -> prod), with health gate checks and automatic rollback on failed post-deploy smoke checks.

  9. Cloud inference path creates data sovereignty and lock-in exposure.

  10. LIMITATIONS.md documents Anthropic cap behavior and cloud dependency path.
  11. Risk: non-EU processing and external control surface dependency for mission flows.
  12. Improvement: prioritize local model execution path for default mission mode, cloud only as explicitly enabled fallback with data classification gating.

3) Data loss, migration, and auditability risks

  1. Audit durability and verification depth are not fully evidenced in tests.
  2. Runner writes one record per call and hashes output body, which is a strong baseline.
  3. Risk: degraded sink or partial write behavior can reduce forensic confidence.

  4. Improvement: add crash-consistency tests for audit append behavior and periodic integrity verification job that checks line-level schema and hash consistency metadata.

  5. Reset and irreversible tool pathways need stronger guardrails in docs and tests.

  6. Irreversible classes exist in policy tiering.
  7. Risk: accidental enablement under permissive mode during operator workflows.
  8. Improvement: add mandatory dual-control token for T3 operations and explicit replayable audit evidence in integration tests.

4) Concurrency and determinism

  1. Deterministic replay is not first-class.
  2. Engine uses time and tick progression but deterministic seed controls are not surfaced as a clear contract.
  3. Risk: inability to reproduce a mission-state sequence exactly for incident review.

  4. Improvement: add deterministic seed, deterministic noise generator selection, and scenario event log export/replay command pair.

  5. Tick-loop load coupling is currently simplified.

  6. Power load is drawn from idle profile value rather than live compute/inference coupling.
  7. Risk: underestimation of thermal-power interaction and endurance variability.
  8. Improvement: implement cross-coupled compute -> thermal -> power derate feedback with bounded step solver.

5) Error handling and observability

  1. Error handling returns bounded strings, but structured machine-parsable errors are limited.
  2. Runner catches exceptions and returns [error Class: message] style output.
  3. Risk: controllers cannot reliably differentiate transient, policy, and invariant violations.

  4. Improvement: standardize error envelope JSON with code, class, retryable, and correlation_id fields.

  5. Degraded-mode signaling can be expanded.

  6. device_info surfaces audit.degraded which is useful.
  7. Improvement: add degraded flags for estimator divergence, stale sensor paths, profile fallback, and policy regex compile failures.

6) Test and verification posture

  1. Coverage appears focused on selected unit and smoke paths.
  2. There are unit, integration, and e2e folders, with limited evidence of scenario replay conformance checks.

  3. Improvement: add property-based invariants for energy conservation, monotonicity constraints, and safe-state transitions under randomized injections.

  4. Calibration against real benchmarks needs stronger continuous process.

  5. Documentation requires realistic values and BOM-first updates, which is excellent.
  6. Improvement: add calibration CI task that compares profile and model-card constants to docs/bom.md references and fails on undocumented drift.

Feature roadmap to improve realism

A) Physics and subsystem realism

  1. Add thermal RC network model with ambient and enclosure conduction, fan curve, and compute power coupling.
  2. Add battery hysteresis and temperature-dependent internal resistance curve, plus aging state (cycle_count, soh_pct).
  3. Add comms channel model with path loss exponent variants, shadowing, burst error model, and optional terrain plugin seam.
  4. Add operator biometrics physiology-lite model (hydration/strain coupling) before full biophysical model.

B) Mission realism and scenario expressiveness

  1. Extend scenario DSL with uncertainty annotations and adversarial injectors.
  2. Add mission phase templates (patrol, relay, stationary overwatch, extraction) with reusable load/comms envelopes.
  3. Add fault campaign runner that sweeps injector matrices and outputs pass/fail against STPA-derived constraints.

C) Assurance for regulated infrastructure

  1. Introduce signed artifacts for profile YAML and scenario bundles.
  2. Add reproducible run manifest: code revision, profile hash, scenario hash, seed, policy mode, tool transcript hashes.
  3. Add mandatory evidence package export for each run: key states, denied operations, degraded flags, and conformance report.

D) EU sovereignty oriented architecture options

  1. Default to self-hosted model runners for sensitive modes.
  2. Add configurable data localization controls for logs, metrics, and any external adapter output.
  3. Provide a no-cloud operational profile that hard-disables cloud inference and external telemetry adapters.

Proposed implementation plan (minimal to high impact)

  1. Immediate (small diffs, high confidence):
  2. Add fidelity metadata fields to tool outputs.
  3. Add strict profile-load option with explicit startup failure behavior.

  4. Add structured runner error envelope.

  5. Near term (moderate effort):

  6. Deterministic replay framework with seed and event-log export.
  7. Energy and thermal invariant property tests.

  8. Audit integrity periodic checker and report tool.

  9. Mid term (larger realism gains):

  10. Thermal and compute coupling implementation.
  11. Comms propagation and burst-noise model.
  12. Scenario fault campaign executor with STPA requirement assertions.

Unknowns and minimal checks needed

  1. Unknown: current CI branch protection and release promotion controls.
  2. Minimal check: inspect CI workflow files and deployment gate scripts.
  3. Unknown: real-world calibration error of current power/APU model against benchmark traces.
  4. Minimal check: run replay against measured trace set and compute RMSE/MAPE.
  5. Unknown: runtime performance at target tick rates with full subsystem coupling.
  6. Minimal check: benchmark with representative scenario profiles and publish p50/p95 loop latency.
  • Rollout: ship one feature flag at a time (strict_profile_load, deterministic_replay, structured_errors) with default-off where compatibility risk exists.
  • Rollback: disable the feature flag and redeploy previous tagged release; preserve audit and scenario artifacts for post-incident review.