Reference Architecture: Maximum-Autonomy IT Operations

Abstract: A reference architecture for maximising operational autonomy in IT infrastructure while maintaining compliance, safety, and auditability. The architecture targets 70–85% automation of operational toil with explicit human decision gates at irreducible control points. It covers five operational layers (platform, patching, security, compliance, change management), observability, agentic AI integration, and a phased implementation roadmap.

Executive Summary

Core principle: Automate the predictable; gate the consequential.

Every domain has a hard ceiling – the point beyond which automation requires human judgement, regulatory accountability, or decisions under genuine uncertainty. This architecture is designed around those ceilings, not against them.

1. Architecture Principles

1.1 Design Axioms

1. Immutability over mutation. Prefer replacing infrastructure over patching in place. Immutable images, declarative state, GitOps reconciliation.
2. Deterministic over probabilistic. Automation actions must be predictable and testable. LLM/AI-driven actions are advisory until validated in a closed-loop with deterministic verification.
3. Least privilege, least blast radius. Every automated agent operates with scoped permissions and bounded blast radius. Canary before fleet. Feature flag before hard deploy.
4. Evidence by default. Every automated action produces a signed, timestamped audit trail. If you can’t prove it happened correctly, it didn’t.
5. Fail safe, not fail silent. Unknown states trigger safe-halt or degraded mode, never silent continuation.
6. Human gates are structural, not temporary. Certain decisions require human accountability by regulation and by good engineering. These are not automation gaps to be closed – they are load-bearing control points.

1.2 Autonomy Levels (Graduated Model)

Adapted from the operational maturity model emerging in the Agentic SRE space:

Target state for this architecture: L3–L4 for standard operations, L1–L2 for risk-bearing changes.

No component should operate at a level beyond what its verification evidence supports.

2. Layer 1 – Immutable, Self-Healing Base Platform

2.1 Purpose

Provide a compute substrate that recovers from common failure modes without human intervention. This is the most mature automation domain and should be built first.

2.2 Reference Stack

Operating system: Immutable, minimal-attack-surface OS.

• Talos Linux (Kubernetes-native, API-managed, no SSH, no shell)
• Flatcar Container Linux (auto-updating, immutable root)
• Bottlerocket (AWS-native, API-managed)

Selection criteria: No general-purpose OS in the compute plane. General-purpose Linux (Ubuntu, RHEL) only for management/bastion nodes with CIS hardening.

Container orchestration: Kubernetes (managed or self-hosted Talos).

• Control plane: minimum 3 nodes, etcd with automated backup and restore
• Node auto-repair: cloud provider node auto-repair or Cluster API machine health checks
• Pod self-healing: liveness/readiness probes, PodDisruptionBudgets, topology spread constraints

Service mesh: Optional but recommended for mTLS, circuit breaking, retry budgets.

• Istio, Linkerd, or Cilium service mesh
• Automatic mTLS rotation (cert-manager with short-lived certificates)

2.3 Self-Healing Capabilities

2.4 What Cannot Be Automated

• Novel failure modes not covered by existing remediation playbooks
• Cascading failures crossing service boundaries in unpredictable ways
• Split-brain scenarios requiring data reconciliation decisions
• Infrastructure architecture changes (adding regions, changing topology)

2.5 Implementation Pattern

┌─────────────────────────────────────────────────────┐
│                   Git Repository                     │
│  (Infrastructure state, Kubernetes manifests,        │
│   Helm charts, Kustomize overlays)                   │
└──────────────┬──────────────────────────────────────┘
               │ GitOps sync
               ▼
┌──────────────────────────┐    ┌─────────────────────┐
│  Flux / ArgoCD           │◄───│  Drift Detection    │
│  (Continuous reconcile)  │    │  (alert on manual    │
│                          │    │   cluster changes)   │
└──────────┬───────────────┘    └─────────────────────┘
           │
           ▼
┌──────────────────────────────────────────────────────┐
│  Kubernetes Cluster (Talos / managed K8s)             │
│  ┌──────────┐  ┌──────────┐  ┌───────────────────┐  │
│  │ HPA/VPA  │  │ PDB      │  │ Cluster API /     │  │
│  │ (scaling)│  │ (budget) │  │ Node auto-repair  │  │
│  └──────────┘  └──────────┘  └───────────────────┘  │
│  ┌──────────────────┐  ┌────────────────────────┐   │
│  │ cert-manager     │  │ Service mesh (mTLS,    │   │
│  │ (cert rotation)  │  │  circuit breaker)      │   │
│  └──────────────────┘  └────────────────────────┘   │
└──────────────────────────────────────────────────────┘

2.6 Evidence Capture

• GitOps sync status and history (Flux/ArgoCD events)
• Node lifecycle events (creation, deletion, repair)
• Scaling events (HPA/VPA decisions with reasoning)
• Certificate rotation events (cert-manager logs)
• Drift detection alerts with before/after state

3. Layer 2 – Continuous Patching Pipeline

3.1 Purpose

Automatically apply security and dependency patches with verification gates, maintaining compliance SLAs for patch windows while minimising human intervention.

3.2 Patching Domains

3.3 Automated Patching Pipeline

CVE Feed / Upstream Release
        │
        ▼
┌───────────────────────┐
│  Vulnerability Scanner │ ◄── Trivy, Grype, Snyk
│  (continuous scan of   │     scanning container
│   images + deps)       │     registry + repos
└──────────┬────────────┘
           │ New CVE or dependency update detected
           ▼
┌───────────────────────┐
│  Renovate / Dependabot │
│  (auto-PR with         │
│   changelog + diff)    │
└──────────┬────────────┘
           │ PR opened
           ▼
┌───────────────────────┐
│  CI Pipeline           │
│  Build, test, SAST,    │
│  DAST, SCA, container  │
│  scan, SBOM, signing   │
└───────────┬───────────┘
            │ All checks pass
            ▼
┌─────────────────────────────────────────────┐
│  Auto-merge Policy Engine                    │
│  IF severity < CRITICAL                      │
│  AND test coverage >= threshold              │
│  AND no breaking API changes                 │
│  AND dependency is in approved-list           │
│  AND SBOM diff is within policy              │
│  THEN auto-merge to staging branch           │
│  ELSE require human review                   │
└──────────────┬──────────────────────────────┘
               │
               ▼
┌───────────────────────┐
│  Canary Deployment     │
│  (Argo Rollouts /      │
│   Flagger)             │
│  Monitor: error rate,  │
│  latency, success rate,│
│  resource usage        │
│  Auto-promote if SLO   │
│  met; rollback if not  │
└──────────┬────────────┘
           │
           ▼
┌───────────────────────┐
│  Progressive Rollout   │
│  5% → 25% → 50% → 100%│
│  with SLO gates at     │
│  each stage            │
└───────────────────────┘

3.4 Auto-Merge Rules (Policy-as-Code)

These rules determine which patches can proceed without human review. They should be conservative and tightened over time based on incident data.

# Example: Renovate auto-merge policy (conceptual)
auto_merge_criteria:
  # Patch version bumps of well-known, low-risk deps
  - match:
      update_type: "patch"
      dependency_type: "production"
      severity: ["low", "medium"]
    requires:
      ci_pass: true
      test_coverage_delta: ">= 0"  # no coverage regression
      breaking_changes: false
      sbom_policy_check: pass
    action: auto_merge

  # Security patches – critical CVEs get fast-tracked
  # but still require canary verification
  - match:
      update_type: "any"
      cve_severity: "critical"
      cisa_kev: true  # Known Exploited Vulnerability
    requires:
      ci_pass: true
    action: auto_merge_to_canary
    escalation: page_oncall_if_canary_fails

  # Everything else: human review
  - match:
      update_type: "major"
    action: require_human_review

  - match:
      dependency_type: "database_engine"
    action: require_human_review

  - match:
      update_type: "minor"
      breaking_changes: true
    action: require_human_review

3.5 OS-Level Patching (Immutable Image Rebuild)

For immutable OS (Talos, Flatcar, Bottlerocket):

1. Upstream publishes new image with security fixes
2. CI pipeline builds new machine image incorporating the update
3. Image is scanned (Trivy/vulnerability assessment)
4. Staged node replacement: drain → replace → verify, one node at a time
5. PodDisruptionBudgets ensure workload availability during rollout
6. Rollback: revert to previous image if node health checks fail

For traditional OS (management nodes):

1. Unattended-upgrades for security patches (automatic)
2. Ansible playbooks for coordinated upgrades
3. Snapshot before, apply, verify, rollback if failed
4. Kernel updates: scheduled maintenance window, human approval

3.6 Compliance Patch SLAs

Regulatory frameworks set expectations for patch timelines. These should be encoded as policy:

3.7 Human Gates (Non-Automatable)

• Database engine major version upgrades (schema compatibility)
• Kubernetes control plane upgrades (API deprecation review)
• Kernel updates on bare-metal with custom drivers
• Any patch that changes API contracts or data formats
• First-time patching of a new dependency (no historical data)

3.8 Evidence Capture

• SBOM for every deployed image (CycloneDX or SPDX)
• Vulnerability scan results at build time and runtime
• Signed image digests (cosign / Sigstore)
• Canary metrics during bake period
• Rollback events with reason codes
• Patch compliance dashboard (time-to-patch by severity)

4. Layer 3 – Security Automation and Autonomous Defense

4.1 Purpose

Detect, contain, and respond to security threats with minimal human latency for known attack patterns, while maintaining human oversight for novel threats and availability-impacting responses.

4.2 Defense-in-Depth Stack

Layer 6: Compliance-as-Code (OPA, Kyverno, Cedar)
         Continuous policy enforcement, admission control

Layer 5: SOAR (Tines, Shuffle, XSOAR)
         Playbook-driven automated response

Layer 4: SIEM / Correlation (Wazuh, Elastic SIEM)
         Event correlation, alert enrichment, threat intelligence

Layer 3: Runtime Security (Falco, Tetragon, Sysdig)
         Syscall monitoring, behavioural detection, eBPF

Layer 2: Network Security (Cilium, Calico, NP-as-code)
         Network policy, DNS filtering, egress control

Layer 1: Supply Chain (Trivy, cosign, SLSA, admission)
         Image signing, SBOM, vulnerability gates

Layer 0: Identity (Keycloak/OIDC, RBAC, SPIFFE/SPIRE)
         Zero-trust identity, workload identity, least privilege

4.3 Automated Response Playbooks

4.4 Policy-as-Code (Admission Control)

Prevent bad state from entering the cluster rather than detecting it after the fact.

Kubernetes admission:

• Kyverno or OPA/Gatekeeper for pod security standards
• Image signature verification (cosign + admission webhook)
• No privileged containers, no host networking (except explicit allowlist)
• Resource limits required on all pods
• Network policies required for all namespaces

Cloud-level:

• AWS SCP / Azure Policy / GCP Organization Policy for guardrails
• Terraform / OpenTofu with plan validation (OPA on plan output)
• No direct console changes – all changes through IaC pipeline

Runtime:

• Falco rules for syscall-level behavioural detection
• Tetragon / eBPF for kernel-level enforcement
• Wazuh for host-level integrity monitoring (FIM) and log analysis

4.5 Vulnerability Management Pipeline

Continuous Scanning
    ├── Registry scan (Trivy) — on push and scheduled
    ├── Runtime scan (Trivy operator) — running containers
    ├── Host scan (Wazuh) — OS and installed packages
    ├── IaC scan (Checkov, tfsec) — in CI pipeline
    └── Dependency scan (SCA) — in CI pipeline
            │
            ▼
    Prioritisation Engine
    ├── CVSS score
    ├── EPSS (Exploit Prediction Scoring)
    ├── CISA KEV (Known Exploited)
    ├── Reachability analysis (is the vuln actually reachable?)
    ├── Asset criticality (what does this run on?)
    └── Exposure context (internet-facing? internal?)
            │
            ▼
    Action routing
    ├── Critical + Exploited + Reachable → Emergency patch
    ├── High + Reachable → Fast-track to patching pipeline
    ├── Medium/Low or Not Reachable → Standard patching SLA
    └── Accepted risk → Document in risk register, review quarterly

4.6 Human Gates (Non-Automatable)

• Novel attack patterns requiring investigation
• Actions that would impact production availability (killing services, blocking IP ranges)
• Incident response decisions with legal/regulatory implications
• Threat intelligence assessment (is this a false positive or a real campaign?)
• Risk acceptance decisions (accepting a vulnerability that can’t be patched)
• Security architecture changes

4.7 Evidence Capture

• SIEM event logs with correlation IDs
• Automated response execution logs (SOAR audit trail)
• Forensic captures (container snapshots, memory dumps) for incidents
• Policy enforcement logs (admission webhook decisions)
• Vulnerability scan history and remediation timelines
• Image signatures and SBOM for deployed artifacts

5. Layer 4 – Compliance Automation and Continuous Assurance

5.1 Purpose

Maintain continuous compliance posture with automated evidence generation, policy enforcement, and drift detection, while preserving human accountability for risk decisions and regulatory attestation.

5.2 Compliance Automation Model

Compliance Control Plane

  Policy Engine      Evidence Store     Audit Dashboard
  (OPA/Kyverno/      (immutable,        (continuous
   Cedar)             signed,            posture)
                      timestamped)

            Control Mapping Layer

  ISO 27001 <-> NIS2 <-> SOC 2 <-> DORA <-> CIS <-> NIST

  Maps technical controls to regulatory requirements
  One control can satisfy multiple frameworks

5.3 Continuous Control Monitoring

5.4 Evidence Generation (OSCAL-Based)

OSCAL (Open Security Controls Assessment Language) provides a machine-readable format for compliance evidence:

1. System Security Plan (SSP): Generated from IaC and policy-as-code definitions
2. Assessment Plan: Automated test definitions mapped to controls
3. Assessment Results: Continuous scan and test results in OSCAL format
4. Plan of Action and Milestones (POA&M): Auto-generated from failed controls

Evidence pipeline:

Control check runs → Results stored (immutable, signed) →
Mapped to framework requirements → Dashboard updated →
Auditor accesses dashboard + evidence store

5.5 Regulatory Framework Requirements

NIS2 (applicable if operating in EU critical sectors):

• Risk management measures (Article 21): Automated control enforcement
• Incident reporting: 24-hour early warning, 72-hour notification, 1-month final report
• Supply chain security: SBOM, vendor assessment automation
• Management accountability: Human sign-off required – cannot be automated

DORA (applicable if in EU financial sector):

• ICT risk management framework: Continuous control monitoring
• Incident classification and reporting: 4-hour classification, then tiered reporting
• Threat-Led Penetration Testing (TLPT): Must be conducted – cannot be purely automated
• Third-party risk management: Contract-level controls, cannot be fully automated

SOC 2 / ISO 27001:

• Continuous control monitoring replaces point-in-time audits for many controls
• Management review and risk treatment decisions: Human gate
• Internal audit: Can be assisted by automation but requires human judgement
• Certification audit: External auditor, fully human process

5.6 GitSecOps: Git as the Source of Compliance Truth

The strongest pattern for demonstrating compliance to auditors:

• All infrastructure state in Git – signed commits, PR-based review
• All policy in Git – OPA/Kyverno policies version-controlled
• All changes traceable – commit → PR → CI → deploy → verify
• Immutable evidence – build logs, scan results, deployment events stored with integrity protection
• Automated mapping – technical controls mapped to regulatory requirements

This turns audit from “show me your documents” to “here is the commit history, the policy enforcement logs, and the continuous compliance dashboard.”

5.7 Human Gates (Structurally Required by Regulation)

These are not automation gaps. They are requirements imposed by every major compliance framework:

• Risk acceptance decisions: A named human must accept residual risk
• Management review: ISO 27001 Clause 9.3, NIS2 Article 20 – senior management must review
• Incident severity classification: Initial triage can be automated, final classification needs human judgement
• Vendor risk assessment: Questionnaires can be automated, risk decisions cannot
• Audit response: External auditors interact with humans
• Policy approval: Policy-as-code must be approved by a human before enforcement
• Exception management: Granting exceptions to policy requires documented human decision

5.8 Evidence Capture

• OSCAL-formatted assessment results
• Git commit history with signed commits (GPG/SSH)
• CI/CD pipeline execution logs
• Policy enforcement decision logs (admission webhook audit)
• Access review reports
• Incident timeline and response evidence
• Management review meeting minutes (human-generated)
• Risk register with decision trail

6. Layer 5 – Change Management and Autonomous Deployment

6.1 Purpose

Enable safe, fast, automated deployment of standard changes while maintaining rigorous gates for non-standard and emergency changes.

6.2 Change Classification

6.3 Standard Change Automation (L4)

Standard changes are the largest volume and the highest-value automation target. These are pre-approved change types where the blast radius is bounded and verification is automated.

Examples of standard changes:

• Application deployment (same architecture, same APIs, code changes only)
• Dependency patch (within auto-merge policy)
• Scaling adjustments (within defined limits)
• Feature flag toggle (within defined scope)
• Certificate rotation
• Configuration value change (within defined schema)

Pipeline:

Developer pushes code
        │
        ▼
  CI Pipeline (build, test, scan, sign)
        │ All gates pass
        ▼
  PR Auto-merge (if policy met)
        │
        ▼
  GitOps Sync (Flux/ArgoCD) → Canary Deploy (Argo Rollouts)
                                    │
                              SLO Verification
                              (bake time: 15–30 min)
                                    │
                              Pass? Promote
                              Fail? Rollback
        │
        ▼
  Progressive Rollout: 5% → 25% → 100%
  with SLO gates at each stage

6.4 Feature Flags (Decouple Deploy from Release)

Feature flags enable deploying code without activating it, then progressively enabling:

• Deploy: Code ships to production, feature is off
• Canary release: Feature enabled for 1–5% of traffic
• Progressive rollout: Gradual increase based on metrics
• Full release: Feature enabled for all traffic
• Kill switch: Instant disable without redeployment

Tools: LaunchDarkly, Flipt (self-hosted), Unleash, OpenFeature SDK

Integration with SLOs: Feature flags should be wired to SLO monitoring. If enabling a feature degrades SLOs beyond threshold, auto-disable.

6.5 Rollback Strategy

Every deployment must have a tested rollback path:

Critical rule: Never deploy a schema migration that cannot be rolled back, or a migration that requires the new code to function. Deploy migrations and code changes in separate steps (expand-contract pattern).

6.6 SLO-Gated Deployment

Every automated deployment should be gated on SLO health:

# Conceptual: Argo Rollouts analysis template
analysis:
  metrics:
    - name: error-rate
      provider: prometheus
      query: |
        sum(rate(http_requests_total{status=~"5.."}[5m]))
        / sum(rate(http_requests_total[5m]))
      threshold:
        max: 0.01  # 1% error rate
      interval: 60s

    - name: latency-p99
      provider: prometheus
      query: |
        histogram_quantile(0.99,
          sum(rate(http_request_duration_seconds_bucket[5m]))
          by (le))
      threshold:
        max: 0.5  # 500ms p99
      interval: 60s

    - name: success-rate
      provider: prometheus
      query: |
        sum(rate(http_requests_total{status=~"2.."}[5m]))
        / sum(rate(http_requests_total[5m]))
      threshold:
        min: 0.995  # 99.5% success rate
      interval: 60s

  # Rollback if ANY metric fails
  # Promote only if ALL metrics pass for full bake time

6.7 Human Gates

• Non-standard changes (new service, new dependency, architecture change)
• Emergency changes (abbreviated review, but still human-approved)
• Changes crossing compliance boundaries
• First deployment of a new service (no baseline metrics)
• Changes to the deployment pipeline itself

6.8 Evidence Capture

• Git PR history with reviews and approvals
• CI pipeline logs (build, test, scan results)
• Canary analysis results (metrics during bake period)
• Rollback events with trigger reason
• Feature flag change history
• Deployment timeline (start, promote, complete, or rollback)

7. Observability – The Nervous System

7.1 Purpose

Observability is the foundation that enables all other layers. Without high-quality telemetry, self-healing is guesswork, SLO gating is impossible, and compliance evidence is incomplete.

7.2 Observability Stack

Dashboards & Alerts: Grafana

  Metrics            Logs               Traces
  Prometheus /       Loki / OpenSearch   Tempo / Jaeger
  Mimir / Thanos     / Elastic

              OpenTelemetry Collector
              (unified pipeline)

  Applications      Infrastructure      Security
  (OTel SDK)        (node exporter,     (Falco,
                     kube-state)         Wazuh)

7.3 SLI/SLO Framework

Define SLIs and SLOs for every user-facing service:

Error budget: The difference between 100% and the SLO target is the error budget. Automation decisions consume error budget. If error budget is exhausted, freeze automated deployments until budget recovers.

7.4 Alert Design (Anti-Alert-Fatigue)

• Alert on SLO burn rate, not individual metrics
• Multi-window, multi-burn-rate alerts (fast burn = page, slow burn = ticket)
• Actionable alerts only: Every alert must have a runbook link and a defined action
• Silence expected noise: Planned maintenance, known conditions
• Review alert quality monthly: Track alert-to-incident ratio, false positive rate

7.5 Evidence Capture

• SLO compliance reports (monthly, quarterly)
• Error budget consumption history
• Alert history and response times
• Dashboard snapshots at time of incidents
• Telemetry retention proof (meeting regulatory requirements)

8. Agentic AI in Operations

AI agents are entering infrastructure operations: triaging alerts, suggesting remediations, and beginning to execute bounded actions autonomously. The engineering case is strong. The governance case requires deliberate architecture.

8.1 Current Maturity

8.2 Guardrails

AI agents in this architecture must operate within the same enforcement model as any other automated component:

• Scoped authority: Read-only by default. Write actions require policy-as-code approval, not prompt instructions.
• Structured evidence: Every action produces a signed record: trigger, context, decision, execution, outcome.
• Human gates at consequence boundaries: Actions that cross security, availability, or compliance thresholds require human approval.
• Circuit breakers: If the agent’s error rate or SLO impact exceeds thresholds, it halts automatically.

8.3 Adoption Path

Start at L1 (read-only advisory), build trust evidence over 3–6 months, then graduate to L2 (human-approved actions). Advance to L3 only for well-understood action classes with documented success rates. Do not skip phases – each builds the evidence needed to justify the next.

For a detailed treatment of guardrail architecture, evidence requirements, and the accountability gap in EU-regulated environments, see Agentic AI in Regulated Infrastructure.

9. Cross-Cutting Concerns

10. Implementation Roadmap

Phase 1: Foundation (Months 1–3)

Goal: Immutable base platform with GitOps and basic observability.

• Deploy Kubernetes (managed or Talos) with GitOps (Flux or ArgoCD)
• Implement pod security standards (Kyverno/OPA baseline)
• Deploy OpenTelemetry Collector + Prometheus + Grafana + Loki
• Define SLIs/SLOs for existing services
• Implement cert-manager with automated certificate rotation
• Establish Git repository structure for infrastructure state
• Configure node auto-repair and HPA/VPA

Phase 2: Patching and Supply Chain (Months 3–6)

Goal: Automated patching pipeline with compliance SLAs.

• Deploy Renovate/Dependabot with auto-merge policies
• Implement CI pipeline with SAST, SCA, container scanning
• Set up image signing (cosign/Sigstore) and admission verification
• Generate SBOMs for all deployed images
• Implement canary deployment (Argo Rollouts / Flagger)
• Define and enforce patch SLA policy
• Establish vulnerability prioritisation workflow

Phase 3: Security Automation (Months 6–9)

Goal: Automated detection and response for known threat patterns.

• Deploy Falco/Tetragon for runtime security
• Deploy Wazuh for host-level monitoring
• Implement SOAR playbooks for top 10 threat patterns
• Configure network policies as code (default-deny)
• Implement egress filtering
• Deploy SIEM correlation rules
• Establish incident response automation (containment playbooks)

Phase 4: Compliance Automation (Months 9–12)

Goal: Continuous compliance posture with automated evidence.

• Map technical controls to regulatory frameworks (NIS2/DORA/SOC2/ISO27001)
• Implement OSCAL-based evidence generation
• Deploy continuous compliance dashboard
• Automate access reviews
• Establish risk register with automated findings intake
• Conduct first automated compliance assessment
• Schedule management review (human gate)

Phase 5: Advanced Autonomy (Months 12+)

Goal: Expand automation envelope with AI-assisted operations.

• Deploy AI-assisted alert triage (read-only, advisory)
• Implement SLO-based error budget policies
• Expand SOAR playbooks based on incident data
• Evaluate AI agent tools for supervised remediation
• Refine auto-merge policies based on 6+ months of data
• Conduct autonomy level assessment for each component
• Document trust evidence for current autonomy levels

11. Anti-Patterns

12. Decision Log

Appendix A: Tool Reference

Appendix B: Regulatory Quick Reference