diff --git a/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md b/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md new file mode 100644 index 0000000..9e3b09a --- /dev/null +++ b/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md @@ -0,0 +1,156 @@ +# IF.TTT Failure Mode Analysis v1 (Why traceability is not optional) + +**Author:** Danny Stocker (`ds@infrafabric.io`) +**Canonical (static mirror):** `https://infrafabric.io/static/hosted/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md` +**Repo source:** `https://git.infrafabric.io/danny/hosted/src/branch/main/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md` +**SHA256 (sidecar):** +- `https://infrafabric.io/static/hosted/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md.sha256` +- `https://git.infrafabric.io/danny/hosted/raw/branch/main/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md.sha256` + +**Subject:** Failure mode analysis of removing cryptographic chain-of-custody from an AI governance stack +**Protocol:** IF.SEC.threat.model.v1 +**Status:** FORENSIC ASSESSMENT (not evidence; not certification) +**Date (UTC):** 2025-12-21 + +--- + +This document answers one question: + +What actually breaks when you replace IF.TTT (Traceable/Transparent/Trustworthy) with “standard logs”? + +This is not hypothetical theatre. This failure pattern has already harmed people at scale. + +--- + +## 1) The compromise (how it really happens) + +An organization removes **cryptographic signatures + immutable audit chaining** and replaces it with mutable JSON/text logging (Splunk/Datadog/etc). + +The rationales are not “2ms latency”. The rationales are organizational: + +- “Compliance says monthly audit exports are sufficient.” +- “We don’t want key management in the hot path.” +- “We can’t afford per-request signing costs at current volume.” +- “The new CTO comes from a startup background; they optimize for speed, not forensics.” + +The shift is simple: + +- From **proven identity** (key-based, non-repudiable) +- To **asserted identity** (string-based, deniable) + +--- + +## 2) Immediate effects (technical) + +Removing cryptographic binding causes immediate epistemic collapse: + +1) **Identity spoofing** +Any actor can write `{"actor":"admin"}` into a log stream. Without signature verification, “admin” is a string, not a proof. + +2) **Repudiation becomes cheap** +An operator can deny actions. The system has no mathematical contradiction to present. + +3) **Post-hoc editing becomes indistinguishable from “cleanup”** +Even honest operators edit logs during incident response (redaction, dedupe, “fix formatting”). Without a hash chain, edits are invisible. + +4) **Time becomes an argument, not evidence** +Wall-clock timestamps are not cryptographic time. A system clock can drift or be altered. If the log is mutable, time is now a narrative. + +--- + +## 3) Cascade failures (systemic) + +IF.TTT is the load-bearing wall. When it fails, upstream components lose their ability to prove they acted correctly. + +- **IF.GUARD failure** +An actor bypasses governance and injects a fake “Approved” record later. Without non-repudiable signatures and an immutable chain, the council cannot prove they *did not* approve it. + +- **IF.emotion corruption** +Retrieval context can be poisoned (bad document inserted) and the retrieval log can be edited to look “clean”. The system produces harmful output; the investigation cannot recover the root cause. + +- **Swarm communications collapse** +Inter-agent trust degrades: a message “from Swarm B” is no longer provably from Swarm B. Man-in-the-middle and replay confusion become operationally plausible. + +--- + +## 4) Real incident scenario (how you lose in court) + +**The “hot patch” liability event** + +1) An LLM system is integrated into a support workflow (medical, legal, crisis, finance). +2) Under pressure, an engineer hot-patches a prompt/config (“make it more helpful”). +3) The system produces harmful guidance. +4) A user captures the harmful output (screenshot, screen recording, saved HTML, exported conversation). +5) During investigation, the engineer “cleans up” the deployment log line (panic, shame, or misguided privacy redaction). + +Outcome: + +- The organization produces logs claiming the system used safe configuration. +- The plaintiff produces the harmful output. +- The discrepancy becomes **indefensible** because the chain-of-custody is missing. + +In high-liability domains, “we have logs” is not a defense. It is a liability amplifier. + +--- + +## 5) Historical precedents (pattern, not analogy) + +These are different domains. The shared mechanism is always the same: **mutable records + privileged access + institutional denial**. + +- **UK Post Office Horizon scandal** — forensic dispute about ledger truth and privileged system access. + - Background: `https://en.wikipedia.org/wiki/British_Post_Office_scandal` + +- **Boeing 737 MAX / MCAS** — catastrophic outcomes coupled to opaque system behavior and disputed control/alerting assumptions. + - Background: `https://en.wikipedia.org/wiki/Boeing_737_MAX_groundings` + +- **Wells Fargo fake accounts scandal** — incentive pressure plus record-keeping failures and downstream denial dynamics. + - Background: `https://en.wikipedia.org/wiki/Wells_Fargo_account_fraud_scandal` + +- **Volkswagen emissions scandal** — engineered “compliance theatre” in software, with auditors unable to trust vendor assertions. + - Background: `https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal` + +IF.TTT is designed to prevent the *mechanism*, not to claim these incidents had identical root causes. + +--- + +## 6) Defense validation (what IF.TTT prevents) + +IF.TTT prevents the “mutable record” class of failure by enforcing: + +1) **Non-repudiation** +Actions are signed by keys. “Who did it” becomes a proof, not a debate. + +2) **Tamper-evident history** +Events are hash-chained; deletions/edits break verification. + +3) **Bounded completeness (within boundary)** +Requests that cross the witness boundary are witnessed in REQ_SEEN with inclusion proofs. + +4) **Portable evidence** +Incidents can be exported as bundles with manifests + verifier commands, enabling third-party review without trusting the operator. + +This turns disputes into verification. + +--- + +## 7) Residual risks (what IF.TTT does not magically solve) + +Even with IF.TTT deployed, real risks remain: + +1) **Key compromise** +If a private key is stolen, an attacker can sign bad actions that verify as “valid”. Mitigation requires hardware-backed keys, rotation, and incident response. + +2) **Rubber-stamp governance** +The audit trail can be perfect while the decision is wrong (fatigue, negligence, incentives). Mitigation requires IF.GUARD diversity + escalation rules + post-incident review. + +3) **Boundary gaps** +Events dropped before the witness boundary (edge, network) may be outside scope unless edge witnessing is enforced. + +--- + +## Conclusion + +Removing IF.TTT turns the system from **high-assurance intelligence** into a **black-box liability generator**. + +The cost of “not having receipts” is not measured in milliseconds. It is measured in ruined investigations, regulatory exposure, and human harm. + diff --git a/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md.sha256 b/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md.sha256 new file mode 100644 index 0000000..991cfb1 --- /dev/null +++ b/IF_TTT_FAILURE_MODE_ANALYSIS_v1.md.sha256 @@ -0,0 +1 @@ +9b78ba5394e2cd85cde2fa0bfd8941bafc593d58593f942d5d2f11d857058c19 IF_TTT_FAILURE_MODE_ANALYSIS_v1.md