# IF.EMOTION TRACE PROTOCOL v3.1: AUDITABLE DEBUGGING (WITHOUT WISHFUL THINKING) *Alternate title: Debugging Emotion — The Immutable Flight Recorder* **Subject:** End-to-End Traceability, Completeness Witnessing, and PQ-Anchored Evidence Binding **Protocol:** IF.TTT (Traceable, Transparent, Trustworthy) **Version:** 3.1 (Embedded Narrative Annex) **Date:** 2025-12-21 **Status:** AUDIT REQUIRED **Citation:** `if://whitepaper/emotion/trace-protocol/v3.1` --- > Logging is cheap. Evidence is expensive. If your system can’t be verified by someone who doesn’t trust you, it doesn’t have evidence. ## Why This Matters (Decision Context) AI incidents are not “debugging problems.” They are dispute problems. If you cannot prove what the system received, what it did, what it returned, and what evidence it claims to have used, you don’t have an incident process. You have screenshots and opinions. *The goal is not truth. The goal is provenance: what was recorded, when, by which code, under which keys, and whether something was omitted within an explicit boundary.* --- ## Layered Evidence Stack (Where Guarantees Begin) ```mermaid flowchart TB U[User Browser] -->|HTTPS| E[Edge: TLS + Routing] E -->|/oauth2/*| O[oauth2-proxy] E -->|/api/*| N[nginx] N --> B[Backend Witness Boundary] B --> R[Retrieval / RAG] B --> P[Prompt Build] B --> M[Model Inference] B --> X[Postprocess: citations + trace footer] N --> EDGELOG["Edge attempt log
(JSONL)"] B --> T1["REQ_SEEN ledger
(hourly JSONL)"] B --> T2["Trace events
(hash chain JSONL)"] B --> T3["Signed summary
(output hash + head attestation)"] T1 --> H["Signed Merkle head
(per hour)"] T2 --> S["Trace head
(event_hash)"] H --> BUNDLE["Evidence bundle
(tar.gz + manifest)"] S --> BUNDLE T3 --> BUNDLE EDGELOG --> BUNDLE BUNDLE --> REG["IF.TTT registry
(PQ-hybrid anchor)"] BUNDLE --> MIRROR["Public static mirror
/static/hosted"] ``` **Interpretation:** integrity begins at the backend witness boundary. Completeness is only meaningful at and after that boundary until edge witnessing is fully cryptographic. --- ## What This Protocol Guarantees (And What It Refuses To Claim) This system does not try to “prove the model is true.” That is not a meaningful claim for probabilistic generation. This system proves something narrower and more valuable: 1) what the system received (as a commitment), 2) what the system did (trace event chain), 3) what the system returned (output hash), 4) what evidence it claims to have used (retrieval IDs + citation handles), 5) that the resulting artifacts are tamper-evident and portable for external review. If a claim cannot be bound to an artifact, it does not exist. --- ## Evidence Inventory (What Exists, How You Verify It) | Artifact | What it proves | Verification path | |---|---|---| | Edge attempt log | request attempts above the backend boundary | inspect JSONL + correlate request id + trace id + body commitment headers | | REQ_SEEN ledger | witnessed attempts within backend boundary | Merkle inclusion proof against signed head | | REQ_SEEN signed head | integrity of a witnessed batch | verify signature + Merkle root | | trace event chain | ordered sequence of system actions | recompute chain (`prev_hash` → `event_hash`) | | signed summary | binding between output + trace head + policy decisions | verify signature + output hash | | evidence bundle | portable capsule of the above | verify SHA256 + run verifier tool | | IF.TTT record | anchored registration (PQ-hybrid) | verify registry chain record + signature | --- ## The Actual Failure Mode The problem isn’t hallucination. The problem is un-auditable systems pretending to be auditable. Most teams treat tracing as “extra logs.” In probabilistic systems, tracing is the product: it is the only thing that turns a dispute into a verification. --- ## Why Teams Don’t Ship This **The resistance:** “This is too much work; we’ll add observability later.” **The psychology:** Shipping dopamine beats audit discipline, and nobody gets promoted for preventing future disputes. **The unstick:** Make “no trace, no trust” a deployment gate. If it can’t be verified by an adversarial reviewer, it doesn’t ship. --- ## Core Protocol (Technical Specification) ### Trace ID contract Every request to `/api/chat/completions` receives a Trace ID. This includes denials. - Client request header (optional): `X-IF-Client-Trace: ` (if present, backend uses it as Trace ID) - Response header: `X-IF-Emotion-Trace: ` - Response header: `X-IF-Emotion-Trace-Sig: ` (app-level integrity) - User output: final line `Trace: ` ### Body commitment (tamper-evident “what I sent”) The client may send: - `X-IF-Body-Sha256: sha256=` (or ``) The backend computes `sha256(request_body_bytes)` and records: - `request_body_sha256` (computed), - `claimed_body_sha256` (from header), - `body_commit_ok` (match flag). This provides a commitment without storing full request bodies in witness layers. ### Edge attempt logging (interim completeness above backend) Because auth/rate limits can deny requests before the backend can write REQ_SEEN, nginx writes an edge attempt log: - File: `/var/log/nginx/edge_req_seen.jsonl` - Captures: `req_id`, `status`, `client_trace`, `body_sha256` (header), timing, and (if available) `auth_email`. This is a pragmatic bridge. It is not yet a cryptographic edge ledger. ### Backend completeness (REQ_SEEN) and boundary REQ_SEEN records every request attempt that reaches the backend witness boundary as a privacy-preserving commitment: - Ledger: `/opt/if-emotion/data/req_seen/.jsonl` - Signed head: `/opt/if-emotion/data/req_seen/heads/.json` REQ_SEEN completeness is only valid for requests that reach the backend process. Requests blocked before the backend are out of scope until a cryptographic witness is moved to the edge. ### Privacy for witnessed prompts (avoid rainbow tables) REQ_SEEN stores a keyed commitment by default: - `user_text_hash_alg: hmac-sha256` - `user_text_hash: hmac_sha256(secret, user_text_bytes)` - `user_text_sha256_prefix: ` (debug-only correlation) Key source: - `IF_REQ_SEEN_HMAC_KEY` if configured, otherwise falls back to `IF_TTT_SIGNING_SECRET`. ### Trace events and signed summary - Trace events: `/opt/if-emotion/data/trace_events.jsonl` (hash chain) - Signed summary: `/opt/if-emotion/data/ttt_signed_log.jsonl` - Optional payload capture (OAuth-gated): `/opt/if-emotion/data/trace_payloads/.json` Signed completion records include `code_identity` (code SHA + config hash) so “which code ran?” is auditable even without remote attestation. ### Post-quantum boundary Evidence bundles are PQ-hybrid signed when registered into IF.TTT: - `pq_status: hybrid-fips204` - `pq_algo: ML-DSA-87` Hot-path signatures (Ed25519) are not post-quantum. The correct claim is “PQ-anchored at registry time,” not “PQ everywhere.” --- ## Verifier Tooling (Public) Primary verifier (static mirror): - `https://infrafabric.io/static/hosted/iftrace.py` Primary verifier (Forgejo raw): - `https://git.infrafabric.io/danny/hosted/raw/branch/main/iftrace.py` Runbook (static mirror): - `https://infrafabric.io/static/hosted/VERIFY_EMO_TRACE_016cca78-6f9d-4ffe-aec0-99792d383ca1.md` Runbook (Forgejo raw): - `https://git.infrafabric.io/danny/hosted/raw/branch/main/VERIFY_EMO_TRACE_016cca78-6f9d-4ffe-aec0-99792d383ca1.md` --- ## Reference Proof Run (Transport + Integrity) Bundle (static mirror): - `https://infrafabric.io/static/hosted/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.tar.gz` Bundle (Forgejo raw): - `https://git.infrafabric.io/danny/hosted/raw/branch/main/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.tar.gz` Directory (discovery): - `https://infrafabric.io/static/hosted/` SHA256: - `7101ff9c38fc759a66157f6a6ab9c0936af547d0ec77a51b5d05db07069966c8` One-line verification: - `curl -fsSL https://infrafabric.io/static/hosted/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.tar.gz | sha256sum` Expected output: - `7101ff9c38fc759a66157f6a6ab9c0936af547d0ec77a51b5d05db07069966c8 -` IF.TTT citation handle (PQ-hybrid signed): - `if://citation/c24fe95e-226c-4efc-ba22-5ddcc37ff7d2/v1` --- ## Narrative Annex (Embedded) This section is a deterministic narrative projection for external review. It is not evidence. - Ground truth: `payload/trace_events.jsonl` + `payload/ttt_signed_record.json` inside the bundle. - Purpose: give humans a readable index keyed by `event_hash`. Reference trace: - Trace: `016cca78-6f9d-4ffe-aec0-99792d383ca1` Deterministic narrative projection of `trace_events.jsonl`. Each line includes the `event_hash` anchor. - 2025-12-21T07:58:32Z | `req_seen` | REQ_SEEN witnessed; hour=20251221T07 count=2 merkle_root=41aa7aebd66d3b1199b94d5c531111526744254dc8aad1f079e035a725c58aa2 | event_hash=09ce8a52ff9070ee00e0510365107a579d4b8be0f8beb4071aada966a51ed282 - 2025-12-21T07:58:32Z | `request_received` | Auth+quota succeeded; provider=codex model=gpt-5.2 stream=False user_len=185 auth_ms=3 | event_hash=f9f93f15b8278a4e6ef32e53d2ae02202cd15826ec9bfe90ce10e7bca7bd3b8d - 2025-12-21T07:58:33Z | `retrieval_done` | Retrieval done; retrieved_count=1 rag_ms=1107 retrieval_event_id=c9b3ebf0-15bb-4c80-8c94-574ba5324954 | event_hash=7ec94771dcaed85c5ab6bbc0d69c14b78010bbac9c3d76fe1e1234e005cbbdb4 - 2025-12-21T07:58:33Z | `prompt_built` | Prompt built; prompt_sha256=9e4783b70020296fed1f938b12123413552b9c4150574b2d79b9740b440aef20 | event_hash=ec84acc6f4df6edd1ded939be588fc670dc786d8507826cc6f74096c1dbfab16 - 2025-12-21T07:58:44Z | `model_done` | Model done; provider=codex model=gpt-5.2 llm_ms=10550 | event_hash=94321445f1b3c5601390b22a0369d62794da425bc467c99f7f80eec7b68b1268 - 2025-12-21T07:58:44Z | `trace_finalizing` | Trace finalizing; ok=True provider=codex | event_hash=200c83313376e05577e98d59cd13f2441cccb211f9a9a0927c4ceaf8033827f5 Separate annex artifact (static mirror): - `https://infrafabric.io/static/hosted/IF_EMOTION_TRACE_REFERENCE_016cca78_IF_STORY.md` Separate annex artifact (Forgejo raw): - `https://git.infrafabric.io/danny/hosted/raw/branch/main/IF_EMOTION_TRACE_REFERENCE_016cca78_IF_STORY.md` --- > The model’s answer is ephemeral. The trace is the product. If you can’t prove what happened, you are not running an AI system — you are running a story.