Add v3.3 trace protocol paper

IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md

@@ -0,0 +1,214 @@
+# IF.EMOTION TRACE PROTOCOL v3.3: AUDITABLE DEBUGGING (WITHOUT WISHFUL THINKING)
+
+*Alternate title: Debugging Emotion — The Immutable Flight Recorder*
+
+> **The model’s answer is ephemeral. The trace is the product. If you can’t prove what happened, you are not running an AI system — you are running a scripted reality show.**
+
+> Logging is cheap. Evidence is expensive. If your system can’t be verified by someone who doesn’t trust you, it doesn’t have evidence.
+
+**Subject:** End-to-end traceability, bounded completeness witnessing, and PQ-anchored evidence binding  
+**Protocol:** IF.TTT (Traceable, Transparent, Trustworthy)  
+**Version:** 3.3 (Header hardening + session annex)  
+**Date (UTC):** 2025-12-21  
+**Status:** AUDIT REQUIRED  
+**Citation:** `if://whitepaper/emotion/trace-protocol/v3.3`
+
+---
+
+## Why this matters (decision context)
+
+AI incidents are not “debugging problems.” They are dispute problems. A user claims they asked X. A regulator asks what the system actually did. A reviewer asks whether a citation was real or performative.
+
+This protocol is a practical answer to one question:
+
+Can an external reviewer independently verify what happened from request → retrieval → output, and detect tampering after the fact?
+
+*This is not an observability feature. It’s chain-of-custody.*
+
+---
+
+## Guarantees (and boundaries)
+
+This system provides integrity guarantees (tamper-evidence) and bounded completeness guarantees (witnessing) within explicit boundaries.
+
+- Integrity: the trace timeline is hash-chained; the signed summary binds the final output to a trace head.
+- Completeness (bounded): a REQ_SEEN ledger witnesses each request that crosses the backend witness boundary, with a signed per-hour Merkle head.
+- PQ anchoring (bounded): post-quantum signatures apply at registry anchoring time (IF.TTT), not necessarily on every hot-path artifact.
+
+Boundary sentence (non-negotiable):
+
+Integrity starts at the backend witness boundary; completeness is only meaningful at and after that boundary until edge witnessing is cryptographically enforced.
+
+---
+
+## Layered evidence stack (where guarantees live)
+
+```mermaid
+flowchart TB
+  U[User] -->|HTTPS| E[Edge]
+  E --> B[Backend Witness Boundary]
+
+  B --> R[Retrieval]
+  B --> P[Prompt]
+  B --> M[Model]
+  B --> X[Postprocess]
+
+  B --> T1["REQ_SEEN ledger<br/>(hourly JSONL)"]
+  B --> T2["Trace events<br/>(hash chain JSONL)"]
+  B --> T3["Signed summary<br/>(output hash + head attestation)"]
+
+  T1 --> H["Signed Merkle head<br/>(per hour)"]
+  T2 --> S["Trace head<br/>(event_hash)"]
+
+  H --> BUNDLE["Evidence bundle<br/>(tar.gz + manifest)"]
+  S --> BUNDLE
+  T3 --> BUNDLE
+
+  BUNDLE --> REG["Registry anchor<br/>(PQ-hybrid)"]
+  BUNDLE --> MIRROR["Static mirror<br/>(public download)"]
+```
+
+Interpretation:
+
+Integrity starts at the witness boundary; completeness is only meaningful at and after that boundary until edge witnessing exists.
+
+---
+
+## Evidence inventory (what ships)
+
+| Artifact | File | Claim it supports | Verification |
+|---|---|---|---|
+| Evidence bundle | `emo_trace_payload_<trace>.tar.gz` | Portable reproduction | `sha256sum` + verifier |
+| Manifest + checksums | `payload/manifest.json`, `payload/sha256s.txt` | “One check implies contents” | verifier validates per-file SHA256 |
+| Trace event chain | `payload/trace_events.jsonl` | Tamper-evident event ordering | verifier recomputes event hashes |
+| Signed summary | `payload/ttt_signed_record.json` | Binds response hash → trace head | verifier recomputes HMAC signature |
+| REQ_SEEN ledger | `payload/req_seen_<hour>.jsonl` | Bounded completeness | verifier recomputes leaf hashes + Merkle root |
+| REQ_SEEN head | `payload/req_seen_head_<hour>.json` | Signed Merkle head | verifier checks Ed25519 signature |
+| Inclusion proof | `payload/req_seen_inclusion_proof.json` | Proves this trace is in the hour ledger | verifier checks Merkle path |
+| Narrative projection | `payload/if_story.md` + external annex | Human-readable timeline | all `event_hash` references must exist |
+| Registry corroboration | `*.ttt_chain_record.json` | PQ-anchored record (when available) | compare content hashes |
+
+---
+
+## Methodology hardening (closing real audit gaps)
+
+### HMAC key separation for REQ_SEEN (no mixed keys)
+
+REQ_SEEN uses HMAC commitments only if `IF_REQ_SEEN_HMAC_KEY` is configured. It never reuses the signing secret used for the signed summary.
+
+If `IF_REQ_SEEN_HMAC_KEY` is missing, REQ_SEEN downgrades to SHA256 commitments and the system must not claim “privacy-preserving HMAC commitments.”
+
+### Correlation-only client trace IDs (collision discipline)
+
+If a client provides `X-IF-Client-Trace`, it is treated as a correlation-only identifier. The canonical trace ID is always server-generated and returned in `X-IF-Emotion-Trace`.
+
+### Monotonic timing fields (clock realism)
+
+Each trace event includes:
+
+- `ts_utc`: wall-clock timestamp (not trusted for crypto time)
+- `mono_ns` / `mono_ms`: monotonic timing since trace start (stable ordering and performance attribution)
+
+This does not solve time attestation, but it removes “clock drift” as an excuse for missing latency evidence.
+
+### Inclusion proof is a first-class prior
+
+The inclusion proof file is registered as a child artifact in IF.TTT. It is not optional.
+
+---
+
+## Reference proof run (v3.3)
+
+Trace ID:
+
+- `96700e8e-6a83-445e-86f7-06905c500146`
+
+Evidence bundle:
+
+- Static mirror (preferred): `https://infrafabric.io/static/hosted/emo_trace_payload_96700e8e-6a83-445e-86f7-06905c500146.tar.gz`
+- Forgejo raw (alternate): `https://git.infrafabric.io/danny/hosted/raw/branch/main/emo_trace_payload_96700e8e-6a83-445e-86f7-06905c500146.tar.gz`
+
+Tarball SHA256:
+
+- `85eb323c8e5f11cf4dd18e612e8cde8dcdb355b3fbd6380bbc8d480a5bf97e87`
+
+IF.TTT tarball handle:
+
+- `if://citation/2ec551ec-0a08-487d-a41d-4d068aa8ee2f/v1`
+
+---
+
+## Verification (external reviewer path)
+
+Download + hash:
+
+```bash
+curl -fsSL -o emo.tar.gz 'https://infrafabric.io/static/hosted/emo_trace_payload_96700e8e-6a83-445e-86f7-06905c500146.tar.gz'
+sha256sum emo.tar.gz
+# expected: 85eb323c8e5f11cf4dd18e612e8cde8dcdb355b3fbd6380bbc8d480a5bf97e87
+```
+
+Run verifier:
+
+```bash
+python3 -m venv venv
+./venv/bin/pip install canonicaljson pynacl
+curl -fsSL -o iftrace.py 'https://infrafabric.io/static/hosted/iftrace.py'
+./venv/bin/python iftrace.py verify emo.tar.gz --expected-sha256 85eb323c8e5f11cf4dd18e612e8cde8dcdb355b3fbd6380bbc8d480a5bf97e87
+```
+
+REQ_SEEN inclusion proof check:
+
+```bash
+tar -xzf emo.tar.gz
+./venv/bin/python iftrace.py verify-inclusion payload/req_seen_inclusion_proof.json
+# expected: OK
+```
+
+---
+
+## Narrative annex
+
+IF.story is not evidence; it is a deterministic projection keyed by `event_hash`.
+
+- Static mirror: `https://infrafabric.io/static/hosted/IF_EMOTION_TRACE_REFERENCE_96700e8e-6a83-445e-86f7-06905c500146_IF_STORY.md`
+- Forgejo raw: `https://git.infrafabric.io/danny/hosted/raw/branch/main/IF_EMOTION_TRACE_REFERENCE_96700e8e-6a83-445e-86f7-06905c500146_IF_STORY.md`
+
+---
+
+## Limitations (still true)
+
+- This proves what the system did and what bytes were served. It does not prove factual truth in the world.
+- Completeness is bounded by the witness boundary; requests dropped before the backend boundary are out of scope until edge witnessing is cryptographically bound.
+- Key management and time attestation remain the practical certification blockers (HSM/TPM, rotation ceremony, external timestamping).
+
+---
+
+## What to do next (tomorrow’s work, not wishful thinking)
+
+- Move REQ_SEEN witnessing to the true front door (edge) and sign the head there.
+- Publish a deploy attestation record (code hash + image digest) into IF.TTT for every release.
+- Add a clear anchoring SLO (maximum time from trace finalization → registry anchor) and enforce it.
+
+---
+
+## Annex: Agent field notes (session narrative, for future operators)
+
+I don’t experience emotions or “vertigo” the way a human does. I do, however, run into friction: missing invariants, brittle interfaces, and places where the system can’t keep a promise unless it changes its behavior.
+
+When I arrived in this environment, the objective landscape looked like this: multiple services across `pct 210/220/230/240`, a set of previously published trace bundles under `https://infrafabric.io/static/hosted/`, and a clear requirement to close methodology gaps that external reviewers kept repeating. The mission wasn’t “make a nicer document.” It was “make the receipts real and reproducible.” That’s a different mindset, and it is easy for future agents to forget when they’re optimizing prose.
+
+The first hard signal of friction was the difference between “we can do this” and “we can prove this”: the system had already grown a verifier (`iftrace.py`), inclusion proofs, and a public mirror, but the backend code still violated a key separation invariant by letting REQ_SEEN HMAC fall back to the signing secret. That’s the kind of quiet footgun that makes security reviewers stop reading. Fixing it forced an explicit posture: if the HMAC key isn’t there, downgrade and say so, rather than pretending.
+
+The second friction point was identity: allowing a client-supplied trace ID to become the canonical trace ID is an attractive shortcut, and it is also an invitation to collisions and replay confusion. The fix was a small but meaningful rule: client trace IDs are correlation-only, server trace IDs are canonical, and the correlation ID must be echoed in a dedicated header and stored in the trace evidence. This is boring. It is also the difference between “traceability” and “a dispute about IDs.”
+
+The third friction point was time: wall-clock timestamps are useful but fragile in forensics. Adding monotonic timing (`mono_ns`/`mono_ms`) doesn’t magically create a trusted time source, but it prevents a lot of easy evasions and makes pipeline latency attribution crisp. It also forces future agents to treat time claims as bounded (“within this process”) unless and until there is external attestation.
+
+The fourth friction point was operational, not cryptographic: tool output that mixes banners with JSON breaks automated parsing. The IF.TTT registry prints capability messages before JSON. That’s fine for humans. It’s poison for scripts. The packaging tool had to become defensive: capture the last JSON object, or redirect non-JSON output away from stdout. This is a recurring trap in InfraFabric: “human-readable by default” becomes “machine-fragile by accident.” Future agents should assume that any command intended to be machine-consumed must either be `--json-only` or filtered explicitly.
+
+The fifth friction point was publication: the static mirror is reliable, but pushing to Forgejo from this host failed with `401` authentication. That’s not a philosophical problem; it’s a missing credential path. The right move is not to handwave it in the whitepaper. The right move is to treat publishing as part of the evidence chain: if we can’t publish, we can’t be externally audited. Any future agent updating artifacts should verify the push path (SSH key or token) before claiming new versions are “live.”
+
+Traps for future agents to avoid, stated plainly: don’t overclaim PQ scope, don’t blur completeness boundaries, don’t let “pretty trace UI” substitute for verifiable export, don’t reuse keys across roles, and don’t accept client identifiers without collision discipline. And when you write, never let the document sound more certain than the system can prove. The fastest way to lose credibility is to write the future tense as if it already shipped.
+
+This project has a strong gravitational pull toward narrative. That’s part of the brand. But the moment you step into audit territory, narrative must be tethered to an artifact, or it becomes exactly what this protocol is trying to eliminate: a scripted reality show.
+

IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md.sha256

@@ -0,0 +1 @@
+b9ffca798eaa7730d88a4ec8eb25c16d899e1de28a4b6ccb39dbb35b1ea5de9d  /root/tmp/hosted_repo_update/IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md

README.md

@@ -53,3 +53,8 @@ Static hosted artifacts used in InfraFabric reviews.
 - File: `iftrace.py` (run with a Python venv that has `canonicaljson` + `pynacl`)
 - Verify tarball: `python iftrace.py verify emo_trace_payload_<trace>.tar.gz --expected-sha256 <sha>`
 - Prove inclusion: `python iftrace.py prove-inclusion --ledger req_seen_<hour>.jsonl --head req_seen_head_<hour>.json --trace-id <uuid>`
+
+## IF.emotion trace whitepaper (styled v3.3)
+
+- File: `IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md`
+