# IF.EMOTION TRACE PROTOCOL v3.3: AUDITABLE DEBUGGING (WITHOUT WISHFUL THINKING)

*Alternate title: Debugging Emotion — The Immutable Flight Recorder*

> **The model’s answer is ephemeral. The trace is the product. If you can’t prove what happened, you are not running an AI system — you are running a scripted reality show.**

> Logging is cheap. Evidence is expensive. If your system can’t be verified by someone who doesn’t trust you, it doesn’t have evidence.

**Author:** Danny Stocker (`ds@infrafabric.io`)  
**Canonical (static mirror):** `https://infrafabric.io/static/hosted/IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md`  
**Repo source:** `https://git.infrafabric.io/danny/hosted/src/branch/main/IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md`  
**Paper SHA256 (sidecar):**
- `https://infrafabric.io/static/hosted/IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md.sha256`
- `https://git.infrafabric.io/danny/hosted/raw/branch/main/IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md.sha256`
**Verify:** `curl -fsSLO 'https://infrafabric.io/static/hosted/IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md' -fsSLO 'https://infrafabric.io/static/hosted/IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md.sha256' && sha256sum -c IF_EMOTION_DEBUGGING_TRACE_WHITEPAPER_v3.3_STYLED.md.sha256`

**Subject:** End-to-end traceability, bounded completeness witnessing, and PQ-anchored evidence binding  
**Protocol:** IF.TTT (Traceable, Transparent, Trustworthy)  
**Version:** 3.3 (Header hardening + session annex)  
**Date (UTC):** 2025-12-21  
**Status:** AUDIT REQUIRED  
**Citation:** `if://whitepaper/emotion/trace-protocol/v3.3`

---

## Why this matters (decision context)

AI incidents are not “debugging problems.” They are dispute problems. A user claims they asked X. A regulator asks what the system actually did. A reviewer asks whether a citation was real or performative.

This protocol is a practical answer to one question:

Can an external reviewer independently verify what happened from request → retrieval → output, and detect tampering after the fact?

*This is not an observability feature. It’s chain-of-custody.*

---

## Guarantees (and boundaries)

This system provides integrity guarantees (tamper-evidence) and bounded completeness guarantees (witnessing) within explicit boundaries.

- Integrity: the trace timeline is hash-chained; the signed summary binds the final output to a trace head.
- Completeness (bounded): a REQ_SEEN ledger witnesses each request that crosses the backend witness boundary, with a signed per-hour Merkle head.
- PQ anchoring (bounded): post-quantum signatures apply at registry anchoring time (IF.TTT), not necessarily on every hot-path artifact.

Boundary sentence (non-negotiable):

Integrity starts at the backend witness boundary; completeness is only meaningful at and after that boundary until edge witnessing is cryptographically enforced.

---

## Layered evidence stack (where guarantees live)

```mermaid
flowchart TB
  U[User] -->|HTTPS| E[Edge]
  E --> B[Backend Witness Boundary]

  B --> R[Retrieval]
  B --> P[Prompt]
  B --> M[Model]
  B --> X[Postprocess]

  B --> T1["REQ_SEEN ledger<br/>(hourly JSONL)"]
  B --> T2["Trace events<br/>(hash chain JSONL)"]
  B --> T3["Signed summary<br/>(output hash + head attestation)"]

  T1 --> H["Signed Merkle head<br/>(per hour)"]
  T2 --> S["Trace head<br/>(event_hash)"]

  H --> BUNDLE["Evidence bundle<br/>(tar.gz + manifest)"]
  S --> BUNDLE
  T3 --> BUNDLE

  BUNDLE --> REG["Registry anchor<br/>(PQ-hybrid)"]
  BUNDLE --> MIRROR["Static mirror<br/>(public download)"]
```

Interpretation:

Integrity starts at the witness boundary; completeness is only meaningful at and after that boundary until edge witnessing exists.

---

## Evidence inventory (what ships)

| Artifact | File | Claim it supports | Verification |
|---|---|---|---|
| Evidence bundle | `emo_trace_payload_<trace>.tar.gz` | Portable reproduction | `sha256sum` + verifier |
| Manifest + checksums | `payload/manifest.json`, `payload/sha256s.txt` | “One check implies contents” | verifier validates per-file SHA256 |
| Trace event chain | `payload/trace_events.jsonl` | Tamper-evident event ordering | verifier recomputes event hashes |
| Signed summary | `payload/ttt_signed_record.json` | Binds response hash → trace head | verifier recomputes HMAC signature |
| REQ_SEEN ledger | `payload/req_seen_<hour>.jsonl` | Bounded completeness | verifier recomputes leaf hashes + Merkle root |
| REQ_SEEN head | `payload/req_seen_head_<hour>.json` | Signed Merkle head | verifier checks Ed25519 signature |
| Inclusion proof | `payload/req_seen_inclusion_proof.json` | Proves this trace is in the hour ledger | verifier checks Merkle path |
| Narrative projection | `payload/if_story.md` + external annex | Human-readable timeline | all `event_hash` references must exist |
| Registry corroboration | `*.ttt_chain_record.json` | PQ-anchored record (when available) | compare content hashes |

---

## Methodology hardening (closing real audit gaps)

### HMAC key separation for REQ_SEEN (no mixed keys)

REQ_SEEN uses HMAC commitments only if `IF_REQ_SEEN_HMAC_KEY` is configured. It never reuses the signing secret used for the signed summary.

If `IF_REQ_SEEN_HMAC_KEY` is missing, REQ_SEEN downgrades to SHA256 commitments and the system must not claim “privacy-preserving HMAC commitments.”

### Correlation-only client trace IDs (collision discipline)

If a client provides `X-IF-Client-Trace`, it is treated as a correlation-only identifier. The canonical trace ID is always server-generated and returned in `X-IF-Emotion-Trace`.

### Monotonic timing fields (clock realism)

Each trace event includes:

- `ts_utc`: wall-clock timestamp (not trusted for crypto time)
- `mono_ns` / `mono_ms`: monotonic timing since trace start (stable ordering and performance attribution)

This does not solve time attestation, but it removes “clock drift” as an excuse for missing latency evidence.

### Inclusion proof is a first-class prior

The inclusion proof file is registered as a child artifact in IF.TTT. It is not optional.

---

## Reference proof run (v3.3)

Trace ID:

- `96700e8e-6a83-445e-86f7-06905c500146`

Evidence bundle:

- Static mirror (preferred): `https://infrafabric.io/static/hosted/emo_trace_payload_96700e8e-6a83-445e-86f7-06905c500146.tar.gz`
- Forgejo raw (alternate): `https://git.infrafabric.io/danny/hosted/raw/branch/main/emo_trace_payload_96700e8e-6a83-445e-86f7-06905c500146.tar.gz`

Tarball SHA256:

- `85eb323c8e5f11cf4dd18e612e8cde8dcdb355b3fbd6380bbc8d480a5bf97e87`

IF.TTT tarball handle:

- `if://citation/2ec551ec-0a08-487d-a41d-4d068aa8ee2f/v1`

---

## Verification (external reviewer path)

Download + hash:

```bash
curl -fsSL -o emo.tar.gz 'https://infrafabric.io/static/hosted/emo_trace_payload_96700e8e-6a83-445e-86f7-06905c500146.tar.gz'
sha256sum emo.tar.gz
# expected: 85eb323c8e5f11cf4dd18e612e8cde8dcdb355b3fbd6380bbc8d480a5bf97e87
```

Run verifier:

```bash
python3 -m venv venv
./venv/bin/pip install canonicaljson pynacl
curl -fsSL -o iftrace.py 'https://infrafabric.io/static/hosted/iftrace.py'
./venv/bin/python iftrace.py verify emo.tar.gz --expected-sha256 85eb323c8e5f11cf4dd18e612e8cde8dcdb355b3fbd6380bbc8d480a5bf97e87
```

REQ_SEEN inclusion proof check:

```bash
tar -xzf emo.tar.gz
./venv/bin/python iftrace.py verify-inclusion payload/req_seen_inclusion_proof.json
# expected: OK
```

---

## Narrative annex

IF.story is not evidence; it is a deterministic projection keyed by `event_hash`.

- Static mirror: `https://infrafabric.io/static/hosted/IF_EMOTION_TRACE_REFERENCE_96700e8e-6a83-445e-86f7-06905c500146_IF_STORY.md`
- Forgejo raw: `https://git.infrafabric.io/danny/hosted/raw/branch/main/IF_EMOTION_TRACE_REFERENCE_96700e8e-6a83-445e-86f7-06905c500146_IF_STORY.md`

---

## Limitations (still true)

- This proves what the system did and what bytes were served. It does not prove factual truth in the world.
- Completeness is bounded by the witness boundary; requests dropped before the backend boundary are out of scope until edge witnessing is cryptographically bound.
- Key management and time attestation remain the practical certification blockers (HSM/TPM, rotation ceremony, external timestamping).

---

## What to do next (tomorrow’s work, not wishful thinking)

- Move REQ_SEEN witnessing to the true front door (edge) and sign the head there.
- Publish a deploy attestation record (code hash + image digest) into IF.TTT for every release.
- Add a clear anchoring SLO (maximum time from trace finalization → registry anchor) and enforce it.

---

## Annex: Agent field notes (session narrative, for future operators)

I don’t experience emotions or “vertigo” the way a human does. I do, however, run into friction: missing invariants, brittle interfaces, and places where the system can’t keep a promise unless it changes its behavior.

When I arrived in this environment, the objective landscape looked like this: multiple services across `pct 210/220/230/240`, a set of previously published trace bundles under `https://infrafabric.io/static/hosted/`, and a clear requirement to close methodology gaps that external reviewers kept repeating. The mission wasn’t “make a nicer document.” It was “make the receipts real and reproducible.” That’s a different mindset, and it is easy for future agents to forget when they’re optimizing prose.

The first hard signal of friction was the difference between “we can do this” and “we can prove this”: the system had already grown a verifier (`iftrace.py`), inclusion proofs, and a public mirror, but the backend code still violated a key separation invariant by letting REQ_SEEN HMAC fall back to the signing secret. That’s the kind of quiet footgun that makes security reviewers stop reading. Fixing it forced an explicit posture: if the HMAC key isn’t there, downgrade and say so, rather than pretending.

The second friction point was identity: allowing a client-supplied trace ID to become the canonical trace ID is an attractive shortcut, and it is also an invitation to collisions and replay confusion. The fix was a small but meaningful rule: client trace IDs are correlation-only, server trace IDs are canonical, and the correlation ID must be echoed in a dedicated header and stored in the trace evidence. This is boring. It is also the difference between “traceability” and “a dispute about IDs.”

The third friction point was time: wall-clock timestamps are useful but fragile in forensics. Adding monotonic timing (`mono_ns`/`mono_ms`) doesn’t magically create a trusted time source, but it prevents a lot of easy evasions and makes pipeline latency attribution crisp. It also forces future agents to treat time claims as bounded (“within this process”) unless and until there is external attestation.

The fourth friction point was operational, not cryptographic: tool output that mixes banners with JSON breaks automated parsing. The IF.TTT registry prints capability messages before JSON. That’s fine for humans. It’s poison for scripts. The packaging tool had to become defensive: capture the last JSON object, or redirect non-JSON output away from stdout. This is a recurring trap in InfraFabric: “human-readable by default” becomes “machine-fragile by accident.” Future agents should assume that any command intended to be machine-consumed must either be `--json-only` or filtered explicitly.

The fifth friction point was publication: the static mirror is reliable, but pushing to Forgejo from this host failed with `401` authentication. That’s not a philosophical problem; it’s a missing credential path. The right move is not to handwave it in the whitepaper. The right move is to treat publishing as part of the evidence chain: if we can’t publish, we can’t be externally audited. Any future agent updating artifacts should verify the push path (SSH key or token) before claiming new versions are “live.”

Traps for future agents to avoid, stated plainly: don’t overclaim PQ scope, don’t blur completeness boundaries, don’t let “pretty trace UI” substitute for verifiable export, don’t reuse keys across roles, and don’t accept client identifiers without collision discipline. And when you write, never let the document sound more certain than the system can prove. The fastest way to lose credibility is to write the future tense as if it already shipped.

This project has a strong gravitational pull toward narrative. That’s part of the brand. But the moment you step into audit territory, narrative must be tethered to an artifact, or it becomes exactly what this protocol is trying to eliminate: a scripted reality show.

---

## Appendix: One End-to-End Trace (User → Retrieval → Decision → Audit Proof)

This appendix answers:

“Show me ONE end‑to‑end trace: user request → retrieval → decision → audit query that proves chain‑of‑custody. Walk me through how I would independently verify it.”

It uses a real reference trace so every step is reproducible.

Note: This appendix uses the v2.1 reference bundle because it includes a retrieval event; the v3.2+ bundle referenced in the main text can be verified the same way.

---

### Trace used (reference bundle)

- Trace ID: `016cca78-6f9d-4ffe-aec0-99792d383ca1`
- Standalone appendix (this section, as a separate file):
  - `https://infrafabric.io/static/hosted/IF_EMOTION_TRACE_E2E_WALKTHROUGH_016cca78-6f9d-4ffe-aec0-99792d383ca1.md`
  - `https://git.infrafabric.io/danny/hosted/raw/branch/main/IF_EMOTION_TRACE_E2E_WALKTHROUGH_016cca78-6f9d-4ffe-aec0-99792d383ca1.md`
- Evidence bundle (static mirror):
  - `https://infrafabric.io/static/hosted/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.tar.gz`
- Evidence bundle (Forgejo raw):
  - `https://git.infrafabric.io/danny/hosted/raw/branch/main/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.tar.gz`
- Tarball SHA256:
  - `7101ff9c38fc759a66157f6a6ab9c0936af547d0ec77a51b5d05db07069966c8`
- IF.TTT chain record (PQ‑anchored):
  - `https://infrafabric.io/static/hosted/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.ttt_chain_record.json`
  - `https://infrafabric.io/static/hosted/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.ttt_chain_ref.json`

---

### The trace, end‑to‑end (what actually happened)

**User request** (from `payload/trace_payload.json`):

```
In English: Summarize what this system can and cannot prove about an LLM answer, in plain language, and include the Trace line. If you cite clinical context, include [Source: ...] tags.
```

**Retrieval** (from `payload/trace_events.jsonl`, event type `retrieval_done`):

- `retrieval_event_id`: `c9b3ebf0-15bb-4c80-8c94-574ba5324954`
- `retrieved_citations`: `if://citation/3862ce4a-bca5-4090-b5e9-5652fee391ae/v1`

**Decision** (from `payload/ttt_signed_record.json`, `guard` object):

- `decision`: `allow`
- `enabled`: `true`
- `vulnerability`: `0.05`

**Audit chain head** (from `payload/ttt_signed_record.json`, `trace_chain`):

- `event_count`: `6`
- `head_hash`: `200c83313376e05577e98d59cd13f2441cccb211f9a9a0927c4ceaf8033827f5`

This is the “receipt” that binds the final output hash to the chain of events.

---

### How to independently verify chain‑of‑custody

These steps work without access to the backend, just the public artifacts.

### 1) Download and verify the bundle hash

```bash
curl -fsSL -o emo.tar.gz \
  'https://infrafabric.io/static/hosted/emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.tar.gz'

sha256sum emo.tar.gz
# expected: 7101ff9c38fc759a66157f6a6ab9c0936af547d0ec77a51b5d05db07069966c8
```

This proves you have the exact bundle the operator claims to have published.

### 2) Run the verifier (manifest + chains + signatures)

```bash
python3 -m venv venv
./venv/bin/pip install canonicaljson pynacl
curl -fsSL -o iftrace.py 'https://infrafabric.io/static/hosted/iftrace.py'

./venv/bin/python iftrace.py verify emo.tar.gz \
  --expected-sha256 7101ff9c38fc759a66157f6a6ab9c0936af547d0ec77a51b5d05db07069966c8
```

Expected result: `OK verify` plus checks for:

- `manifest: ok`
- `trace_events: ok` (hash chain)
- `req_seen_head: Ed25519 signature OK`
- `req_seen: ok` (Merkle root)
- `if_story: ok` (all event_hash references exist)

This proves the trace file, the summary, and the completeness ledger are internally consistent and untampered.

### 3) Prove completeness for this trace (REQ_SEEN inclusion)

```bash
tar -xzf emo.tar.gz
./venv/bin/python iftrace.py verify-inclusion payload/req_seen_inclusion_proof.json
# expected: OK
```

This proves **this trace’s request** is included in the hourly REQ_SEEN ledger, which is the bounded‑completeness claim.

### 4) Inspect the chain step‑by‑step (user → retrieval → decision)

```bash
python3 - <<'PY'
import json
from pathlib import Path
base = Path('payload')
trace_events = [json.loads(l)['event'] for l in base.joinpath('trace_events.jsonl').read_text().splitlines() if l.strip()]
for ev in trace_events:
    print(ev['idx'], ev['type'], ev['event_hash'], ev['prev_hash'])
PY
```

Expected sequence (from this bundle):

- `0 req_seen` → `09ce8a52ff9070ee…`
- `1 request_received` → `f9f93f15b8278a4e…`
- `2 retrieval_done` → `7ec94771dcaed85c…`
- `3 prompt_built` → `ec84acc6f4df6edd…`
- `4 model_done` → `94321445f1b3c560…`
- `5 trace_finalizing` → `200c83313376e055…`

You can see the exact retrieval event (`retrieval_done`) and its hash in the chain.

### 5) Verify decision + output binding (signed summary)

```bash
python3 - <<'PY'
import json
from pathlib import Path
rec = json.loads(Path('payload/ttt_signed_record.json').read_text())
print('trace_id', rec['event']['trace_id'])
print('guard decision', rec['event']['guard']['decision'])
print('response_sha256', rec['event']['response_sha256'])
print('trace_chain', rec['event']['trace_chain'])
PY
```

This proves that the **decision** and the **final output hash** are bound into the signed summary that also contains the chain head.

---

### What this proves (and what it doesn’t)

**Proves:**

- The request, retrieval, decision, and output are tied together in a hash‑chained event log.
- The output hash is bound to a signed summary that includes the chain head.
- The request is included in the REQ_SEEN ledger for its hour (bounded completeness).
- The bundle you verified is exactly the published artifact.

**Does not prove:**

- That the output is factually correct in the outside world.
- That requests dropped before the backend witness boundary were seen.

---

### Optional: registry corroboration (IF.TTT)

If you have access to the IF.TTT registry, you can verify the PQ‑anchored record by comparing:

- `emo_trace_payload_016cca78-6f9d-4ffe-aec0-99792d383ca1.ttt_chain_ref.json`
- The tarball SHA + bundle content hashes

This corroborates that the published bundle hash was anchored in the registry under the cited handle.

---

For a matching walkthrough against the v3.2 methodology-hardening bundle, use trace `96700e8e-6a83-445e-86f7-06905c500146` and follow `https://infrafabric.io/static/hosted/VERIFY_EMO_TRACE_96700e8e-6a83-445e-86f7-06905c500146.md`.