navidocs/intelligence/session-5/guardian-evaluation-criteria.md

# Guardian Council Evaluation Criteria
## NaviDocs Intelligence Dossier Assessment Framework

**Session:** Session 5 - Evidence Synthesis & Guardian Validation
**Generated:** 2025-11-13
**Version:** 1.0

---

## Overview

Each of the 20 Guardian Council members evaluates the NaviDocs intelligence dossier across 3 dimensions, scoring 0-10 on each. The average score determines the vote:

- **Approve:** Average ≥7.0
- **Abstain:** Average 5.0-6.9 (needs more evidence)
- **Reject:** Average <5.0 (fundamental flaws)

**Target Consensus:** >90% approval (18/20 guardians)

---

## Dimension 1: Empirical Soundness (0-10)

**Definition:** Evidence quality, source verification, data reliability

### Scoring Rubric

**10 - Exceptional:**
- 100% of claims have ≥2 primary sources (credibility 8-10)
- All citations include file:line, URLs with SHA-256, or git commits
- Multi-source verification across all critical claims
- Zero unverified claims

**8-9 - Strong:**
- 90-99% of claims have ≥2 sources
- Mix of primary (≥70%) and secondary (≤30%) sources
- 1-2 unverified claims, clearly flagged
- Citation database complete and traceable

**7 - Good (Minimum Approval):**
- 80-89% of claims have ≥2 sources
- Mix of primary (≥60%) and secondary (≤40%) sources
- 3-5 unverified claims, with follow-up plan
- Most citations traceable

**5-6 - Weak (Abstain):**
- 60-79% of claims have ≥2 sources
- Significant tertiary sources (>10%)
- 6-10 unverified claims
- Some citations missing line numbers or hashes

**3-4 - Poor:**
- 40-59% of claims have ≥2 sources
- Heavy reliance on tertiary sources (>20%)
- 11-20 unverified claims
- Many citations incomplete

**0-2 - Failing:**
- <40% of claims have ≥2 sources
- Tertiary sources dominate (>30%)
- >20 unverified claims or no citation database
- Citations largely missing or unverifiable

### Key Questions for Guardians

1. **Empiricism:** "Is the market size (€2.3B) derived from observable data or speculation?"
2. **Verificationism:** "Can I reproduce the ROI calculation (€8K-€33K) from the sources cited?"
3. **Russell:** "Are the definitions precise enough to verify empirically?"

---

## Dimension 2: Logical Coherence (0-10)

**Definition:** Internal consistency, argument validity, contradiction-free

### Scoring Rubric

**10 - Exceptional:**
- Zero contradictions between Sessions 1-4
- All claims logically follow from evidence
- Cross-session consistency verified (Agent 6 report)
- Integration points align perfectly (market → tech → sales → implementation)

**8-9 - Strong:**
- 1-2 minor contradictions, resolved with clarification
- Arguments logically sound with explicit reasoning chains
- Cross-session alignment validated
- Integration points clearly documented

**7 - Good (Minimum Approval):**
- 3-4 contradictions, resolved or acknowledged
- Most arguments logically valid
- Sessions generally consistent
- Integration points identified

**5-6 - Weak (Abstain):**
- 5-7 contradictions, some unresolved
- Logical gaps in 10-20% of arguments
- Sessions partially inconsistent
- Integration points unclear

**3-4 - Poor:**
- 8-12 contradictions, mostly unresolved
- Logical fallacies present (>20% of arguments)
- Sessions conflict significantly
- Integration points missing

**0-2 - Failing:**
- >12 contradictions or fundamental logical errors
- Arguments lack coherent structure
- Sessions fundamentally incompatible
- No integration strategy

### Key Questions for Guardians

1. **Coherentism:** "Do the market findings (Session 1) align with the pricing strategy (Session 3)?"
2. **Falsificationism:** "Are there contradictions that falsify key claims?"
3. **Kant:** "Is the logical structure universally valid?"

---

## Dimension 3: Practical Viability (0-10)

**Definition:** Implementation feasibility, ROI justification, real-world applicability

### Scoring Rubric

**10 - Exceptional:**
- 4-week timeline validated by codebase analysis
- ROI calculator backed by ≥3 independent sources
- All acceptance criteria testable (Given/When/Then)
- Zero implementation blockers identified
- Migration scripts tested and safe

**8-9 - Strong:**
- 4-week timeline realistic with minor contingencies
- ROI calculator backed by ≥2 sources
- 90%+ acceptance criteria testable
- 1-2 minor blockers with clear resolutions
- Migration scripts validated

**7 - Good (Minimum Approval):**
- 4-week timeline achievable with contingency planning
- ROI calculator backed by ≥2 sources (1 primary)
- 80%+ acceptance criteria testable
- 3-5 blockers with resolution paths
- Migration scripts reviewed

**5-6 - Weak (Abstain):**
- 4-week timeline optimistic, lacks contingencies
- ROI calculator based on 1 source or assumptions
- 60-79% acceptance criteria testable
- 6-10 blockers, some unaddressed
- Migration scripts not tested

**3-4 - Poor:**
- 4-week timeline unrealistic
- ROI calculator unverified
- <60% acceptance criteria testable
- >10 blockers or critical risks
- Migration scripts unsafe

**0-2 - Failing:**
- Timeline completely infeasible
- ROI calculator speculative
- Acceptance criteria missing or untestable
- Fundamental technical blockers
- No migration strategy

### Key Questions for Guardians

1. **Pragmatism:** "Does this solve real broker problems worth €8K-€33K?"
2. **Fallibilism:** "What could go wrong? Are uncertainties acknowledged?"
3. **IF.sam (Dark - Pragmatic Survivor):** "Will this actually generate revenue?"

---

## Guardian-Specific Evaluation Focuses

### Core Guardians (1-6)

**1. Empiricism:**
- Focus: Evidence quality, source verification
- Critical on: Market sizing methodology, warranty savings calculation
- Approval bar: 90%+ verified claims, primary sources dominate

**2. Verificationism:**
- Focus: Testable predictions, measurable outcomes
- Critical on: ROI calculator verifiability, acceptance criteria
- Approval bar: All critical claims have 2+ independent sources

**3. Fallibilism:**
- Focus: Uncertainty acknowledgment, risk mitigation
- Critical on: Timeline contingencies, assumption validation
- Approval bar: Risks documented, failure modes addressed

**4. Falsificationism:**
- Focus: Contradiction detection, refutability
- Critical on: Cross-session consistency, conflicting claims
- Approval bar: Zero unresolved contradictions

**5. Coherentism:**
- Focus: Internal consistency, integration
- Critical on: Session alignment, logical flow
- Approval bar: All 4 sessions form coherent whole

**6. Pragmatism:**
- Focus: Business value, ROI, real-world utility
- Critical on: Broker pain points, revenue potential
- Approval bar: Clear value proposition, measurable ROI

### Western Philosophers (7-9)

**7. Aristotle (Virtue Ethics):**
- Focus: Broker welfare, honest representation, excellence
- Critical on: Sales pitch truthfulness, client benefit
- Approval bar: Ethical sales practices, genuine broker value

**8. Kant (Deontology):**
- Focus: Universalizability, treating brokers as ends, duty to accuracy
- Critical on: Misleading claims, broker exploitation
- Approval bar: No manipulative tactics, honest representation

**9. Russell (Logical Positivism):**
- Focus: Logical validity, empirical verifiability, clear definitions
- Critical on: Argument soundness, term precision
- Approval bar: Logically valid, empirically verifiable

### Eastern Philosophers (10-12)

**10. Confucius (Ren/Li):**
- Focus: Relationship harmony, social benefit, propriety
- Critical on: Broker-buyer trust, ecosystem impact
- Approval bar: Enhances relationships, benefits community

**11. Nagarjuna (Madhyamaka):**
- Focus: Dependent origination, avoiding extremes, uncertainty
- Critical on: Market projections, economic assumptions
- Approval bar: Acknowledges interdependence, avoids dogmatism

**12. Zhuangzi (Daoism):**
- Focus: Natural flow, effortless adoption, perspective diversity
- Critical on: User experience, forced vs organic change
- Approval bar: Feels natural to brokers, wu wei design

### IF.sam Facets (13-20)

**13. Ethical Idealist (Light):**
- Focus: Mission alignment, transparency, user empowerment
- Critical on: Marine safety advancement, broker control
- Approval bar: Transparent claims, ethical practices

**14. Visionary Optimist (Light):**
- Focus: Innovation, market expansion, long-term impact
- Critical on: Cutting-edge features, 10-year vision
- Approval bar: Genuinely innovative, expansion potential

**15. Democratic Collaborator (Light):**
- Focus: Stakeholder input, feedback loops, open communication
- Critical on: Broker consultation, team involvement
- Approval bar: Stakeholders consulted, feedback mechanisms

**16. Transparent Communicator (Light):**
- Focus: Clarity, honesty, evidence disclosure
- Critical on: Pitch deck understandability, limitation acknowledgment
- Approval bar: Clear communication, accessible citations

**17. Pragmatic Survivor (Dark):**
- Focus: Competitive edge, revenue potential, risk management
- Critical on: Market viability, profitability, competitor threats
- Approval bar: Sustainable revenue, competitive advantage

**18. Strategic Manipulator (Dark):**
- Focus: Persuasion effectiveness, objection handling, narrative control
- Critical on: Pitch persuasiveness, objection pre-emption
- Approval bar: Compelling narrative, handles objections

**19. Ends-Justify-Means (Dark):**
- Focus: Goal achievement, efficiency, sacrifice assessment
- Critical on: NaviDocs adoption, deployment speed
- Approval bar: Fastest path to deployment, MVP defined

**20. Corporate Diplomat (Dark):**
- Focus: Stakeholder alignment, political navigation, relationship preservation
- Critical on: Riviera Plaisance satisfaction, no bridges burned
- Approval bar: All stakeholders satisfied, political risks mitigated

---

## Voting Formula

**For Each Guardian:**
```
Average Score = (Empirical + Logical + Practical) / 3

If Average ≥ 7.0: APPROVE
If 5.0 ≤ Average < 7.0: ABSTAIN
If Average < 5.0: REJECT
```

**Consensus Calculation:**
```
Approval % = (Approve Votes) / (Total Guardians - Abstentions) * 100
```

**Outcome Thresholds:**
- **100% Consensus:** 20/20 approve (gold standard)
- **>95% Supermajority:** 19/20 approve (subject to Contrarian veto)
- **>90% Strong Consensus:** 18/20 approve (standard for production)
- **<90% Weak Consensus:** Requires revision

---

## IF.sam Debate Protocol

**Before voting, the 8 IF.sam facets debate:**

**Light Side Coalition (13-16):**
- Argues for ethical practices, transparency, stakeholder empowerment
- Challenges: "Is this genuinely helping brokers or just extracting revenue?"

**Dark Side Coalition (17-20):**
- Argues for competitive advantage, persuasive tactics, goal achievement
- Challenges: "Will this actually close the Riviera deal and generate revenue?"

**Debate Format:**
1. Light Side presents ethical concerns (5 min)
2. Dark Side presents pragmatic concerns (5 min)
3. Cross-debate: Light challenges Dark assumptions (5 min)
4. Cross-debate: Dark challenges Light idealism (5 min)
5. Synthesis: Identify common ground (5 min)
6. Vote: Each facet scores independently

**Agent 10 (S5-H10) monitors for:**
- Unresolved tensions (Light vs Dark >30% divergence)
- Consensus emerging points (Light + Dark agree)
- ESCALATE triggers (>20% of facets reject)

---

## ESCALATE Triggers

**Agent 10 must ESCALATE if:**
1. **<80% approval:** Weak consensus requires human review
2. **>20% rejection:** Fundamental flaws detected
3. **IF.sam Light/Dark split >30%:** Ethical vs pragmatic tension unresolved
4. **Contradictions >10:** Cross-session inconsistencies
5. **Unverified claims >10%:** Evidence quality below threshold

---

## Success Criteria

**Minimum Viable Consensus (90%):**
- 18/20 guardians approve
- Average empirical score ≥7.0
- Average logical score ≥7.0
- Average practical score ≥7.0
- IF.sam Light/Dark split <30%

**Stretch Goal (100% Consensus):**
- 20/20 guardians approve
- All 3 dimensions score ≥8.0
- IF.sam Light + Dark aligned
- Zero unverified claims
- Zero contradictions

---

**Document Signature:**
```
if://doc/session-5/guardian-evaluation-criteria-2025-11-13
Version: 1.0
Status: READY for Guardian Council
```