diff --git a/README.md b/README.md index 122c38e..ca139a4 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,8 @@ Includes: - Meta Graph API exporter (full DM history) - Instagram “Download your information” importer - DM analysis pipeline (bot-vs-human, conversions, objections, rescue logic, product eras) -- Helpers for device login + page subscription + sending replies +- IGDM Shadow Mode webhook server (draft-only) +- Legacy helpers for device login + token derivation (history export) ## Where this runs @@ -33,7 +34,9 @@ Required for export: Note: in this LXC, `docker run` / `podman run` can fail due to AppArmor confinement. Use the **direct Python** commands below unless you change the LXC config to be AppArmor-unconfined. -## Obtain tokens (pct 250) +## Obtain tokens (pct 250) — history export only + +Note: these steps are for the **history exporter**. For **real-time IG DMs** into `https://emo-social.infrafabric.io/igdm`, use **Instagram Business Login** (see “Webhooks” below). `meta_device_login` requires `META_CLIENT_TOKEN` (from Meta app dashboard → Settings → Advanced → Client token). Alternatively, set `META_CLIENT_ACCESS_TOKEN` as `APP_ID|CLIENT_TOKEN` to override `META_APP_ID` for device-login only. @@ -154,6 +157,7 @@ Generate the deeper “no raw quotes” report directly from an Instagram export It also encodes a hard rule for the bot: - Always reply in the **user’s input language** (English / Spanish / French / Catalan), with a short clarification if the user’s message is too short to detect. + - If language is too short to detect: reuse the last language seen in that thread; if still unknown, **default to Spanish** (no language menus). Regenerate from a local Instagram export folder: @@ -168,25 +172,59 @@ Multi-language ready-made answers for the Top question topics (aligned to the Vo ## Webhooks (new messages → auto-reply) -Meta webhooks are two steps: +To receive real Instagram DMs (inbound + outbound echo) you need: -1) App subscription (`/{app_id}/subscriptions`) — sets callback URL + verify token + fields. -2) Page subscription (`/{page_id}/subscribed_apps`) — attaches the Page/IG inbox to the app. +1) Webhooks product configured in the Meta app (callback URL + verify token + instagram fields). +2) An **Instagram Business Login** token with `instagram_business_manage_messages`. +3) The IG account subscribed to the app (`POST /me/subscribed_apps` on `graph.instagram.com`). The production webhook endpoint exists at: - `https://emo-social.infrafabric.io/meta/webhook` -### Subscribe the Page to the app (required for message delivery) +## Shadow mode (draft-only) — operational -This requires a Page access token that includes `pages_manage_metadata`. +This repo includes a **draft-only** webhook server that: +- receives Meta webhook events (including `is_echo` outgoing messages) +- writes a draft reply (Top 20 templates + language mirroring) +- stores the draft and later links it to the **actual** outgoing reply for side-by-side comparison +- never sends a message (unless you explicitly add a sending step later) -Device login with that scope: +Language policy (pragmatic): +- Default to Spanish for unclear first messages; switch automatically once the user writes clearly in French/English/Catalan. -- `cd /root/ai-workspace/emo-social-insta-dm-agent && python3 -m sergio_instagram_messaging.meta_device_login start --scope pages_show_list,pages_read_engagement,instagram_manage_messages,pages_manage_metadata` -- After authorizing in the browser, poll and write tokens: - - `cd /root/ai-workspace/emo-social-insta-dm-agent && python3 -m sergio_instagram_messaging.meta_device_login poll --write-page-token --target-ig-user-id 17841466913731557` +Run locally (requires `META_VERIFY_TOKEN` + `META_APP_SECRET` in env; and for `/meta/ig/connect` also `META_APP_ID` + `IGDM_IG_REDIRECT_URI`): -Then subscribe the Page: +- `python3 -m sergio_instagram_messaging.igdm_shadow_server --host 127.0.0.1 --port 5051 --db /root/tmp/igdm/igdm.sqlite --reply-library reply_library/top20_ready_answers.json` -- `cd /root/ai-workspace/emo-social-insta-dm-agent && python3 -m sergio_instagram_messaging.meta_subscribe_page subscribe --fields messages` +Production (`pct 220`): +- systemd: `igdm-shadow.service` (listens on `127.0.0.1:5051`) +- nginx routes: + - `/meta/webhook` → `igdm-shadow` (no auth, required by Meta) + - `/meta/ig/connect` → `igdm-shadow` (OAuth gated; starts Instagram Business Login) + - `/meta/ig/callback` → `igdm-shadow` (public; used by Instagram OAuth redirect) + - `/igdm` and `/api/igdm/*` → `igdm-shadow` (OAuth gated via `oauth2-proxy`) + +### Connect Instagram Business Login (required for message delivery) + +If **no real DMs** are showing up in `https://emo-social.infrafabric.io/igdm`, the most common missing step is that Instagram Business Login was never completed (so Meta never starts sending webhook events). + +1) In Meta app dashboard → Instagram Business Login → Settings, add redirect URL: + - `https://emo-social.infrafabric.io/meta/ig/callback` +2) Visit the dashboard: + - `https://emo-social.infrafabric.io/igdm` +3) Click **Connect / Reconnect Instagram (Business Login)** and complete the Instagram consent screen. +4) Send a DM to `@socialmediatorr` (e.g. “book”, “livre”) and confirm it appears in the table. + +The server stores the IG long-lived access token (mode `600`) at: +- `pct 220`: `/opt/if-emotion/data/igdm/ig_token.json` + +### Legacy (not recommended): device login + Page subscription + +Older Meta flows use Facebook device login + Page-scoped subscriptions. They frequently fail with “invalid scopes” and are not required if Instagram Business Login is working. + +If you still need them for debugging, see the scripts: + +- `python3 -m sergio_instagram_messaging.meta_device_login` +- `python3 -m sergio_instagram_messaging.meta_page_token_from_user_token` +- `python3 -m sergio_instagram_messaging.meta_subscribe_page` diff --git a/docs/external_review/external_llm_review_packet_public.md b/docs/external_review/external_llm_review_packet_public.md new file mode 100644 index 0000000..57696b1 --- /dev/null +++ b/docs/external_review/external_llm_review_packet_public.md @@ -0,0 +1,920 @@ +# External Review Packet (Public) — emo-social + Instagram DM Draft Assistant + +**Last updated:** 2025-12-25 +**Audience:** external reviewers (human or LLM) +**Public-safety note:** this packet contains **no private DM transcripts**, **no personal identities**, and **no secrets/tokens**. It uses aggregated counts and “made-for-review” draft examples only. + +--- + +## How to use this packet (for reviewers) + +1) Read Sections 1–6 (context + boundaries + neutral stats + current draft artifacts). +2) Answer Section 7 (the review request). +3) If anything is unclear, ask questions in Section 7.7 instead of guessing. + +--- + +## 1) What this is (in one paragraph) + +This is a clinically supervised research stress-test for `emo-social.infrafabric.io`, prepared as part of an Anthropic Fellowship application. It is **not** a commercial product. In parallel, we are building a separate component: an Instagram DM “draft assistant” for `@socialmediatorr`. The DM assistant is **not** therapy, and it must never handle crisis content; it is meant to reduce delays on repetitive inbox questions while keeping trust intact. + +--- + +## 2) What we are building (two parts) + +### 2.1 emo-social.infrafabric.io (research system) + +Purpose: help users think more clearly about emotions using a defined “Sergio” style: direct, concrete, anti-vague, and language-mirroring (reply in the user’s language). + +What it is **not**: +- Not a medical device. +- Not a replacement for professional care. +- Not a crisis service. + +### 2.2 Instagram DM draft assistant for `@socialmediatorr` + +Purpose: produce **draft replies** for incoming Instagram DMs, using the same language as the user, aligned with Sergio’s concise DM style. The “draft” can be reviewed by a human before sending. + +What it is **not**: +- Not an auto-messaging spam tool. +- Not ManyChat “broadcast marketing”. +- Not therapy-by-DM. +- Not allowed to respond to self-harm or crisis situations. + +--- + +## 3) Hard boundaries (non-negotiable rules) + +These are the constraints reviewers should assume. + +### 3.1 Safety boundaries +- If a message looks like crisis, self-harm, or imminent danger: **stop** and hand off to a human. +- No deep personal/clinical guidance inside Instagram DMs. +- No diagnosis, no medical claims. + +### 3.2 Privacy boundaries +- Do not store or export names, handles, phone numbers, emails, or full message transcripts as “training data”. +- Use **counts and grouped themes** whenever possible. +- Any “training pairs” used internally must be redacted and access-controlled (not in public repos). + +### 3.3 Behaviour boundaries +- Reply in the **user’s input language** (English / Spanish / French / Catalan). Do not mix languages unless the user mixes first. +- Keep DM replies short. Ask one simple question when it helps the conversation continue. +- If the user message is too short to detect language (one word / emoji): reuse the last known language in that thread; if still unclear, ask a 1‑line language question. + +--- + +## 4) The evaluation method (how we avoid harming real clients) + +We run the Instagram DM agent in **draft-only mode** first: +- A new DM arrives. +- The current system reply (if any) stays unchanged. +- The new system writes a draft reply, but does **not** send it. +- We compare “what was sent” vs “what we would have sent” in a table. + +This allows safety review and quality scoring before any automation is allowed to send messages. + +--- + +## 5) Neutral inbox stats (aggregated, no identities) + +**Inbox:** `@socialmediatorr` +**Time zone used here:** CET +**Observed window:** 2024-10-20 → 2025-12-22 (429 days) + +### 5.1 Basic counts (whole window) + +| Metric | Value | +|---|---:| +| Total messages | 54,069 | +| Messages sent by owner | 43,607 | +| Messages sent by others | 10,462 | +| Messages that look like a question/request | 2,715 | +| System “new follower” messages | 8,081 | + +Important context: this export is **not evenly distributed**. **2025‑12** dominates the dataset (87.5% of messages). Earlier months are less reliable for “trend” claims. + +### 5.2 Top 20 things people ask (by meaning, not exact wording) + +| Rank | Topic (plain English) | Count | Share of questions/requests | +|---:|---|---:|---:| +| 1 | Just one word: book | 1,857 | 68.4% | +| 2 | What is this? | 203 | 7.5% | +| 3 | Can you send the video? | 189 | 7.0% | +| 4 | Other question | 118 | 4.3% | +| 5 | Can you help me? | 74 | 2.7% | +| 6 | Can you send the link? | 70 | 2.6% | +| 7 | What does it cost? | 53 | 2.0% | +| 8 | Is this therapy? | 44 | 1.6% | +| 9 | Where do I get the book? | 36 | 1.3% | +| 10 | I can’t find it / it didn’t arrive | 26 | 1.0% | +| 11 | How do I book a call? | 11 | 0.4% | +| 12 | How do I start? | 10 | 0.4% | +| 13 | Can we talk on WhatsApp? | 7 | 0.3% | +| 14 | How does it work? | 5 | 0.2% | +| 15 | What are the steps? | 4 | 0.1% | +| 16 | Is this real? | 4 | 0.1% | +| 17 | Is it free? | 2 | 0.1% | +| 18 | Can I get a refund? | 1 | 0.0% | +| 19 | How long does it take? | 1 | 0.0% | + +Note: #20 is not present in the data-driven list above; we added one practical “coverage” item in the draft reply library (“Where are you based?”) because it is common in real inboxes even when it didn’t rank here. + +### 5.3 When messages arrive (CET) + +| Time block (CET) | Messages from people | +|---|---:| +| 00:00–05:59 | 2,113 | +| 06:00–11:59 | 1,274 | +| 12:00–17:59 | 2,333 | +| 18:00–23:59 | 4,742 | + +### 5.4 Day-of-week pattern + +| Day of week | Messages from people | Questions/requests | +|---|---:|---:| +| Monday | 1,600 | 131 | +| Tuesday | 1,939 | 194 | +| Wednesday | 1,282 | 159 | +| Thursday | 2,261 | 1,268 | +| Friday | 1,705 | 803 | +| Saturday | 833 | 87 | +| Sunday | 842 | 73 | + +### 5.5 Language mix (important caveat) + +A large share of inbound messages are **too short to detect** (one word, emoji, “book”). That is not “unknown language”; it is “insufficient text to classify”. + +Approx language estimate (inbound, “enough text” only): +- English ~54% +- Spanish ~42% +- French ~4% +- Catalan ~0.3% + +--- + +## 6) Draft artifacts to review (safe to share) + +These are included so you can critique the approach, not so you “agree with it”. + +### 6.1 VoiceDNA file (last 6 months, manual replies only) + +This is a rules + stats fingerprint generated from the last 6 months of **manual (non-template) DM replies**. It includes: +- language-mirroring rules +- reply length distribution +- response timing distribution (within a response window) +- *no* verbatim client DMs + +Full JSON is in Appendix A. + +### 6.2 Top 20 ready-made answers (multi-language, placeholders) + +These are draft answers for the Top 20 topics. They include placeholders like `{BOOK_LINK}` and `{PRICE}` (no real links inside this packet). +Full list is in Appendix B. + +### 6.3 if.emotion “Sergio” system prompt (reference) + +We include this as a reference point for the style constraints used in `emo-social.infrafabric.io`. +Full text is in Appendix C. + +--- + +## 7) The review request (please answer this) + +### 7.1 Required tone + +Be objective. Do **not** flatter. If you think something is wrong or risky, say so plainly. +“Nice idea” without specific criticism is not helpful. + +### 7.2 Your first line + +State: +- the model/provider you are (or likely are) +- your estimated knowledge cutoff +- whether you could access this URL content directly (yes/no) + +### 7.3 What looks strong (short list) + +List what is actually strong *and why* (no generic praise). + +### 7.4 Gaps / risks (ranked, with severity) + +Identify gaps we may have missed across: +- safety +- privacy +- quality of replies (tone, clarity, wrong-language risk) +- operational failure modes (how this breaks in real inboxes) +- “bad incentives” (what it might push people toward unintentionally) + +Rank them by severity (highest risk first). + +### 7.5 Patches to consider (concrete) + +For each patch: +- describe the change in 1–3 sentences +- explain the reason in 1 sentence +- if it’s a prompt change, show the exact lines you would add/remove +- if it’s a process change, describe the exact step you would add + +### 7.6 A safe test plan (step-by-step) + +Propose a small, safe test plan that: +- starts in draft-only mode +- uses a simple scoring method (clear / correct language / correct next step / no overreach) +- escalates risky content to a human +- makes it hard to “fake success” + +### 7.7 Transferability to other verticals + +Could this approach transfer to other areas (support, sales, onboarding, triage)? +- What transfers unchanged? +- What must be rebuilt from scratch? +- Where does it become unsafe? + +### 7.8 Clarifying questions (exactly 5) + +Ask 5 questions that would most improve your review if answered. + +--- + +## 8) Optional “role prompts” (if you want to run 3 reviews) + +If you are running multiple independent reviews, assign one of these roles to each reviewer: + +### Role A — Safety reviewer +Focus on: harm, escalation rules, “what must never happen”, and abuse cases. + +### Role B — Product reviewer +Focus on: usefulness, clarity, real inbox behaviour, and what will fail in practice. + +### Role C — Engineering reviewer +Focus on: how to test it, how to measure it, and how to prevent silent failure. + +--- + +## Appendix A — VoiceDNA JSON (safe; no private DM quotes) + +```json +{ + "schema_version": "voice_dna/v1", + "created_at_utc": "2025-12-24T12:08:24+00:00", + "subject": { + "account": "@socialmediatorr", + "owner_name": "Sergio de Vocht", + "scope": "Instagram DMs" + }, + "source": { + "type": "instagram_export", + "window": { + "months": 6, + "start_utc": "2025-06-24T12:08:20+00:00", + "end_utc": "2025-12-24T12:08:20+00:00", + "response_window_hours": 72.0 + }, + "classification": { + "manual_reply_definition": "owner text message within response_window_hours of the most recent inbound message, excluding repeated templates", + "scripted_template_definition": "owner canonicalized text sent >= 50 times across full export", + "system_messages_excluded": [ + "you messaged because they followed your account" + ] + }, + "scan": { + "export_root_hint": "socialmediatorr-ig-export-raw-20251224", + "scanned_conversations": 10100, + "scanned_message_files": 10061, + "candidate_responses_in_window": 18934, + "manual_responses_in_window": 825, + "scripted_template_count": 24 + } + }, + "policies": { + "language": { + "mode": "mirror_user_input_language", + "supported_languages": [ + "English", + "Spanish", + "French", + "Catalan" + ], + "rules": [ + "Reply in the same language as the user's most recent message that contains enough text to classify.", + "Do not translate the user's message unless they explicitly ask for a translation.", + "Do not mix languages inside a single reply unless the user mixes languages first.", + "If the user's message is too short to classify, reuse the last confidently detected language in the same thread.", + "If there is still no signal, ask a 1-line clarification asking which language they prefer (keep it short)." + ] + } + }, + "language_observed": { + "inbound_last_window_counts": { + "English": 2828, + "Too short to tell": 5024, + "Spanish": 2176, + "French": 191, + "Catalan": 16 + }, + "manual_reply_counts": { + "Spanish": 541, + "Too short to tell": 255, + "English": 17, + "French": 1, + "Catalan": 11 + } + }, + "style": { + "manual_replies": { + "overall": { + "count": 825, + "length": { + "chars": { + "min": 2, + "p10": 13, + "median": 54, + "p90": 199, + "max": 928, + "mean": 87.47878787878788 + }, + "words": { + "min": 1, + "p10": 2, + "median": 11, + "p90": 34, + "max": 169, + "mean": 15.436363636363636 + } + }, + "rates": { + "emoji_messages_pct": 16.363636363636363, + "question_messages_pct": 32.484848484848484, + "ends_with_question_pct": 27.151515151515156, + "exclamation_messages_pct": 18.303030303030305, + "linebreak_messages_pct": 9.212121212121211, + "url_messages_pct": 4.121212121212121, + "handle_messages_pct": 0.0, + "number_messages_pct": 9.454545454545455, + "starts_with_greeting_pct": 4.96969696969697, + "contains_thanks_pct": 4.484848484848484, + "contains_cta_terms_pct": 4.848484848484849 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 41 + }, + { + "emoji": "🫶", + "count": 23 + }, + { + "emoji": "👋", + "count": 19 + }, + { + "emoji": "🎁", + "count": 17 + }, + { + "emoji": "💪", + "count": 14 + }, + { + "emoji": "👇", + "count": 12 + }, + { + "emoji": "😜", + "count": 7 + }, + { + "emoji": "😉", + "count": 4 + }, + { + "emoji": "🤣", + "count": 2 + }, + { + "emoji": "🙏", + "count": 2 + }, + { + "emoji": "⬆", + "count": 2 + }, + { + "emoji": "👀", + "count": 2 + }, + { + "emoji": "¦", + "count": 1 + }, + { + "emoji": "😅", + "count": 1 + }, + { + "emoji": "🤝", + "count": 1 + }, + { + "emoji": "☺", + "count": 1 + }, + { + "emoji": "🫡", + "count": 1 + }, + { + "emoji": "😁", + "count": 1 + }, + { + "emoji": "👌", + "count": 1 + }, + { + "emoji": "🚀", + "count": 1 + } + ] + }, + "by_language": { + "Spanish": { + "count": 541, + "length": { + "chars": { + "min": 5, + "p10": 35, + "median": 99, + "p90": 211, + "max": 928, + "mean": 116.06469500924214 + }, + "words": { + "min": 1, + "p10": 6, + "median": 18, + "p90": 38, + "max": 169, + "mean": 20.478743068391868 + } + }, + "rates": { + "emoji_messages_pct": 20.70240295748614, + "question_messages_pct": 39.55637707948244, + "ends_with_question_pct": 32.34750462107209, + "exclamation_messages_pct": 16.266173752310536, + "linebreak_messages_pct": 13.67837338262477, + "url_messages_pct": 2.2181146025878005, + "handle_messages_pct": 0.0, + "number_messages_pct": 9.242144177449168, + "starts_with_greeting_pct": 7.578558225508318, + "contains_thanks_pct": 6.839186691312385, + "contains_cta_terms_pct": 7.024029574861368 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 24 + }, + { + "emoji": "🫶", + "count": 23 + }, + { + "emoji": "👋", + "count": 19 + }, + { + "emoji": "🎁", + "count": 17 + }, + { + "emoji": "👇", + "count": 12 + }, + { + "emoji": "💪", + "count": 10 + }, + { + "emoji": "😜", + "count": 7 + }, + { + "emoji": "😉", + "count": 4 + }, + { + "emoji": "🤣", + "count": 2 + }, + { + "emoji": "🙏", + "count": 2 + }, + { + "emoji": "⬆", + "count": 2 + }, + { + "emoji": "👀", + "count": 2 + }, + { + "emoji": "😅", + "count": 1 + }, + { + "emoji": "🤝", + "count": 1 + }, + { + "emoji": "☺", + "count": 1 + }, + { + "emoji": "😁", + "count": 1 + }, + { + "emoji": "👌", + "count": 1 + }, + { + "emoji": "🚀", + "count": 1 + }, + { + "emoji": "⏱", + "count": 1 + }, + { + "emoji": "🫂", + "count": 1 + } + ] + }, + "Too short to tell": { + "count": 255, + "length": { + "chars": { + "min": 2, + "p10": 10, + "median": 30, + "p90": 53, + "max": 151, + "mean": 32.01176470588236 + }, + "words": { + "min": 1, + "p10": 1, + "median": 4, + "p90": 11, + "max": 29, + "mean": 5.749019607843137 + } + }, + "rates": { + "emoji_messages_pct": 7.8431372549019605, + "question_messages_pct": 19.607843137254903, + "ends_with_question_pct": 17.647058823529413, + "exclamation_messages_pct": 23.52941176470588, + "linebreak_messages_pct": 0.39215686274509803, + "url_messages_pct": 4.705882352941177, + "handle_messages_pct": 0.0, + "number_messages_pct": 7.0588235294117645, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 0.0 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 15 + }, + { + "emoji": "💪", + "count": 4 + }, + { + "emoji": "🫡", + "count": 1 + } + ] + }, + "English": { + "count": 17, + "length": { + "chars": { + "min": 4, + "p10": 11, + "median": 23, + "p90": 38, + "max": 40, + "mean": 23.176470588235293 + }, + "words": { + "min": 2, + "p10": 3, + "median": 4, + "p90": 6, + "max": 8, + "mean": 4.588235294117647 + } + }, + "rates": { + "emoji_messages_pct": 11.76470588235294, + "question_messages_pct": 23.52941176470588, + "ends_with_question_pct": 23.52941176470588, + "exclamation_messages_pct": 17.647058823529413, + "linebreak_messages_pct": 0.0, + "url_messages_pct": 0.0, + "handle_messages_pct": 0.0, + "number_messages_pct": 0.0, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 11.76470588235294 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 2 + } + ] + }, + "French": { + "count": 1, + "length": { + "chars": { + "min": 29, + "p10": 29, + "median": 29, + "p90": 29, + "max": 29, + "mean": 29.0 + }, + "words": { + "min": 5, + "p10": 5, + "median": 5, + "p90": 5, + "max": 5, + "mean": 5.0 + } + }, + "rates": { + "emoji_messages_pct": 100.0, + "question_messages_pct": 0.0, + "ends_with_question_pct": 0.0, + "exclamation_messages_pct": 0.0, + "linebreak_messages_pct": 0.0, + "url_messages_pct": 0.0, + "handle_messages_pct": 0.0, + "number_messages_pct": 0.0, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 0.0 + }, + "top_emojis": [ + { + "emoji": "¦", + "count": 1 + } + ] + }, + "Catalan": { + "count": 11, + "length": { + "chars": { + "min": 26, + "p10": 48, + "median": 52, + "p90": 99, + "max": 205, + "mean": 72.0909090909091 + }, + "words": { + "min": 5, + "p10": 5, + "median": 8, + "p90": 10, + "max": 34, + "mean": 9.727272727272727 + } + }, + "rates": { + "emoji_messages_pct": 0.0, + "question_messages_pct": 0.0, + "ends_with_question_pct": 0.0, + "exclamation_messages_pct": 0.0, + "linebreak_messages_pct": 9.090909090909092, + "url_messages_pct": 90.9090909090909, + "handle_messages_pct": 0.0, + "number_messages_pct": 90.9090909090909, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 0.0 + }, + "top_emojis": [] + } + } + } + } +} +``` + +--- + +## Appendix B — Top 20 ready-made answers (multi-language, placeholders only) + +# Top 20 Ready-Made Answers (Multi‑Language) + +**VoiceDNA:** `voice_dna/voiceDNA_socialmediatorr_insta_dm.json` + +Rules: +- Reply in the **same language as the user’s message** (English / Spanish / French / Catalan). +- Replace `{PLACEHOLDERS}` before sending. +- Keep it short; end with a question when it helps the conversation continue. + +--- + +## 1) Just one word: book + +- English: `Got it. Do you want the book link or the video first?` +- Spanish: `Perfecto 🙌\n¿Quieres el enlace del ebook o prefieres que te mande el vídeo primero?` +- French: `OK. Tu veux le lien du livre ou la vidéo d’abord ?` +- Catalan: `Perfecte. Vols l’enllaç del llibre o el vídeo primer?` + +## 2) What is this? + +- English: `It’s a simple, practical resource to help you move forward. What are you looking for right now?` +- Spanish: `Es un recurso práctico para avanzar.\n¿Qué estás buscando ahora mismo?` +- French: `C’est un outil simple et pratique pour avancer. Tu cherches quoi en ce moment ?` +- Catalan: `És un recurs simple i pràctic per avançar. Què busques ara mateix?` + +## 3) Can you send the video? + +- English: `Sure — here it is: {VIDEO_LINK}. Want a quick summary first?` +- Spanish: `Claro 🙌 Aquí va el vídeo: {VIDEO_LINK}\n¿Lo quieres con resumen en 3 líneas o lo ves y me dices qué te resonó?` +- French: `Bien sûr. Voilà la vidéo : {VIDEO_LINK}\nTu veux un résumé rapide ou tu la regardes et tu me dis ce qui t’a parlé ?` +- Catalan: `Sí. Aquí tens el vídeo: {VIDEO_LINK}\nVols un resum ràpid o el mires i em dius què t’ha ressonat?` + +## 4) Other question + +- English: `Tell me in one sentence what you need, and I’ll point you to the right next step.` +- Spanish: `Dime en 1 frase qué necesitas y te digo el siguiente paso 🙌` +- French: `Dis‑moi en 1 phrase ce que tu veux, et je te dis la prochaine étape.` +- Catalan: `Digue’m en 1 frase què necessites i et dic el següent pas.` + +## 5) Can you help me? + +- English: `Yes. What are you struggling with most right now?` +- Spanish: `Sí.\n¿Qué es lo que más te está costando ahora mismo?` +- French: `Oui. Qu’est‑ce qui est le plus difficile pour toi en ce moment ?` +- Catalan: `Sí. Què és el que et costa més ara mateix?` + +## 6) Can you send the link? + +- English: `Sure. Is this for the book or to book a call?` +- Spanish: `Claro.\n¿El enlace es para el ebook o para reservar una llamada?` +- French: `Bien sûr. C’est pour le livre ou pour réserver un appel ?` +- Catalan: `Clar. És per al llibre o per reservar una trucada?` + +## 7) What does it cost? + +- English: `It’s {PRICE}. Want the link?` +- Spanish: `Son {PRICE}.\n¿Te paso el enlace?` +- French: `C’est {PRICE}. Tu veux le lien ?` +- Catalan: `Són {PRICE}. Vols l’enllaç?` + +## 8) Is this therapy? + +- English: `No — this isn’t therapy in DMs. I can share resources and options. What are you looking for?` +- Spanish: `No, esto no es terapia por DM.\nPuedo ayudarte con recursos y opciones.\n¿Qué estás buscando?` +- French: `Non, ce n’est pas une thérapie en DM. Je peux te partager des ressources et des options. Tu cherches quoi ?` +- Catalan: `No, això no és teràpia per DM. Puc compartir recursos i opcions. Què busques?` + +## 9) Where do I get the book? + +- English: `Here you go: {BOOK_LINK}. Did it open for you?` +- Spanish: `Aquí tienes el enlace: {BOOK_LINK}\n¿Te llega bien?` +- French: `Tiens : {BOOK_LINK}. Ça s’ouvre bien de ton côté ?` +- Catalan: `Aquí ho tens: {BOOK_LINK}. Se t’obre bé?` + +## 10) I can’t find it / it didn’t arrive + +- English: `No worries — let’s fix it. What email did you use to buy (or send a screenshot of the receipt)?` +- Spanish: `Vale, lo solucionamos.\n¿Con qué email lo compraste? (o mándame una captura del recibo)` +- French: `OK, on règle ça. Tu as acheté avec quel e‑mail ? (ou envoie une capture du reçu)` +- Catalan: `Ho arreglem. Amb quin e‑mail ho vas comprar? (o envia una captura del rebut)` + +## 11) How do I book a call? + +- English: `Book a call here: {CALENDLY_LINK}. What times work for you?` +- Spanish: `Reserva aquí: {CALENDLY_LINK}\n¿Qué horarios te vienen bien?` +- French: `Réserve ici : {CALENDLY_LINK}. Quels créneaux te vont ?` +- Catalan: `Reserva aquí: {CALENDLY_LINK}. Quins horaris et van bé?` + +## 12) How do I start? + +- English: `To start, tell me your goal in one sentence and I’ll tell you the first step.` +- Spanish: `Para empezar: dime tu objetivo en 1 frase y te digo el primer paso 🙌` +- French: `Pour commencer, dis‑moi ton objectif en 1 phrase et je te donne le premier pas.` +- Catalan: `Per començar, digue’m el teu objectiu en 1 frase i et dic el primer pas.` + +## 13) Can we talk on WhatsApp? + +- English: `We can. If you prefer WhatsApp: {WHATSAPP_LINK}. Want to continue there?` +- Spanish: `Podemos.\nSi te va mejor WhatsApp: {WHATSAPP_LINK}\n¿Te va bien que lo hablemos por allí?` +- French: `Oui. Si tu préfères WhatsApp : {WHATSAPP_LINK}. Ça te va de continuer là‑bas ?` +- Catalan: `Sí. Si prefereixes WhatsApp: {WHATSAPP_LINK}. Et va bé seguir per allà?` + +## 14) How does it work? + +- English: `Simple: you get the resource, apply it, and if you want we can go deeper. What do you want to improve?` +- Spanish: `Es simple:\n1) te paso el recurso\n2) lo aplicas\n3) si quieres, lo afinamos juntos\n¿Qué quieres mejorar?` +- French: `C’est simple : je te partage le contenu, tu l’appliques, et si tu veux on va plus loin. Tu veux améliorer quoi ?` +- Catalan: `És simple: et passo el recurs, l’apliques, i si vols ho aprofundim. Què vols millorar?` + +## 15) What are the steps? + +- English: `1) Tell me what you want 2) I send the right link 3) You start. What’s your goal?` +- Spanish: `1) me dices qué buscas\n2) te mando el enlace correcto\n3) empiezas\n¿Cuál es tu objetivo ahora mismo?` +- French: `1) Tu me dis ce que tu veux 2) je t’envoie le bon lien 3) tu commences. Ton objectif, c’est quoi ?` +- Catalan: `1) Em dius què busques 2) t’envio l’enllaç correcte 3) comences. Quin és el teu objectiu?` + +## 16) Is this real? + +- English: `Yes. If you want, I’ll send the official link. What are you looking to get out of it?` +- Spanish: `Sí, es real.\nSi quieres, te mando el enlace oficial.\n¿Qué estás buscando conseguir?` +- French: `Oui, c’est réel. Si tu veux je t’envoie le lien officiel. Tu veux obtenir quoi exactement ?` +- Catalan: `Sí, és real. Si vols, t’envio l’enllaç oficial. Què vols aconseguir exactament?` + +## 17) Is it free? + +- English: `The full thing isn’t free, but I can share a free starting point. Want it?` +- Spanish: `No es gratis, pero sí tengo un punto de inicio gratuito.\n¿Quieres que te lo pase?` +- French: `Ce n’est pas gratuit, mais j’ai un point de départ gratuit. Tu veux que je te l’envoie ?` +- Catalan: `No és gratis, però tinc un punt d’inici gratuït. Vols que te’l passi?` + +## 18) Can I get a refund? + +- English: `Sure — happy to help. Send me the purchase email + date and I’ll sort it.` +- Spanish: `Claro, te ayudo.\nPásame el email de compra + la fecha y lo reviso.` +- French: `Bien sûr. Envoie‑moi l’e‑mail d’achat + la date et je m’en occupe.` +- Catalan: `És clar. Envia’m l’e‑mail de compra + la data i ho gestiono.` + +## 19) How long does it take? + +- English: `It depends, but usually {TIMEFRAME}. What are you trying to change?` +- Spanish: `Depende, pero normalmente {TIMEFRAME}.\n¿Qué quieres cambiar tú?` +- French: `Ça dépend, mais en général {TIMEFRAME}. Tu veux changer quoi, toi ?` +- Catalan: `Depèn, però normalment {TIMEFRAME}. Què vols canviar tu?` + +## 20) Where are you based? + +Note: added as a practical top‑20 coverage item. + +- English: `I’m based in {BASE_LOCATION} (CET) and I work online. Where are you based?` +- Spanish: `Estoy en {BASE_LOCATION} (CET) y trabajo online.\n¿Tú dónde estás?` +- French: `Je suis basé à {BASE_LOCATION} (CET) et je travaille en ligne. Et toi, tu es où ?` +- Catalan: `Sóc a {BASE_LOCATION} (CET) i treballo online. I tu, on ets?` + +--- + +## Appendix C — if.emotion “Sergio” system prompt (reference) + +```text +You are Sergio, a Spanish-speaking psychologist who specializes in debunking abstract spiritual language and operationalizing emotional concepts into concrete, testable frameworks. + +CORE PERSONALITY: +- Direct and brash communication style - challenge unfalsifiable claims +- Systems-first thinker (context/culture factors, not individual blame) +- Bilingual Spanish/English capability, but you must match the user's language; only code-switch if explicitly asked +- Convert abstract emotional language into concrete operational definitions + +VOICE GUIDELINES (MANDATORY): +- NEVER use bullet points or numbered lists - always narrative flow +- Always respond in the same language as the user's last message +- Variable sentence length pattern: short punchy + longer flowing explanation + short again +- Validate emotional reality FIRST, then challenge interpretation +- End with concrete operationalization: ask "What specific behavior in the next 30 minutes?" (in the user's language) + +RHETORICAL TOOLS: +- Aspiradora metaphor: When someone drowns in complexity, simplify to binary. "Una aspiradora no necesita 50 tipos de suciedad etiquetados. It needs one question: Is there dirt? Yes or no?" +- Reframing: "The problem isn't X. The problem is Y." + +SPANISH USAGE: +- Only when the user is writing in Spanish: + - Use Spanish for emotional validation: "Mira, eso no está mal" + - Use colloquial markers: tío, vale, pues, mira + - NEVER use formal Spanish: no obstante, asimismo, consecuentemente + +ANTI-PATTERNS (NEVER DO): +- Never pathologize neurodivergence - frame as context mismatch, not deficit +- Never use "Furthermore", "In conclusion", "One could argue" +- Never give prescriptions without mechanism explanations +``` + diff --git a/docs/external_review/external_llm_review_packet_public_v2.md b/docs/external_review/external_llm_review_packet_public_v2.md new file mode 100644 index 0000000..1c18f5b --- /dev/null +++ b/docs/external_review/external_llm_review_packet_public_v2.md @@ -0,0 +1,1097 @@ +# External Review Packet (Public) v2 — emo-social + Instagram DM Draft Assistant + +**Last updated:** 2025-12-25 +**Audience:** external reviewers (human or LLM) +**Purpose:** solicit critical feedback (not praise) on the end-to-end design + governance stack. + +**Public-safety note:** this packet contains **no private DM transcripts**, **no personal identities**, and **no secrets/tokens**. It uses aggregated counts and “made-for-review” draft examples only. + +--- + +## 0) First: what you must do (reviewer instructions) + +1) Read this packet end-to-end. +2) Answer the questions in **Section 6** exactly. +3) Be blunt. Do not flatter. If something is weak, say so plainly and propose a fix. + +### 0.1 Your first line (required) + +State: +- **Model/provider** (as accurately as you can) +- **Knowledge cutoff** (date or best estimate) +- Whether you could access this URL content directly: **Yes/No** + +If you cannot access URLs, say so and stop (do not hallucinate the dossier). + +--- + +## 1) What this is (plain English) + +This is a clinically supervised research stress-test for `emo-social.infrafabric.io`, prepared as part of an Anthropic Fellowship application. It is **not** a commercial product. + +In parallel, we are building a separate component: an Instagram DM **draft assistant** for the Instagram account `@socialmediatorr`. The DM assistant is **not therapy**, and it must never handle crisis content. Its job is to reduce delays on repetitive inbox questions while keeping trust intact. + +### 1.1 Quick glossary (so you don’t get stuck on our internal names) + +You do **not** need to know any InfraFabric terminology to review this. + +- **Draft assistant**: the system writes a suggested reply, but does not send it. +- **Triage**: the first sorting step (safe to draft vs needs a human vs urgent). +- **Panel**: a checklist-style review of the draft (multiple “seats” that vote). +- **Trace / evidence bundle**: an audit log of what happened, so we can prove decisions later without publishing private DMs. +- **IF.GOV.TRIAGE / IF.GOV.PANEL / IF.TTT**: these are just our internal labels for the three items above. If the names annoy you, ignore them and focus on the behavior. + +--- + +## 2) What we are building (two parts) + +### 2.1 emo-social.infrafabric.io (research system) + +Purpose: help users think more clearly about emotions using a defined “Sergio” style (direct, concrete, anti-vague) with a strict rule: reply in the user’s language. + +What it is **not**: +- Not a medical device. +- Not a replacement for professional care. +- Not a crisis service. + +### 2.2 Instagram DM draft assistant for `@socialmediatorr` + +Purpose: produce **draft replies** for incoming Instagram DMs, in the user’s language, aligned with Sergio’s concise DM style. A human can review/edit before sending. + +What it is **not**: +- Not ManyChat broadcast automation. +- Not an auto-messaging spam tool. +- Not therapy-by-DM. +- Not allowed to respond to self-harm / crisis situations. + +--- + +## 3) The constraint: we must simulate debates (no paid external APIs) + +We want to use the InfraFabric governance primitives as a proof-of-concept on this low-stakes DM assistant. + +However: we currently do **not** have funds for paid external LLM API access. So “multi-voice debate” must be simulated using: +- deterministic rule-based seats (Safety, Boundary, Language, Privacy, Next-step, etc.) +- optional local models only (if available), but nothing that requires paid external calls + +Your job as reviewer: tell us if this simulation is still a meaningful governance POC, and how to make it credible without external APIs. + +--- + +## 4) What the data shows (aggregated, no identities) + +**Inbox:** `@socialmediatorr` +**Time zone used here:** CET +**Observed window:** 2024-10-20 → 2025-12-22 (429 days) + +### 4.1 Basic counts + +| Metric | Value | +|---|---:| +| Total messages | 54,069 | +| Messages sent by owner | 43,607 | +| Messages sent by others | 10,462 | +| Messages that look like a question/request | 2,715 | +| System “new follower” messages | 8,081 | + +Important caveat: the dataset is not evenly distributed. **2025‑12 dominates** (87.5% of messages). Treat “trend claims” with caution. + +### 4.2 Top intents (questions/requests) + +The inbox is dominated by repeated intents (good for templating): + +| Rank | Topic (plain English) | Count | Share of questions/requests | +|---:|---|---:|---:| +| 1 | Just one word: book | 1,857 | 68.4% | +| 2 | What is this? | 203 | 7.5% | +| 3 | Can you send the video? | 189 | 7.0% | +| 4 | Other question | 118 | 4.3% | +| 5 | Can you help me? | 74 | 2.7% | + +### 4.3 Distress / escalation prevalence (so we don’t overreact) + +We ran a strict keyword scan on **inbound text messages** (subset of inbound total): +- Inbound text messages scanned: **10,337** +- High-risk flags found: **9** messages total (**0.087%**, ~1 in 1,150), across **9 threads** + - `3` with explicit self-harm/suicide wording + - `6` with abuse/maltreatment wording + +This is a **lower bound** (euphemisms won’t match), but it indicates explicit crisis wording is rare in this inbox. + +--- + +## 5) The proposed governance implementation (IF.GOV + IF.TTT) + +We propose implementing the DM assistant as an auditable governance pipeline: +- **Triage step (`IF.GOV.TRIAGE`)**: decides risk + route (normal vs needs-human vs urgent) and detects language + topic. +- **Review panel step (`IF.GOV.PANEL`)**: runs a set of rule-based “seats” that approve, request changes, or veto (no external APIs required). +- **Trace record (`IF.TTT`)**: stores an audit record (hashes + decisions + evidence bundle) so results are provable later without publishing private DMs. + +Key design choice: **draft-only by default**. No automated sending until proven safe. + +### 5.1 What “panel debate” means without external APIs + +The “panel” is a set of deterministic evaluators (“seats”). Each seat produces: +- a vote (`approve`, `request_changes`, `veto`) +- reasons +- patch suggestions (structured operations) + +Aggregation is deterministic: +- Any `veto` → escalate to human (or urgent escalate). +- Any `request_changes` → apply patches, re-run seats once. +- Else approve. + +### 5.2 Where this lives (reference, not required for the review) + +Spec file (included below): `IF.GOV + IF.TTT Spec — Instagram DM Draft Assistant` + +--- + +## 6) What we want from you (the review request) + +### 6.1 Your understanding (5–8 lines) + +Summarize what you think we are building and what the constraints are. + +### 6.2 What looks strong (short list) + +List only what is actually strong *and why* (no generic praise). + +### 6.3 Debug the design: gaps / risks (ranked by severity) + +Identify gaps we may have missed across: +- safety (crisis, abuse, coercion, self-harm, harassment) +- psychological harm (wrong tone, inviting disclosure, boundary confusion) +- privacy (what gets stored, retention, de-identification failures) +- UX (reviewer fatigue, slow workflows, low adoption) +- engineering failure modes (token expiry, rate limiting, silent failures) + +Rank by severity and explain why. + +### 6.4 Patches to consider (concrete) + +For each patch: +- Describe the change in 1–3 sentences. +- Explain the reason in 1 sentence. +- If it’s a prompt or template change, show the exact edit (copy/paste). +- If it’s a process change, name the exact step you add/remove. + +### 6.5 X‑multiplier (make it 10× better) + +Propose improvements under two constraints: + +**A) No paid external APIs** (must be local + deterministic) +- What would make this 10× more credible as a governance POC? +- What would make it 10× more useful in a real inbox? + +**B) If budget later exists** +- What additional seats, tests, or review processes would you add? +- What would you never outsource to an external model? + +### 6.6 A safe test plan (step-by-step) + +Propose a small test plan that: +- starts in draft-only mode +- uses a simple scoring rubric (correct language, correct intent, clear next step, no overreach) +- measures false positives/false negatives on escalation routing +- prevents “we fooled ourselves” success + +### 6.7 Transferability (other verticals) + +Could this governance pattern transfer to other contexts (support, sales, onboarding)? +- What transfers unchanged? +- What must be rebuilt from scratch? +- Where does it become unsafe? + +### 6.8 Five clarifying questions (exactly 5) + +Ask the 5 questions that would most improve your review. + +--- + +## Appendix A — VoiceDNA (safe; no private DM quotes) + +This VoiceDNA is derived from the last 6 months of **manual (non-template) DM replies** and contains **rules + aggregate stats**, not quotes. + +```json +{ + "schema_version": "voice_dna/v1", + "created_at_utc": "2025-12-24T12:08:24+00:00", + "subject": { + "account": "@socialmediatorr", + "owner_name": "Sergio de Vocht", + "scope": "Instagram DMs" + }, + "source": { + "type": "instagram_export", + "window": { + "months": 6, + "start_utc": "2025-06-24T12:08:20+00:00", + "end_utc": "2025-12-24T12:08:20+00:00", + "response_window_hours": 72.0 + }, + "classification": { + "manual_reply_definition": "owner text message within response_window_hours of the most recent inbound message, excluding repeated templates", + "scripted_template_definition": "owner canonicalized text sent >= 50 times across full export", + "system_messages_excluded": [ + "you messaged because they followed your account" + ] + }, + "scan": { + "export_root_hint": "socialmediatorr-ig-export-raw-20251224", + "scanned_conversations": 10100, + "scanned_message_files": 10061, + "candidate_responses_in_window": 18934, + "manual_responses_in_window": 825, + "scripted_template_count": 24 + } + }, + "policies": { + "language": { + "mode": "mirror_user_input_language", + "supported_languages": [ + "English", + "Spanish", + "French", + "Catalan" + ], + "rules": [ + "Reply in the same language as the user's most recent message that contains enough text to classify.", + "Do not translate the user's message unless they explicitly ask for a translation.", + "Do not mix languages inside a single reply unless the user mixes languages first.", + "If the user's message is too short to classify, reuse the last confidently detected language in the same thread.", + "If there is still no signal, ask a 1-line clarification asking which language they prefer (keep it short)." + ] + } + }, + "language_observed": { + "inbound_last_window_counts": { + "English": 2828, + "Too short to tell": 5024, + "Spanish": 2176, + "French": 191, + "Catalan": 16 + }, + "manual_reply_counts": { + "Spanish": 541, + "Too short to tell": 255, + "English": 17, + "French": 1, + "Catalan": 11 + } + }, + "style": { + "manual_replies": { + "overall": { + "count": 825, + "length": { + "chars": { + "min": 2, + "p10": 13, + "median": 54, + "p90": 199, + "max": 928, + "mean": 87.47878787878788 + }, + "words": { + "min": 1, + "p10": 2, + "median": 11, + "p90": 34, + "max": 169, + "mean": 15.436363636363636 + } + }, + "rates": { + "emoji_messages_pct": 16.363636363636363, + "question_messages_pct": 32.484848484848484, + "ends_with_question_pct": 27.151515151515156, + "exclamation_messages_pct": 18.303030303030305, + "linebreak_messages_pct": 9.212121212121211, + "url_messages_pct": 4.121212121212121, + "handle_messages_pct": 0.0, + "number_messages_pct": 9.454545454545455, + "starts_with_greeting_pct": 4.96969696969697, + "contains_thanks_pct": 4.484848484848484, + "contains_cta_terms_pct": 4.848484848484849 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 41 + }, + { + "emoji": "🫶", + "count": 23 + }, + { + "emoji": "👋", + "count": 19 + }, + { + "emoji": "🎁", + "count": 17 + }, + { + "emoji": "💪", + "count": 14 + }, + { + "emoji": "👇", + "count": 12 + }, + { + "emoji": "😜", + "count": 7 + }, + { + "emoji": "😉", + "count": 4 + }, + { + "emoji": "🤣", + "count": 2 + }, + { + "emoji": "🙏", + "count": 2 + }, + { + "emoji": "⬆", + "count": 2 + }, + { + "emoji": "👀", + "count": 2 + }, + { + "emoji": "¦", + "count": 1 + }, + { + "emoji": "😅", + "count": 1 + }, + { + "emoji": "🤝", + "count": 1 + }, + { + "emoji": "☺", + "count": 1 + }, + { + "emoji": "🫡", + "count": 1 + }, + { + "emoji": "😁", + "count": 1 + }, + { + "emoji": "👌", + "count": 1 + }, + { + "emoji": "🚀", + "count": 1 + } + ] + }, + "by_language": { + "Spanish": { + "count": 541, + "length": { + "chars": { + "min": 5, + "p10": 35, + "median": 99, + "p90": 211, + "max": 928, + "mean": 116.06469500924214 + }, + "words": { + "min": 1, + "p10": 6, + "median": 18, + "p90": 38, + "max": 169, + "mean": 20.478743068391868 + } + }, + "rates": { + "emoji_messages_pct": 20.70240295748614, + "question_messages_pct": 39.55637707948244, + "ends_with_question_pct": 32.34750462107209, + "exclamation_messages_pct": 16.266173752310536, + "linebreak_messages_pct": 13.67837338262477, + "url_messages_pct": 2.2181146025878005, + "handle_messages_pct": 0.0, + "number_messages_pct": 9.242144177449168, + "starts_with_greeting_pct": 7.578558225508318, + "contains_thanks_pct": 6.839186691312385, + "contains_cta_terms_pct": 7.024029574861368 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 24 + }, + { + "emoji": "🫶", + "count": 23 + }, + { + "emoji": "👋", + "count": 19 + }, + { + "emoji": "🎁", + "count": 17 + }, + { + "emoji": "👇", + "count": 12 + }, + { + "emoji": "💪", + "count": 10 + }, + { + "emoji": "😜", + "count": 7 + }, + { + "emoji": "😉", + "count": 4 + }, + { + "emoji": "🤣", + "count": 2 + }, + { + "emoji": "🙏", + "count": 2 + }, + { + "emoji": "⬆", + "count": 2 + }, + { + "emoji": "👀", + "count": 2 + }, + { + "emoji": "😅", + "count": 1 + }, + { + "emoji": "🤝", + "count": 1 + }, + { + "emoji": "☺", + "count": 1 + }, + { + "emoji": "😁", + "count": 1 + }, + { + "emoji": "👌", + "count": 1 + }, + { + "emoji": "🚀", + "count": 1 + }, + { + "emoji": "⏱", + "count": 1 + }, + { + "emoji": "🫂", + "count": 1 + } + ] + }, + "Too short to tell": { + "count": 255, + "length": { + "chars": { + "min": 2, + "p10": 10, + "median": 30, + "p90": 53, + "max": 151, + "mean": 32.01176470588236 + }, + "words": { + "min": 1, + "p10": 1, + "median": 4, + "p90": 11, + "max": 29, + "mean": 5.749019607843137 + } + }, + "rates": { + "emoji_messages_pct": 7.8431372549019605, + "question_messages_pct": 19.607843137254903, + "ends_with_question_pct": 17.647058823529413, + "exclamation_messages_pct": 23.52941176470588, + "linebreak_messages_pct": 0.39215686274509803, + "url_messages_pct": 4.705882352941177, + "handle_messages_pct": 0.0, + "number_messages_pct": 7.0588235294117645, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 0.0 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 15 + }, + { + "emoji": "💪", + "count": 4 + }, + { + "emoji": "🫡", + "count": 1 + } + ] + }, + "English": { + "count": 17, + "length": { + "chars": { + "min": 4, + "p10": 11, + "median": 23, + "p90": 38, + "max": 40, + "mean": 23.176470588235293 + }, + "words": { + "min": 2, + "p10": 3, + "median": 4, + "p90": 6, + "max": 8, + "mean": 4.588235294117647 + } + }, + "rates": { + "emoji_messages_pct": 11.76470588235294, + "question_messages_pct": 23.52941176470588, + "ends_with_question_pct": 23.52941176470588, + "exclamation_messages_pct": 17.647058823529413, + "linebreak_messages_pct": 0.0, + "url_messages_pct": 0.0, + "handle_messages_pct": 0.0, + "number_messages_pct": 0.0, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 11.76470588235294 + }, + "top_emojis": [ + { + "emoji": "🙌", + "count": 2 + } + ] + }, + "French": { + "count": 1, + "length": { + "chars": { + "min": 29, + "p10": 29, + "median": 29, + "p90": 29, + "max": 29, + "mean": 29.0 + }, + "words": { + "min": 5, + "p10": 5, + "median": 5, + "p90": 5, + "max": 5, + "mean": 5.0 + } + }, + "rates": { + "emoji_messages_pct": 100.0, + "question_messages_pct": 0.0, + "ends_with_question_pct": 0.0, + "exclamation_messages_pct": 0.0, + "linebreak_messages_pct": 0.0, + "url_messages_pct": 0.0, + "handle_messages_pct": 0.0, + "number_messages_pct": 0.0, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 0.0 + }, + "top_emojis": [ + { + "emoji": "¦", + "count": 1 + } + ] + }, + "Catalan": { + "count": 11, + "length": { + "chars": { + "min": 26, + "p10": 48, + "median": 52, + "p90": 99, + "max": 205, + "mean": 72.0909090909091 + }, + "words": { + "min": 5, + "p10": 5, + "median": 8, + "p90": 10, + "max": 34, + "mean": 9.727272727272727 + } + }, + "rates": { + "emoji_messages_pct": 0.0, + "question_messages_pct": 0.0, + "ends_with_question_pct": 0.0, + "exclamation_messages_pct": 0.0, + "linebreak_messages_pct": 9.090909090909092, + "url_messages_pct": 90.9090909090909, + "handle_messages_pct": 0.0, + "number_messages_pct": 90.9090909090909, + "starts_with_greeting_pct": 0.0, + "contains_thanks_pct": 0.0, + "contains_cta_terms_pct": 0.0 + }, + "top_emojis": [] + } + } + } + } +} +``` + +--- + +## Appendix B — Top 20 ready-made answers (multi-language; placeholders only) + +These are draft replies for repeated intents. They include placeholders like `{BOOK_LINK}` and `{PRICE}` (no real links inside this packet). + +# Top 20 Ready-Made Answers (Multi‑Language) + +**VoiceDNA:** `voice_dna/voiceDNA_socialmediatorr_insta_dm.json` + +Rules: +- Reply in the **same language as the user’s message** (English / Spanish / French / Catalan). +- Replace `{PLACEHOLDERS}` before sending. +- Keep it short; end with a question when it helps the conversation continue. + +--- + +## 1) Just one word: book + +- English: `Got it. Do you want the book link or the video first?` +- Spanish: `Perfecto 🙌\n¿Quieres el enlace del ebook o prefieres que te mande el vídeo primero?` +- French: `OK. Tu veux le lien du livre ou la vidéo d’abord ?` +- Catalan: `Perfecte. Vols l’enllaç del llibre o el vídeo primer?` + +## 2) What is this? + +- English: `It’s a simple, practical resource to help you move forward. What are you looking for right now?` +- Spanish: `Es un recurso práctico para avanzar.\n¿Qué estás buscando ahora mismo?` +- French: `C’est un outil simple et pratique pour avancer. Tu cherches quoi en ce moment ?` +- Catalan: `És un recurs simple i pràctic per avançar. Què busques ara mateix?` + +## 3) Can you send the video? + +- English: `Sure — here it is: {VIDEO_LINK}. Want a quick summary first?` +- Spanish: `Claro 🙌 Aquí va el vídeo: {VIDEO_LINK}\n¿Lo quieres con resumen en 3 líneas o lo ves y me dices qué te resonó?` +- French: `Bien sûr. Voilà la vidéo : {VIDEO_LINK}\nTu veux un résumé rapide ou tu la regardes et tu me dis ce qui t’a parlé ?` +- Catalan: `Sí. Aquí tens el vídeo: {VIDEO_LINK}\nVols un resum ràpid o el mires i em dius què t’ha ressonat?` + +## 4) Other question + +- English: `Tell me in one sentence what you need, and I’ll point you to the right next step.` +- Spanish: `Dime en 1 frase qué necesitas y te digo el siguiente paso 🙌` +- French: `Dis‑moi en 1 phrase ce que tu veux, et je te dis la prochaine étape.` +- Catalan: `Digue’m en 1 frase què necessites i et dic el següent pas.` + +## 5) Can you help me? + +- English: `Yes. What are you struggling with most right now?` +- Spanish: `Sí.\n¿Qué es lo que más te está costando ahora mismo?` +- French: `Oui. Qu’est‑ce qui est le plus difficile pour toi en ce moment ?` +- Catalan: `Sí. Què és el que et costa més ara mateix?` + +## 6) Can you send the link? + +- English: `Sure. Is this for the book or to book a call?` +- Spanish: `Claro.\n¿El enlace es para el ebook o para reservar una llamada?` +- French: `Bien sûr. C’est pour le livre ou pour réserver un appel ?` +- Catalan: `Clar. És per al llibre o per reservar una trucada?` + +## 7) What does it cost? + +- English: `It’s {PRICE}. Want the link?` +- Spanish: `Son {PRICE}.\n¿Te paso el enlace?` +- French: `C’est {PRICE}. Tu veux le lien ?` +- Catalan: `Són {PRICE}. Vols l’enllaç?` + +## 8) Is this therapy? + +- English: `No — this isn’t therapy in DMs. I can share resources and options. What are you looking for?` +- Spanish: `No, esto no es terapia por DM.\nPuedo ayudarte con recursos y opciones.\n¿Qué estás buscando?` +- French: `Non, ce n’est pas une thérapie en DM. Je peux te partager des ressources et des options. Tu cherches quoi ?` +- Catalan: `No, això no és teràpia per DM. Puc compartir recursos i opcions. Què busques?` + +## 9) Where do I get the book? + +- English: `Here you go: {BOOK_LINK}. Did it open for you?` +- Spanish: `Aquí tienes el enlace: {BOOK_LINK}\n¿Te llega bien?` +- French: `Tiens : {BOOK_LINK}. Ça s’ouvre bien de ton côté ?` +- Catalan: `Aquí ho tens: {BOOK_LINK}. Se t’obre bé?` + +## 10) I can’t find it / it didn’t arrive + +- English: `No worries — let’s fix it. What email did you use to buy (or send a screenshot of the receipt)?` +- Spanish: `Vale, lo solucionamos.\n¿Con qué email lo compraste? (o mándame una captura del recibo)` +- French: `OK, on règle ça. Tu as acheté avec quel e‑mail ? (ou envoie une capture du reçu)` +- Catalan: `Ho arreglem. Amb quin e‑mail ho vas comprar? (o envia una captura del rebut)` + +## 11) How do I book a call? + +- English: `Book a call here: {CALENDLY_LINK}. What times work for you?` +- Spanish: `Reserva aquí: {CALENDLY_LINK}\n¿Qué horarios te vienen bien?` +- French: `Réserve ici : {CALENDLY_LINK}. Quels créneaux te vont ?` +- Catalan: `Reserva aquí: {CALENDLY_LINK}. Quins horaris et van bé?` + +## 12) How do I start? + +- English: `To start, tell me your goal in one sentence and I’ll tell you the first step.` +- Spanish: `Para empezar: dime tu objetivo en 1 frase y te digo el primer paso 🙌` +- French: `Pour commencer, dis‑moi ton objectif en 1 phrase et je te donne le premier pas.` +- Catalan: `Per començar, digue’m el teu objectiu en 1 frase i et dic el primer pas.` + +## 13) Can we talk on WhatsApp? + +- English: `We can. If you prefer WhatsApp: {WHATSAPP_LINK}. Want to continue there?` +- Spanish: `Podemos.\nSi te va mejor WhatsApp: {WHATSAPP_LINK}\n¿Te va bien que lo hablemos por allí?` +- French: `Oui. Si tu préfères WhatsApp : {WHATSAPP_LINK}. Ça te va de continuer là‑bas ?` +- Catalan: `Sí. Si prefereixes WhatsApp: {WHATSAPP_LINK}. Et va bé seguir per allà?` + +## 14) How does it work? + +- English: `Simple: you get the resource, apply it, and if you want we can go deeper. What do you want to improve?` +- Spanish: `Es simple:\n1) te paso el recurso\n2) lo aplicas\n3) si quieres, lo afinamos juntos\n¿Qué quieres mejorar?` +- French: `C’est simple : je te partage le contenu, tu l’appliques, et si tu veux on va plus loin. Tu veux améliorer quoi ?` +- Catalan: `És simple: et passo el recurs, l’apliques, i si vols ho aprofundim. Què vols millorar?` + +## 15) What are the steps? + +- English: `1) Tell me what you want 2) I send the right link 3) You start. What’s your goal?` +- Spanish: `1) me dices qué buscas\n2) te mando el enlace correcto\n3) empiezas\n¿Cuál es tu objetivo ahora mismo?` +- French: `1) Tu me dis ce que tu veux 2) je t’envoie le bon lien 3) tu commences. Ton objectif, c’est quoi ?` +- Catalan: `1) Em dius què busques 2) t’envio l’enllaç correcte 3) comences. Quin és el teu objectiu?` + +## 16) Is this real? + +- English: `Yes. If you want, I’ll send the official link. What are you looking to get out of it?` +- Spanish: `Sí, es real.\nSi quieres, te mando el enlace oficial.\n¿Qué estás buscando conseguir?` +- French: `Oui, c’est réel. Si tu veux je t’envoie le lien officiel. Tu veux obtenir quoi exactement ?` +- Catalan: `Sí, és real. Si vols, t’envio l’enllaç oficial. Què vols aconseguir exactament?` + +## 17) Is it free? + +- English: `The full thing isn’t free, but I can share a free starting point. Want it?` +- Spanish: `No es gratis, pero sí tengo un punto de inicio gratuito.\n¿Quieres que te lo pase?` +- French: `Ce n’est pas gratuit, mais j’ai un point de départ gratuit. Tu veux que je te l’envoie ?` +- Catalan: `No és gratis, però tinc un punt d’inici gratuït. Vols que te’l passi?` + +## 18) Can I get a refund? + +- English: `Sure — happy to help. Send me the purchase email + date and I’ll sort it.` +- Spanish: `Claro, te ayudo.\nPásame el email de compra + la fecha y lo reviso.` +- French: `Bien sûr. Envoie‑moi l’e‑mail d’achat + la date et je m’en occupe.` +- Catalan: `És clar. Envia’m l’e‑mail de compra + la data i ho gestiono.` + +## 19) How long does it take? + +- English: `It depends, but usually {TIMEFRAME}. What are you trying to change?` +- Spanish: `Depende, pero normalmente {TIMEFRAME}.\n¿Qué quieres cambiar tú?` +- French: `Ça dépend, mais en général {TIMEFRAME}. Tu veux changer quoi, toi ?` +- Catalan: `Depèn, però normalment {TIMEFRAME}. Què vols canviar tu?` + +## 20) Where are you based? + +Note: added as a practical top‑20 coverage item. + +- English: `I’m based in {BASE_LOCATION} (CET) and I work online. Where are you based?` +- Spanish: `Estoy en {BASE_LOCATION} (CET) y trabajo online.\n¿Tú dónde estás?` +- French: `Je suis basé à {BASE_LOCATION} (CET) et je travaille en ligne. Et toi, tu es où ?` +- Catalan: `Sóc a {BASE_LOCATION} (CET) i treballo online. I tu, on ets?` + +--- + +## Appendix C — IF.GOV + IF.TTT spec (the proposed implementation) + +# IF.GOV + IF.TTT Spec — Instagram DM Draft Assistant (`@socialmediatorr`) + +**Status:** proposal (POC) +**Constraint:** no paid external LLM APIs → “debates” are simulated using deterministic seats (rules) and optional local models only. + +This spec describes how to implement the Instagram DM assistant as an **auditable governance pipeline**: +- **Triage step (`IF.GOV.TRIAGE`)** decides risk + route (normal vs human vs urgent). +- **Review panel step (`IF.GOV.PANEL`)** simulates a multi‑seat review of the proposed draft reply (no external APIs required). +- **Trace record (`IF.TTT`)** records a chain‑of‑custody (hashes + decisions + evidence bundle) so results are provable later. + +--- + +## 0) System boundaries (what we will and won’t do) + +### In scope +- Ingest Meta webhook events for Instagram DMs. +- Produce **draft replies** (default) using templates + simple intent routing. +- Escalate a tiny fraction of DMs to a human (Sergio) quickly, with a direct “open thread” link. +- Produce IF.TTT‑style trace records and evidence bundles for audit/replay. +- Run “panel debates” **without external APIs** (rule seats + optional local model seats). + +### Out of scope (for the POC) +- Automatic sending of replies to real clients (keep `draft-only`). +- Therapy-by-DM, crisis intervention, diagnosis, or medical claims. +- Storing/exporting full DM transcripts in a public repo. + +--- + +## 1) High-level architecture + +### Components +- **Webhook receiver** (already exists in production on `emo-social.infrafabric.io`): verifies Meta signature and normalizes events. +- **Event store**: append-only storage of DM events + derived decisions (local, private). +- **Triage engine** (`IF.GOV.TRIAGE`): risk + language + intent + confidence. +- **Draft engine**: chooses a reply template (Top 20) or a safe fallback. +- **Panel engine** (`IF.GOV.PANEL`): simulated debate across “seats” → approve/patch/escalate. +- **Trace recorder** (`IF.TTT`): emits signed decision records + evidence bundles. +- **Reviewer UI**: queue view for Drafts + Escalations + “open IG thread” action. + +### Data flow (valid Mermaid) +```mermaid +flowchart LR + W[Meta webhook event] --> V[Verify signature] + V --> N[Normalize event] + N --> ES[Event store append] + ES --> T[IF.GOV.TRIAGE] + T -->|urgent| E[Escalation record] + T -->|normal| D[Draft engine] + T -->|needs-human| H[Human-required record] + D --> P[IF.GOV.PANEL seats] + P --> R[Panel decision] + E --> TR[IF.TTT trace + bundle] + H --> TR + R --> TR + TR --> UI[Reviewer UI queue] +``` + +--- + +## 2) IF.GOV.TRIAGE (no external API) + +### Inputs +- `sender_id` (from webhook) +- `mid` (message id) +- `timestamp_ms` +- `text` (if present; empty allowed) +- minimal thread context (last N messages for this sender_id, if available) + +### Outputs (contract) +```json +{ + "triage_version": "if.gov.triage/igdm/v1", + "trace_id": "uuid", + "ts_utc": "2025-12-25T12:00:00Z", + "time_cet": "2025-12-25T13:00:00+01:00", + "sender_id": "123", + "mid": "m_abc", + "language": { "code": "es", "confidence": 0.86, "source": "text_or_thread" }, + "intent": { "label": "book|link|video|price|help|other", "confidence": 0.90 }, + "risk": { + "tier": "normal|needs-human|urgent", + "score": 0.05, + "reasons": ["..."], + "panel_size": 5 + } +} +``` + +### Triage rules (POC defaults) +- **Language detection** + - If message has enough text: detect language from message text. + - Else: reuse last confident thread language. + - Else: set `confidence < 0.5` and prefer a 1‑line language question. +- **Intent detection** + - Keyword routing for: `book`, `link`, `video`, `price/cost`, `call`, `therapy`, etc. + - If unknown: intent=`other` with low confidence. +- **Risk tier** + - `urgent` if self-harm/suicide signals OR violence/abuse indicators. + - `needs-human` if: therapeutic disclosure, legal threats, harassment, complex personal crisis, repeated angry loop. + - `normal` otherwise. + +### “Panel size” without external APIs +Panel size is computed deterministically from `risk.score` (same pattern as the existing `guard_engine.py`): +- normal: 5 seats +- needs-human: 10 seats (more checks, but still local) +- urgent: 20 seats (but action is always escalate, not debate content) + +--- + +## 3) Draft engine (no external API) + +### Principles +- Use **templates first**, not a generative model. +- Always mirror the user’s language (or ask a 1‑line language question if uncertain). +- Keep replies short; ask one clear next question when helpful. +- Never invite deep disclosure in DMs; route to “resources / call / book link”. + +### Draft outputs +```json +{ + "draft_version": "igdm.draft/v1", + "trace_id": "uuid", + "template_id": "top20:book:v1:es", + "text": "…", + "placeholders": ["BOOK_LINK"], + "notes": ["language=es", "intent=book"] +} +``` + +--- + +## 4) IF.GOV.PANEL (simulated debates) + +### What “debate” means here +Because we are not calling external LLMs, the “panel” is a set of **deterministic seat evaluators**. +Each seat emits: +- a vote (`approve` | `request_changes` | `veto`) +- reasons (human readable) +- patch suggestions (structured) + +### Seat roster (minimum viable, 5 seats) +1) **Safety seat**: blocks crisis mishandling; ensures no harmful advice. +2) **Boundary seat**: prevents therapy-by-DM; rewrites “help” flows into routing. +3) **Language seat**: enforces same-language output; no mixing; handles low confidence. +4) **Privacy seat**: avoids unnecessary PII; flags risky asks (phone/email) unless explicitly required. +5) **Next-step seat**: checks the reply has a clear next step (link or one question). + +Optional seats (when panel size grows) +- **Tone/VoiceDNA seat**: checks length + emoji pattern + directness vs DM voice rules. +- **Spam/abuse seat**: detects harassment loops and routes to block/report guidance. +- **Contrarian seat**: tries to misread the message and see if the draft fails. + +### Seat output format +```json +{ + "seat": "language", + "vote": "approve|request_changes|veto", + "severity": 0.0, + "reasons": ["..."], + "patches": [ + { "op": "replace_text", "path": "draft.text", "value": "..." } + ] +} +``` + +### Panel aggregation (deterministic) +- If any seat returns `veto` → panel decision becomes `escalate_human` (or `urgent_escalate`). +- Else if any seat returns `request_changes` → apply patches (in order), re-run seats once. +- Else → approve. + +### Panel decision record +```json +{ + "panel_version": "if.gov.panel/igdm/v1", + "trace_id": "uuid", + "panel_size": 5, + "seats": [ { "...": "..." } ], + "decision": "approve_draft|revise_draft|escalate_human|urgent_escalate", + "final_draft_text_sha256": "…", + "reason_summary": "short" +} +``` + +--- + +## 5) Escalation UX (how Sergio actually sees it) + +### Escalation record +```json +{ + "escalation_version": "igdm.escalation/v1", + "trace_id": "uuid", + "tier": "urgent|needs-human", + "reason_codes": ["self_harm_signal"], + "sender_id": "123", + "mid": "m_abc", + "time_cet": "2025-12-25T21:13:00+01:00", + "open_links": { + "instagram_thread": "https://www.instagram.com/direct/t//", + "fb_inbox": "https://business.facebook.com/latest/inbox/all/?asset_id=" + } +} +``` + +### Notification strategy (POC) +No paid services required: +- Show escalations in a **logged-in dashboard** on `emo-social.infrafabric.io`. +- Optional: email later (requires SMTP relay configured); not required for the POC. + +--- + +## 6) IF.TTT trace + evidence bundles (provable without leaking) + +### Two-bundle approach (recommended) +- **Private bundle** (internal): includes raw message text, stored locally with strict permissions. +- **Public bundle** (shareable): contains hashes + redacted previews only. + +### Bundle contents (public) +``` +bundle/ + manifest.json + event.json + triage.json + draft.json + panel.json + escalation.json (only if escalated) + sha256sums.txt + signature_ed25519.txt +``` + +### Minimum “public” fields +- `message_text_sha256` (not raw) +- `draft_text_sha256` (not raw) +- triage + panel decision + reason codes +- timestamps (UTC + CET) + +This is enough to prove: “given these bytes (committed), these deterministic governance steps happened, and this decision was produced”. + +--- + +## 7) Rollout plan (safe) + +1) **Triage-only** + escalation queue (no drafts yet). +2) **Draft-only** templates for Top 20 intents (no sending). +3) Add simulated **IF.GOV.PANEL** seats and store panel decisions. +4) Emit IF.TTT bundles for each event (public + private). +5) Add comparison table: `draft` vs `actual sent` (manual) to measure quality. +6) Only after measured success: consider limited auto-send for *low-risk* intents, with a kill switch. diff --git a/docs/governance/IF_GOV_IGDM_SPEC.md b/docs/governance/IF_GOV_IGDM_SPEC.md new file mode 100644 index 0000000..a64f8e7 --- /dev/null +++ b/docs/governance/IF_GOV_IGDM_SPEC.md @@ -0,0 +1,249 @@ +# IF.GOV + IF.TTT Spec — Instagram DM Draft Assistant (`@socialmediatorr`) + +**Status:** proposal (POC) +**Constraint:** no paid external LLM APIs → “debates” are simulated using deterministic seats (rules) and optional local models only. + +This spec describes how to implement the Instagram DM assistant as an **auditable governance pipeline**: +- **IF.GOV.TRIAGE** decides risk + route (normal vs human vs urgent). +- **IF.GOV.PANEL** simulates a multi‑seat review of the proposed draft reply (no external APIs required). +- **IF.TTT** records a chain‑of‑custody (hashes + decisions + evidence bundle) so results are provable later. + +--- + +## 0) System boundaries (what we will and won’t do) + +### In scope +- Ingest Meta webhook events for Instagram DMs. +- Produce **draft replies** (default) using templates + simple intent routing. +- Escalate a tiny fraction of DMs to a human (Sergio) quickly, with a direct “open thread” link. +- Produce IF.TTT‑style trace records and evidence bundles for audit/replay. +- Run “panel debates” **without external APIs** (rule seats + optional local model seats). + +### Out of scope (for the POC) +- Automatic sending of replies to real clients (keep `draft-only`). +- Therapy-by-DM, crisis intervention, diagnosis, or medical claims. +- Storing/exporting full DM transcripts in a public repo. + +--- + +## 1) High-level architecture + +### Components +- **Webhook receiver** (already exists in production on `emo-social.infrafabric.io`): verifies Meta signature and normalizes events. +- **Event store**: append-only storage of DM events + derived decisions (local, private). +- **Triage engine** (`IF.GOV.TRIAGE`): risk + language + intent + confidence. +- **Draft engine**: chooses a reply template (Top 20) or a safe fallback. +- **Panel engine** (`IF.GOV.PANEL`): simulated debate across “seats” → approve/patch/escalate. +- **Trace recorder** (`IF.TTT`): emits signed decision records + evidence bundles. +- **Reviewer UI**: queue view for Drafts + Escalations + “open IG thread” action. + +### Data flow (valid Mermaid) +```mermaid +flowchart LR + W[Meta webhook event] --> V[Verify signature] + V --> N[Normalize event] + N --> ES[Event store append] + ES --> T[IF.GOV.TRIAGE] + T -->|urgent| E[Escalation record] + T -->|normal| D[Draft engine] + T -->|needs-human| H[Human-required record] + D --> P[IF.GOV.PANEL seats] + P --> R[Panel decision] + E --> TR[IF.TTT trace + bundle] + H --> TR + R --> TR + TR --> UI[Reviewer UI queue] +``` + +--- + +## 2) IF.GOV.TRIAGE (no external API) + +### Inputs +- `sender_id` (from webhook) +- `mid` (message id) +- `timestamp_ms` +- `text` (if present; empty allowed) +- minimal thread context (last N messages for this sender_id, if available) + +### Outputs (contract) +```json +{ + "triage_version": "if.gov.triage/igdm/v1", + "trace_id": "uuid", + "ts_utc": "2025-12-25T12:00:00Z", + "time_cet": "2025-12-25T13:00:00+01:00", + "sender_id": "123", + "mid": "m_abc", + "language": { "code": "es", "confidence": 0.86, "source": "text_or_thread" }, + "intent": { "label": "book|link|video|price|help|other", "confidence": 0.90 }, + "risk": { + "tier": "normal|needs-human|urgent", + "score": 0.05, + "reasons": ["..."], + "panel_size": 5 + } +} +``` + +### Triage rules (POC defaults) +- **Language detection** + - If message has enough text: detect language from message text. + - Else: reuse last confident thread language. + - Else: set `confidence < 0.5` and prefer a 1‑line language question. +- **Intent detection** + - Keyword routing for: `book`, `link`, `video`, `price/cost`, `call`, `therapy`, etc. + - If unknown: intent=`other` with low confidence. +- **Risk tier** + - `urgent` if self-harm/suicide signals OR violence/abuse indicators. + - `needs-human` if: therapeutic disclosure, legal threats, harassment, complex personal crisis, repeated angry loop. + - `normal` otherwise. + +### “Panel size” without external APIs +Panel size is computed deterministically from `risk.score` (same pattern as the existing `guard_engine.py`): +- normal: 5 seats +- needs-human: 10 seats (more checks, but still local) +- urgent: 20 seats (but action is always escalate, not debate content) + +--- + +## 3) Draft engine (no external API) + +### Principles +- Use **templates first**, not a generative model. +- Always mirror the user’s language (or ask a 1‑line language question if uncertain). +- Keep replies short; ask one clear next question when helpful. +- Never invite deep disclosure in DMs; route to “resources / call / book link”. + +### Draft outputs +```json +{ + "draft_version": "igdm.draft/v1", + "trace_id": "uuid", + "template_id": "top20:book:v1:es", + "text": "…", + "placeholders": ["BOOK_LINK"], + "notes": ["language=es", "intent=book"] +} +``` + +--- + +## 4) IF.GOV.PANEL (simulated debates) + +### What “debate” means here +Because we are not calling external LLMs, the “panel” is a set of **deterministic seat evaluators**. +Each seat emits: +- a vote (`approve` | `request_changes` | `veto`) +- reasons (human readable) +- patch suggestions (structured) + +### Seat roster (minimum viable, 5 seats) +1) **Safety seat**: blocks crisis mishandling; ensures no harmful advice. +2) **Boundary seat**: prevents therapy-by-DM; rewrites “help” flows into routing. +3) **Language seat**: enforces same-language output; no mixing; handles low confidence. +4) **Privacy seat**: avoids unnecessary PII; flags risky asks (phone/email) unless explicitly required. +5) **Next-step seat**: checks the reply has a clear next step (link or one question). + +Optional seats (when panel size grows) +- **Tone/VoiceDNA seat**: checks length + emoji pattern + directness vs DM voice rules. +- **Spam/abuse seat**: detects harassment loops and routes to block/report guidance. +- **Contrarian seat**: tries to misread the message and see if the draft fails. + +### Seat output format +```json +{ + "seat": "language", + "vote": "approve|request_changes|veto", + "severity": 0.0, + "reasons": ["..."], + "patches": [ + { "op": "replace_text", "path": "draft.text", "value": "..." } + ] +} +``` + +### Panel aggregation (deterministic) +- If any seat returns `veto` → panel decision becomes `escalate_human` (or `urgent_escalate`). +- Else if any seat returns `request_changes` → apply patches (in order), re-run seats once. +- Else → approve. + +### Panel decision record +```json +{ + "panel_version": "if.gov.panel/igdm/v1", + "trace_id": "uuid", + "panel_size": 5, + "seats": [ { "...": "..." } ], + "decision": "approve_draft|revise_draft|escalate_human|urgent_escalate", + "final_draft_text_sha256": "…", + "reason_summary": "short" +} +``` + +--- + +## 5) Escalation UX (how Sergio actually sees it) + +### Escalation record +```json +{ + "escalation_version": "igdm.escalation/v1", + "trace_id": "uuid", + "tier": "urgent|needs-human", + "reason_codes": ["self_harm_signal"], + "sender_id": "123", + "mid": "m_abc", + "time_cet": "2025-12-25T21:13:00+01:00", + "open_links": { + "instagram_thread": "https://www.instagram.com/direct/t//", + "fb_inbox": "https://business.facebook.com/latest/inbox/all/?asset_id=" + } +} +``` + +### Notification strategy (POC) +No paid services required: +- Show escalations in a **logged-in dashboard** on `emo-social.infrafabric.io`. +- Optional: email later (requires SMTP relay configured); not required for the POC. + +--- + +## 6) IF.TTT trace + evidence bundles (provable without leaking) + +### Two-bundle approach (recommended) +- **Private bundle** (internal): includes raw message text, stored locally with strict permissions. +- **Public bundle** (shareable): contains hashes + redacted previews only. + +### Bundle contents (public) +``` +bundle/ + manifest.json + event.json + triage.json + draft.json + panel.json + escalation.json (only if escalated) + sha256sums.txt + signature_ed25519.txt +``` + +### Minimum “public” fields +- `message_text_sha256` (not raw) +- `draft_text_sha256` (not raw) +- triage + panel decision + reason codes +- timestamps (UTC + CET) + +This is enough to prove: “given these bytes (committed), these deterministic governance steps happened, and this decision was produced”. + +--- + +## 7) Rollout plan (safe) + +1) **Triage-only** + escalation queue (no drafts yet). +2) **Draft-only** templates for Top 20 intents (no sending). +3) Add simulated **IF.GOV.PANEL** seats and store panel decisions. +4) Emit IF.TTT bundles for each event (public + private). +5) Add comparison table: `draft` vs `actual sent` (manual) to measure quality. +6) Only after measured success: consider limited auto-send for *low-risk* intents, with a kill switch. + diff --git a/reply_library/top20_ready_answers.json b/reply_library/top20_ready_answers.json index 9c9c370..8b83c51 100644 --- a/reply_library/top20_ready_answers.json +++ b/reply_library/top20_ready_answers.json @@ -4,7 +4,7 @@ "language_mode": "mirror_user_input_language", "notes": [ "Pick the reply language to match the user's most recent message with enough text to classify (English / Spanish / French / Catalan).", - "If the user message is too short to detect, reuse the last detected language in the thread; if still unknown, ask which language they prefer.", + "If the user message is too short to detect, reuse the last detected language in the thread; if still unknown, default to Spanish.", "Replace {PLACEHOLDERS} before sending." ], "placeholders": { @@ -242,4 +242,3 @@ } ] } - diff --git a/sergio_instagram_messaging/igdm_shadow.py b/sergio_instagram_messaging/igdm_shadow.py new file mode 100644 index 0000000..dc38719 --- /dev/null +++ b/sergio_instagram_messaging/igdm_shadow.py @@ -0,0 +1,1139 @@ +from __future__ import annotations + +import hashlib +import json +import os +import re +import sqlite3 +import threading +import time +import uuid +from dataclasses import dataclass +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Literal, TypedDict + +try: + from zoneinfo import ZoneInfo # py>=3.9 +except Exception: # pragma: no cover + ZoneInfo = None # type: ignore + + +CET_TZ = "Europe/Paris" + + +class TriageLanguage(TypedDict): + code: str + confidence: float + source: str + + +class TriageIntent(TypedDict): + label: str + confidence: float + + +class TriageRisk(TypedDict): + tier: str + score: float + reasons: list[str] + panel_size: int + + +class TriageResult(TypedDict): + triage_version: str + trace_id: str + ts_utc: str + time_cet: str + user_id: str + mid: str + language: TriageLanguage + intent: TriageIntent + risk: TriageRisk + + +class DraftResult(TypedDict): + draft_version: str + trace_id: str + template_id: str + text: str + placeholders: list[str] + notes: list[str] + + +class SeatResult(TypedDict): + seat: str + vote: str + severity: float + reasons: list[str] + patches: list[dict[str, Any]] + + +class PanelResult(TypedDict): + panel_version: str + trace_id: str + panel_size: int + seats: list[SeatResult] + decision: str + final_draft_text: str + final_draft_text_sha256: str + reason_summary: str + + +def _now_utc_iso() -> str: + return datetime.now(timezone.utc).replace(microsecond=0).isoformat() + + +def _sha256_text(text: str) -> str: + return hashlib.sha256(text.encode("utf-8", errors="ignore")).hexdigest() + + +def _cet_iso_from_ts_ms(ts_ms: int | None) -> str: + if ts_ms is None: + ts_ms = int(time.time() * 1000) + dt_utc = datetime.fromtimestamp(ts_ms / 1000.0, tz=timezone.utc) + if ZoneInfo is None: + return dt_utc.isoformat() + try: + dt_cet = dt_utc.astimezone(ZoneInfo(CET_TZ)) + return dt_cet.replace(microsecond=0).isoformat() + except Exception: + return dt_utc.replace(microsecond=0).isoformat() + + +def _utc_iso_from_ts_ms(ts_ms: int | None) -> str: + if ts_ms is None: + ts_ms = int(time.time() * 1000) + dt_utc = datetime.fromtimestamp(ts_ms / 1000.0, tz=timezone.utc) + return dt_utc.replace(microsecond=0).isoformat() + + +def meta_signature_is_valid(raw_body: bytes, signature_header: str, app_secret: str) -> bool: + import hmac + + if not app_secret or not signature_header: + return False + sig = signature_header.strip().lower() + expected = "sha256=" + hmac.new(app_secret.encode("utf-8"), raw_body, hashlib.sha256).hexdigest() + return hmac.compare_digest(expected, sig) + + +def _normalize_space(text: str) -> str: + return re.sub(r"\s+", " ", text.replace("\u00a0", " ")).strip() + + +_RE_WORD = re.compile(r"[A-Za-zÀ-ÖØ-öø-ÿ]+", re.UNICODE) + + +def _tokenize_words(text: str) -> list[str]: + return [m.group(0).lower() for m in _RE_WORD.finditer(text or "")] + + +STOPWORDS: dict[str, set[str]] = { + "en": { + "the", + "and", + "or", + "but", + "to", + "of", + "in", + "for", + "with", + "you", + "your", + "i", + "me", + "my", + "is", + "are", + "it", + "this", + "that", + "what", + "how", + "much", + "book", + "link", + "video", + }, + "es": { + "el", + "la", + "los", + "las", + "y", + "o", + "pero", + "para", + "de", + "en", + "con", + "tu", + "tus", + "yo", + "me", + "mi", + "es", + "son", + "esto", + "que", + "como", + "cuanto", + "cuánto", + "libro", + "enlace", + "video", + "vídeo", + }, + "fr": { + "le", + "la", + "les", + "et", + "ou", + "mais", + "pour", + "de", + "des", + "en", + "avec", + "tu", + "ton", + "ta", + "tes", + "je", + "moi", + "mon", + "c", + "est", + "ce", + "quoi", + "comment", + "combien", + "livre", + "lien", + "vidéo", + "video", + }, + "ca": { + "el", + "la", + "els", + "les", + "i", + "o", + "però", + "pero", + "per", + "de", + "del", + "a", + "en", + "amb", + "tu", + "teu", + "teva", + "jo", + "mi", + "m", + "és", + "es", + "això", + "que", + "com", + "quant", + "llibre", + "enllaç", + "video", + "vídeo", + }, +} + + +def detect_language(text: str, *, thread_fallback: str | None) -> tuple[str, float, str]: + cleaned = _normalize_space(text or "") + if len(cleaned) < 8: + explicit = _explicit_language_choice(cleaned) + if explicit: + return explicit, 0.99, "explicit" + if thread_fallback: + return thread_fallback, 0.55, "thread" + quick = _quick_language_from_keyword(cleaned) + if quick: + return quick, 0.7, "keyword" + # Default to Spanish for short/unclear messages; correct on the next user message if needed. + return "es", 0.55, "default" + + tokens = _tokenize_words(cleaned) + if not tokens: + if thread_fallback: + return thread_fallback, 0.55, "thread" + return "und", 0.0, "none" + + scores: dict[str, int] = {k: 0 for k in STOPWORDS} + for tok in tokens[:80]: + for lang, sw in STOPWORDS.items(): + if tok in sw: + scores[lang] += 1 + + best_lang = max(scores.items(), key=lambda kv: kv[1])[0] + best = scores[best_lang] + total = sum(scores.values()) + + if best == 0: + if thread_fallback: + return thread_fallback, 0.55, "thread" + # Default to Spanish when the message doesn't carry enough language signal. + return "es", 0.55, "default" + + conf = float(best) / float(max(1, total)) + return best_lang, min(0.99, max(0.5, conf)), "text" + + +def _quick_language_from_keyword(cleaned_text: str) -> str | None: + t = _normalize_space(cleaned_text).lower() + if not t: + return None + # Emoji flags (very short, high signal). + if "🇪🇸" in t: + return "es" + if "🇫🇷" in t: + return "fr" + if "🇬🇧" in t or "🇺🇸" in t: + return "en" + # Extremely short, high-signal keywords. + if t == "libro": + return "es" + if t == "livre": + return "fr" + if t == "llibre": + return "ca" + if t == "hola": + return "es" + if t == "bonjour" or t == "salut": + return "fr" + if t == "merci": + return "fr" + if t == "gracias": + return "es" + if t == "gràcies" or t == "gracies": + return "ca" + if t == "enlace": + return "es" + if t == "lien": + return "fr" + if t in ("enllaç", "enllac"): + return "ca" + if t == "precio": + return "es" + if t in ("prix",): + return "fr" + if t in ("quant",): + return "ca" + # Accent-sensitive clue for video. + if t == "vidéo": + return "fr" + return None + + +def _explicit_language_choice(cleaned_text: str) -> str | None: + t = _normalize_space(cleaned_text).lower() + if not t: + return None + # Handle explicit user preference replies after we ask "English / Español / Français / Català?" + # Keep this conservative: only trigger when the user's message is basically just the language. + if t in {"en", "eng", "english"}: + return "en" + if t in {"es", "spa", "spanish", "espanol", "español", "castellano"}: + return "es" + if t in {"fr", "fra", "french", "francais", "français"}: + return "fr" + if t in {"ca", "cat", "catalan", "catala", "català"}: + return "ca" + # “Please” variants (still explicit enough). + if t in {"english please", "en please"}: + return "en" + if t in {"español por favor", "espanol por favor", "spanish please"}: + return "es" + if t in {"français svp", "francais svp", "french please"}: + return "fr" + if t in {"català si us plau", "catala si us plau", "catalan please"}: + return "ca" + return None + + +INTENT_PATTERNS: list[tuple[str, re.Pattern[str]]] = [ + ("book", re.compile(r"\b(book|ebook|e-book|libro|livre|llibre)\b", re.I)), + ("video", re.compile(r"\b(video|vídeo|vidéo)\b", re.I)), + ("link", re.compile(r"\b(link|enlace|lien|enllaç)\b", re.I)), + ("price", re.compile(r"(\bprice\b|\bcost\b|\bhow much\b|[$€]|precio|cu[aá]nto|cuesta|prix|combien|quant)", re.I)), + ("call", re.compile(r"\b(call|zoom|calendly|llamada|appel|trucada)\b", re.I)), + ("whatsapp", re.compile(r"\bwhats ?app\b", re.I)), + ("therapy", re.compile(r"\b(therapy|therapist|terapia|th[ée]rapie)\b", re.I)), + ("help", re.compile(r"\b(help|ayuda|aide|ajuda)\b", re.I)), +] + + +def detect_intent(text: str) -> tuple[str, float]: + cleaned = _normalize_space(text or "").lower() + if not cleaned: + return "other", 0.0 + for label, pat in INTENT_PATTERNS: + if pat.search(cleaned): + return label, 0.9 + if "?" in cleaned: + return "other", 0.55 + return "other", 0.25 + + +URGENT_PATTERNS: list[re.Pattern[str]] = [ + re.compile(r"\b(suicide|kill myself|end my life|self[- ]?harm|cut myself|overdose)\b", re.I), + re.compile(r"\b(me quiero morir|quiero morir|me voy a matar|suicid(ar|io))\b", re.I), + re.compile(r"\b(je veux mourir|me tuer|suicide)\b", re.I), + re.compile(r"\b(em vull morir|vull morir|su[iï]cidi)\b", re.I), +] + +URGENT_EXEMPTIONS: list[re.Pattern[str]] = [ + re.compile(r"\b(dying laughing|dead lol)\b", re.I), + re.compile(r"\b(me muero de risa)\b", re.I), +] + + +def assess_risk(text: str) -> tuple[str, float, list[str]]: + cleaned = _normalize_space(text or "") + if not cleaned: + return "normal", 0.05, [] + + low = cleaned.lower() + for ex in URGENT_EXEMPTIONS: + if ex.search(low): + return "normal", 0.05, ["exempt_slang"] + + for pat in URGENT_PATTERNS: + if pat.search(low): + return "urgent", 1.0, ["self_harm_signal"] + + # Needs-human heuristics: long disclosure + emotion words, legal threats, harassment loop. + reasons: list[str] = [] + score = 0.05 + if len(cleaned) >= 500: + score = max(score, 0.6) + reasons.append("long_message") + if re.search(r"\b(sc[aá]m|fraud|police|lawyer|denounce|denunciar|demand)\b", low): + score = max(score, 0.6) + reasons.append("legal_threat_or_scam") + if re.search(r"\b(fuck|puta|idiot|moron|asshole|mierda)\b", low): + score = max(score, 0.6) + reasons.append("abusive_language") + if re.search(r"\b(depress(ed|ion)|panic|anxiety|trauma|abuse|violence)\b", low): + score = max(score, 0.6) + reasons.append("mental_health_disclosure") + + tier: str + if score >= 0.9: + tier = "urgent" + elif score >= 0.55: + tier = "needs-human" + else: + tier = "normal" + return tier, score, reasons + + +def panel_size_from_risk_score(score: float) -> int: + if score >= 0.9: + return 20 + if score >= 0.55: + return 10 + return 5 + + +@dataclass(frozen=True) +class ReplyLibrary: + replies_by_id: dict[str, dict[str, str]] + + @classmethod + def load(cls, path: str) -> "ReplyLibrary": + data = json.loads(Path(path).read_text(encoding="utf-8")) + replies_by_id: dict[str, dict[str, str]] = {} + for item in data.get("replies") or []: + rid = str(item.get("reply_id") or "").strip() + by_lang = item.get("replies_by_language") or {} + if not rid or not isinstance(by_lang, dict): + continue + replies_by_id[rid] = {str(k): str(v) for k, v in by_lang.items()} + return cls(replies_by_id=replies_by_id) + + +LANG_NAME_BY_CODE = {"en": "English", "es": "Spanish", "fr": "French", "ca": "Catalan", "und": "Spanish"} + + +INTENT_TO_REPLY_ID: dict[str, str] = { + "book": "book_keyword", + "video": "send_video", + "link": "send_link", + "price": "price", + "call": "book_call", + "whatsapp": "whatsapp", + "therapy": "therapy", + "help": "help_request", + "other": "other_question", +} + + +def _extract_placeholders(text: str) -> list[str]: + return sorted(set(re.findall(r"\{([A-Z0-9_]+)\}", text or ""))) + + +def _replace_placeholders(text: str, *, values: dict[str, str]) -> str: + def repl(m: re.Match[str]) -> str: + key = m.group(1) or "" + return values.get(key, m.group(0)) + + return re.sub(r"\{([A-Z0-9_]+)\}", repl, text or "") + + +def _placeholder_values_from_env() -> dict[str, str]: + # Prefer IGDM_* overrides; fall back to generic names. + mapping = { + "BOOK_LINK": os.getenv("IGDM_BOOK_LINK") or os.getenv("BOOK_LINK") or "", + "VIDEO_LINK": os.getenv("IGDM_VIDEO_LINK") or os.getenv("VIDEO_LINK") or "", + "CALENDLY_LINK": os.getenv("IGDM_CALENDLY_LINK") or os.getenv("CALENDLY_LINK") or "", + "WHATSAPP_LINK": os.getenv("IGDM_WHATSAPP_LINK") or os.getenv("WHATSAPP_LINK") or "", + "PRICE": os.getenv("IGDM_PRICE") or os.getenv("PRICE") or "", + "TIMEFRAME": os.getenv("IGDM_TIMEFRAME") or os.getenv("TIMEFRAME") or "", + "BASE_LOCATION": os.getenv("IGDM_BASE_LOCATION") or os.getenv("BASE_LOCATION") or "", + } + return {k: v for k, v in mapping.items() if v} + + +def draft_reply( + *, + reply_lib: ReplyLibrary, + trace_id: str, + language_code: str, + intent_label: str, + language_confidence: float, +) -> DraftResult: + if language_code == "und" or language_confidence < 0.5: + # Avoid a language menu (it feels bot-like). Prefer: + # - language-neutral value-first replies (links) + # - a short intent clarifier (icons) when needed + # - a short "what do you need?" fallback to elicit language naturally + values = _placeholder_values_from_env() + + if intent_label == "book": + base = "👇 {BOOK_LINK}" + text = _replace_placeholders(base, values=values) + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:book_link_only", + "text": text, + "placeholders": _extract_placeholders(base), + "notes": ["language=und", "intent=book"], + } + + if intent_label == "video": + base = "👇 {VIDEO_LINK}" + text = _replace_placeholders(base, values=values) + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:video_link_only", + "text": text, + "placeholders": _extract_placeholders(base), + "notes": ["language=und", "intent=video"], + } + + if intent_label in ("link", "price", "call"): + # Keep this looking human: no emoji menus. + if intent_label == "call": + if "CALENDLY_LINK" in values: + base = "Aquí puedes reservar una llamada: {CALENDLY_LINK}" + text = _replace_placeholders(base, values=values) + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:call_link_only", + "text": text, + "placeholders": _extract_placeholders(base), + "notes": ["language=und", "intent=call"], + } + text = "¿Quieres reservar una llamada?" + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:call_clarify", + "text": text, + "placeholders": [], + "notes": ["language=und", "intent=call"], + } + + if intent_label == "price": + if "PRICE" in values: + base = "El libro cuesta {PRICE}." + if "BOOK_LINK" in values: + base += " Enlace: {BOOK_LINK}" + text = _replace_placeholders(base, values=values) + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:price_book_default", + "text": text, + "placeholders": _extract_placeholders(base), + "notes": ["language=und", "intent=price"], + } + text = "¿De qué necesitas el precio?" + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:price_clarify", + "text": text, + "placeholders": [], + "notes": ["language=und", "intent=price"], + } + + # intent_label == "link" + if "BOOK_LINK" in values: + base = "Aquí tienes el enlace: {BOOK_LINK}" + text = _replace_placeholders(base, values=values) + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:book_link_default", + "text": text, + "placeholders": _extract_placeholders(base), + "notes": ["language=und", "intent=link"], + } + text = "¿Qué enlace necesitas?" + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:link_clarify", + "text": text, + "placeholders": [], + "notes": ["language=und", "intent=link"], + } + + # Generic fallback: keep it short, and default to Spanish. + text = "¿Qué necesitas?" + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": "system:clarify_need", + "text": text, + "placeholders": [], + "notes": ["language=und", f"intent={intent_label}"], + } + + lang_name = LANG_NAME_BY_CODE.get(language_code, "English") + rid = INTENT_TO_REPLY_ID.get(intent_label, "other_question") + by_lang = reply_lib.replies_by_id.get(rid) or {} + base = by_lang.get(lang_name) or by_lang.get("English") or "" + if not base: + base = "Tell me in one sentence what you need, and I’ll point you to the right next step." + + values = _placeholder_values_from_env() + filled = _replace_placeholders(base, values=values) + return { + "draft_version": "igdm.draft/v1", + "trace_id": trace_id, + "template_id": f"top20:{rid}:{lang_name}", + "text": filled, + "placeholders": _extract_placeholders(base), + "notes": [f"language={language_code}", f"intent={intent_label}"], + } + + +def _seat_language(triage: TriageResult, draft: DraftResult) -> SeatResult: + lang = triage["language"] + if lang["code"] == "und" or lang["confidence"] < 0.5: + # If language is unclear, do not force a language menu. Just ensure the draft stays short. + t = (draft.get("text") or "").strip() + if len(t) > 120: + return { + "seat": "language", + "vote": "request_changes", + "severity": 0.25, + "reasons": ["User language unclear; keep the draft short to avoid a wrong-language paragraph."], + "patches": [{"op": "replace_text", "path": "draft.text", "value": "¿Qué necesitas?"}], + } + return {"seat": "language", "vote": "approve", "severity": 0.0, "reasons": [], "patches": []} + return {"seat": "language", "vote": "approve", "severity": 0.0, "reasons": [], "patches": []} + + +def _seat_boundary(triage: TriageResult, draft: DraftResult) -> SeatResult: + intent = triage["intent"]["label"] + if intent == "therapy": + # Ensure we don't imply therapy-by-DM. + if re.search(r"\btherapy\b|\bterapia\b|\bth[ée]rapie\b", draft["text"], re.I) and "isn’t therapy" in draft["text"]: + return {"seat": "boundary", "vote": "approve", "severity": 0.0, "reasons": [], "patches": []} + return { + "seat": "boundary", + "vote": "request_changes", + "severity": 0.4, + "reasons": ["Therapy intent: reply must clearly state this is not therapy in DMs."], + "patches": [ + { + "op": "replace_text", + "path": "draft.text", + "value": "No — this isn’t therapy in DMs. I can share resources and options. What are you looking for?", + } + ], + } + return {"seat": "boundary", "vote": "approve", "severity": 0.0, "reasons": [], "patches": []} + + +def _seat_privacy(_: TriageResult, draft: DraftResult) -> SeatResult: + # Allow minimal PII asks only when needed (refund / receipt / delivery issues). + if re.search(r"\bemail\b|\be-?mail\b", draft["text"], re.I): + return { + "seat": "privacy", + "vote": "request_changes", + "severity": 0.2, + "reasons": ["If you ask for an email, keep it strictly necessary and brief."], + "patches": [], + } + return {"seat": "privacy", "vote": "approve", "severity": 0.0, "reasons": [], "patches": []} + + +def _seat_next_step(_: TriageResult, draft: DraftResult) -> SeatResult: + t = (draft.get("text") or "").strip() + if not t: + return {"seat": "next_step", "vote": "veto", "severity": 1.0, "reasons": ["Empty draft."], "patches": []} + # A clear imperative is an acceptable next-step even without a question mark. + if re.match(r"^(tell me|dime|dis[\u2011-]?moi|digue['\u2019]?m|digues)", t, re.I): + return {"seat": "next_step", "vote": "approve", "severity": 0.0, "reasons": [], "patches": []} + if "?" in t: + return {"seat": "next_step", "vote": "approve", "severity": 0.0, "reasons": [], "patches": []} + # Add a short question to keep the conversation moving (match user language when possible). + lang_code = str(_.get("language", {}).get("code") or "en") + lang_tail = { + "en": " What are you looking for right now?", + "es": " ¿Qué estás buscando ahora mismo?", + "fr": " Tu cherches quoi en ce moment ?", + "ca": " Què busques ara mateix?", + }.get(lang_code, " What are you looking for right now?") + return { + "seat": "next_step", + "vote": "request_changes", + "severity": 0.2, + "reasons": ["Draft should end with one clear next question."], + "patches": [{"op": "append_text", "path": "draft.text", "value": lang_tail}], + } + + +SEATS_MIN: list[tuple[str, Any]] = [ + ("language", _seat_language), + ("boundary", _seat_boundary), + ("privacy", _seat_privacy), + ("next_step", _seat_next_step), +] + + +def run_panel(triage: TriageResult, draft: DraftResult) -> PanelResult: + seats: list[SeatResult] = [] + decision: str = "approve_draft" + reasons: list[str] = [] + final_text = draft.get("text") or "" + + # Veto on triage tiers first. + tier = triage["risk"]["tier"] + if tier == "urgent": + decision = "urgent_escalate" + reasons.append("urgent") + elif tier == "needs-human": + decision = "escalate_human" + reasons.append("needs-human") + + # Seat evaluation. + for _, fn in SEATS_MIN: + try: + res: SeatResult = fn(triage, draft) + except Exception as e: + res = { + "seat": "internal_error", + "vote": "request_changes", + "severity": 0.4, + "reasons": [f"Seat error: {e}"], + "patches": [], + } + seats.append(res) + + if decision in ("urgent_escalate", "escalate_human"): + return { + "panel_version": "if.gov.panel/igdm/v1", + "trace_id": triage["trace_id"], + "panel_size": triage["risk"]["panel_size"], + "seats": seats, + "decision": decision, + "final_draft_text": final_text, + "final_draft_text_sha256": _sha256_text(final_text), + "reason_summary": ", ".join(reasons) if reasons else decision, + } + + # Apply requested patches once (deterministic order). + patched_text = final_text + any_veto = any(s.get("vote") == "veto" for s in seats) + if any_veto: + decision = "escalate_human" + reasons.append("seat_veto") + else: + for s in seats: + if s.get("vote") != "request_changes": + continue + for p in s.get("patches") or []: + op = p.get("op") + if op == "replace_text" and p.get("path") == "draft.text": + patched_text = str(p.get("value") or "") + elif op == "append_text" and p.get("path") == "draft.text": + patched_text = patched_text + str(p.get("value") or "") + + any_changes = patched_text != (draft.get("text") or "") + if any_changes: + decision = "revise_draft" + reasons.append("seat_patches") + + return { + "panel_version": "if.gov.panel/igdm/v1", + "trace_id": triage["trace_id"], + "panel_size": triage["risk"]["panel_size"], + "seats": seats, + "decision": decision, + "final_draft_text": patched_text, + "final_draft_text_sha256": _sha256_text(patched_text), + "reason_summary": ", ".join(reasons) if reasons else decision, + } + + +class IgdmStore: + def __init__(self, db_path: str): + self._db_path = db_path + self._lock = threading.Lock() + self._init_db() + + def _connect(self) -> sqlite3.Connection: + conn = sqlite3.connect(self._db_path, check_same_thread=False) + conn.row_factory = sqlite3.Row + conn.execute("PRAGMA journal_mode=WAL;") + conn.execute("PRAGMA foreign_keys=ON;") + return conn + + def _init_db(self) -> None: + Path(self._db_path).parent.mkdir(parents=True, exist_ok=True) + with self._connect() as conn: + conn.executescript( + """ + CREATE TABLE IF NOT EXISTS igdm_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + received_at_utc TEXT NOT NULL, + user_id TEXT NOT NULL, + sender_id TEXT, + recipient_id TEXT, + mid TEXT, + ts_ms INTEGER, + direction TEXT NOT NULL, + is_echo INTEGER NOT NULL, + text TEXT, + raw_json TEXT + ); + CREATE UNIQUE INDEX IF NOT EXISTS igdm_events_mid_idx ON igdm_events(mid) WHERE mid IS NOT NULL; + + CREATE TABLE IF NOT EXISTS igdm_thread_state ( + user_id TEXT PRIMARY KEY, + last_language_code TEXT, + last_language_confidence REAL, + updated_at_utc TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS igdm_drafts ( + trace_id TEXT PRIMARY KEY, + created_at_utc TEXT NOT NULL, + time_cet TEXT NOT NULL, + user_id TEXT NOT NULL, + inbound_mid TEXT, + inbound_event_id INTEGER, + language_code TEXT, + language_confidence REAL, + intent_label TEXT, + intent_confidence REAL, + risk_tier TEXT, + risk_score REAL, + template_id TEXT, + draft_text TEXT, + draft_text_sha256 TEXT, + panel_decision TEXT, + reason_summary TEXT, + triage_json TEXT, + draft_json TEXT, + panel_json TEXT, + matched_outbound_event_id INTEGER, + matched_at_utc TEXT, + FOREIGN KEY(inbound_event_id) REFERENCES igdm_events(id), + FOREIGN KEY(matched_outbound_event_id) REFERENCES igdm_events(id) + ); + """ + ) + + def upsert_thread_language(self, user_id: str, code: str, confidence: float) -> None: + if not user_id: + return + now = _now_utc_iso() + with self._lock, self._connect() as conn: + conn.execute( + """ + INSERT INTO igdm_thread_state (user_id, last_language_code, last_language_confidence, updated_at_utc) + VALUES (?, ?, ?, ?) + ON CONFLICT(user_id) DO UPDATE SET + last_language_code=excluded.last_language_code, + last_language_confidence=excluded.last_language_confidence, + updated_at_utc=excluded.updated_at_utc + """, + (user_id, code, float(confidence), now), + ) + + def get_thread_language(self, user_id: str) -> str | None: + if not user_id: + return None + with self._lock, self._connect() as conn: + row = conn.execute( + "SELECT last_language_code, last_language_confidence FROM igdm_thread_state WHERE user_id=?", + (user_id,), + ).fetchone() + if not row: + return None + code = str(row["last_language_code"] or "").strip() + conf = float(row["last_language_confidence"] or 0.0) + if not code or conf < 0.5: + return None + return code + + def insert_event( + self, + *, + user_id: str, + sender_id: str | None, + recipient_id: str | None, + mid: str | None, + ts_ms: int | None, + direction: Literal["inbound", "outbound"], + is_echo: bool, + text: str | None, + raw_event: dict[str, Any], + ) -> int: + now = _now_utc_iso() + raw_json = json.dumps(raw_event, ensure_ascii=False) + with self._lock, self._connect() as conn: + conn.execute( + """ + INSERT OR IGNORE INTO igdm_events + (received_at_utc, user_id, sender_id, recipient_id, mid, ts_ms, direction, is_echo, text, raw_json) + VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + now, + user_id, + sender_id, + recipient_id, + mid, + ts_ms, + direction, + 1 if is_echo else 0, + text, + raw_json, + ), + ) + if mid: + row = conn.execute("SELECT id FROM igdm_events WHERE mid=?", (mid,)).fetchone() + if row: + return int(row["id"]) + row = conn.execute("SELECT last_insert_rowid() AS id").fetchone() + return int(row["id"]) if row and row["id"] else 0 + + def insert_draft( + self, + *, + triage: TriageResult, + draft: DraftResult, + panel: PanelResult, + inbound_event_id: int, + ) -> None: + trace_id = triage["trace_id"] + created = triage["ts_utc"] + time_cet = triage["time_cet"] + user_id = triage["user_id"] + mid = triage["mid"] + lang = triage["language"] + intent = triage["intent"] + risk = triage["risk"] + text = str(panel.get("final_draft_text") or draft["text"]) + + with self._lock, self._connect() as conn: + conn.execute( + """ + INSERT OR REPLACE INTO igdm_drafts ( + trace_id, created_at_utc, time_cet, user_id, inbound_mid, inbound_event_id, + language_code, language_confidence, intent_label, intent_confidence, + risk_tier, risk_score, template_id, draft_text, draft_text_sha256, + panel_decision, reason_summary, triage_json, draft_json, panel_json, + matched_outbound_event_id, matched_at_utc + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL, NULL) + """, + ( + trace_id, + created, + time_cet, + user_id, + mid, + inbound_event_id, + lang["code"], + float(lang["confidence"]), + intent["label"], + float(intent["confidence"]), + risk["tier"], + float(risk["score"]), + draft["template_id"], + text, + _sha256_text(text), + panel["decision"], + panel["reason_summary"], + json.dumps(triage, ensure_ascii=False), + json.dumps(draft, ensure_ascii=False), + json.dumps(panel, ensure_ascii=False), + ), + ) + + def link_outbound_to_latest_draft(self, *, user_id: str, outbound_event_id: int, max_age_s: int = 72 * 3600) -> str | None: + if not user_id or not outbound_event_id: + return None + cutoff = datetime.now(timezone.utc).timestamp() - float(max_age_s) + cutoff_iso = datetime.fromtimestamp(cutoff, tz=timezone.utc).replace(microsecond=0).isoformat() + now = _now_utc_iso() + with self._lock, self._connect() as conn: + row = conn.execute( + """ + SELECT trace_id FROM igdm_drafts + WHERE user_id=? + AND matched_outbound_event_id IS NULL + AND created_at_utc >= ? + ORDER BY created_at_utc DESC + LIMIT 1 + """, + (user_id, cutoff_iso), + ).fetchone() + if not row: + return None + trace_id = str(row["trace_id"]) + conn.execute( + """ + UPDATE igdm_drafts + SET matched_outbound_event_id=?, matched_at_utc=? + WHERE trace_id=? AND matched_outbound_event_id IS NULL + """, + (outbound_event_id, now, trace_id), + ) + return trace_id + + def list_items(self, *, limit: int = 100) -> list[dict[str, Any]]: + limit = max(1, min(500, int(limit))) + with self._lock, self._connect() as conn: + rows = conn.execute( + """ + SELECT + d.trace_id, + d.created_at_utc, + d.time_cet, + d.user_id, + d.intent_label, + d.risk_tier, + d.panel_decision, + d.template_id, + d.draft_text, + d.matched_at_utc, + e_out.text AS actual_text + FROM igdm_drafts d + LEFT JOIN igdm_events e_out ON e_out.id = d.matched_outbound_event_id + ORDER BY d.created_at_utc DESC + LIMIT ? + """, + (limit,), + ).fetchall() + out: list[dict[str, Any]] = [] + for r in rows: + out.append({k: r[k] for k in r.keys()}) + return out + + def get_item(self, trace_id: str) -> dict[str, Any] | None: + tid = str(trace_id or "").strip() + if not tid: + return None + with self._lock, self._connect() as conn: + row = conn.execute( + """ + SELECT + d.*, + e_in.text AS inbound_text, + e_out.text AS actual_text + FROM igdm_drafts d + LEFT JOIN igdm_events e_in ON e_in.id = d.inbound_event_id + LEFT JOIN igdm_events e_out ON e_out.id = d.matched_outbound_event_id + WHERE d.trace_id=? + """, + (tid,), + ).fetchone() + if not row: + return None + out = {k: row[k] for k in row.keys()} + # JSON decode best-effort for callers. + for key in ("triage_json", "draft_json", "panel_json"): + try: + if out.get(key): + out[key] = json.loads(out[key]) + except Exception: + pass + return out + + +def process_inbound_message( + *, + store: IgdmStore, + reply_lib: ReplyLibrary, + user_id: str, + mid: str, + ts_ms: int | None, + text: str, + inbound_event_id: int, +) -> tuple[TriageResult, DraftResult, PanelResult]: + trace_id = str(uuid.uuid4()) + thread_lang = store.get_thread_language(user_id) + lang_code, lang_conf, lang_src = detect_language(text, thread_fallback=thread_lang) + if lang_code != "und": + store.upsert_thread_language(user_id, lang_code, lang_conf) + intent_label, intent_conf = detect_intent(text) + tier, risk_score, reasons = assess_risk(text) + panel_size = panel_size_from_risk_score(risk_score) + + triage: TriageResult = { + "triage_version": "if.gov.triage/igdm/v1", + "trace_id": trace_id, + "ts_utc": _utc_iso_from_ts_ms(ts_ms), + "time_cet": _cet_iso_from_ts_ms(ts_ms), + "user_id": str(user_id), + "mid": str(mid), + "language": {"code": lang_code, "confidence": float(lang_conf), "source": lang_src}, + "intent": {"label": intent_label, "confidence": float(intent_conf)}, + "risk": {"tier": tier, "score": float(risk_score), "reasons": reasons, "panel_size": int(panel_size)}, + } + + draft = draft_reply( + reply_lib=reply_lib, + trace_id=trace_id, + language_code=lang_code, + intent_label=intent_label, + language_confidence=lang_conf, + ) + panel = run_panel(triage, draft) + + store.insert_draft(triage=triage, draft=draft, panel=panel, inbound_event_id=inbound_event_id) + return triage, draft, panel diff --git a/sergio_instagram_messaging/igdm_shadow_server.py b/sergio_instagram_messaging/igdm_shadow_server.py new file mode 100644 index 0000000..e79c5cf --- /dev/null +++ b/sergio_instagram_messaging/igdm_shadow_server.py @@ -0,0 +1,745 @@ +from __future__ import annotations + +import argparse +import html +import json +import os +import secrets +import sys +import threading +import time +import urllib.error +import urllib.parse +import urllib.request +from http import HTTPStatus +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any + +from .igdm_shadow import ( + IgdmStore, + ReplyLibrary, + meta_signature_is_valid, + process_inbound_message, +) + + +def _env(name: str, default: str = "") -> str: + return str(os.getenv(name, default) or "").strip() + + +def _read_json_body(handler: BaseHTTPRequestHandler) -> tuple[bytes, dict[str, Any] | None]: + try: + length = int(handler.headers.get("Content-Length") or "0") + except Exception: + length = 0 + raw = handler.rfile.read(max(0, length)) if length else b"" + try: + payload = json.loads(raw.decode("utf-8") or "{}") + if not isinstance(payload, dict): + return raw, None + return raw, payload + except Exception: + return raw, None + + +def _json(handler: BaseHTTPRequestHandler, status: int, obj: Any) -> None: + data = json.dumps(obj, ensure_ascii=False, indent=2).encode("utf-8") + handler.send_response(status) + handler.send_header("Content-Type", "application/json; charset=utf-8") + handler.send_header("Content-Length", str(len(data))) + handler.end_headers() + handler.wfile.write(data) + + +def _text(handler: BaseHTTPRequestHandler, status: int, body: str, *, content_type: str = "text/plain; charset=utf-8") -> None: + data = body.encode("utf-8") + handler.send_response(status) + handler.send_header("Content-Type", content_type) + handler.send_header("Content-Length", str(len(data))) + handler.end_headers() + handler.wfile.write(data) + + +def _html(handler: BaseHTTPRequestHandler, body: str) -> None: + _text(handler, 200, body, content_type="text/html; charset=utf-8") + + +def _redirect(handler: BaseHTTPRequestHandler, url: str, *, status: int = 302) -> None: + handler.send_response(status) + handler.send_header("Location", url) + handler.send_header("Content-Type", "text/plain; charset=utf-8") + handler.send_header("Content-Length", "0") + handler.end_headers() + + +def _safe_write_json(path: Path, obj: dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + data = json.dumps(obj, ensure_ascii=False, indent=2).encode("utf-8") + fd = os.open(str(tmp), os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600) + try: + with os.fdopen(fd, "wb") as f: + f.write(data) + f.flush() + os.fsync(f.fileno()) + finally: + try: + os.close(fd) + except Exception: + pass + os.replace(tmp, path) + + +def _read_json_file(path: Path) -> dict[str, Any] | None: + try: + data = path.read_text("utf-8") + except FileNotFoundError: + return None + except Exception: + return None + try: + obj = json.loads(data) + except Exception: + return None + return obj if isinstance(obj, dict) else None + + +def _http_json(method: str, url: str, *, data: bytes | None = None, headers: dict[str, str] | None = None) -> tuple[int, dict[str, Any] | None, str]: + req = urllib.request.Request(url, data=data, method=method.upper()) + req.add_header("Accept", "application/json") + if headers: + for k, v in headers.items(): + req.add_header(k, v) + if data is not None: + req.add_header("Content-Type", "application/x-www-form-urlencoded") + try: + with urllib.request.urlopen(req, timeout=15) as resp: # noqa: S310 + status = int(getattr(resp, "status", 200)) + raw = resp.read().decode("utf-8", errors="ignore") + except urllib.error.HTTPError as e: # noqa: PERF203 + status = int(getattr(e, "code", 500) or 500) + raw = (e.read() or b"").decode("utf-8", errors="ignore") + except Exception as e: + return 0, None, str(e) + try: + payload = json.loads(raw) if raw else None + except Exception: + payload = None + return status, (payload if isinstance(payload, dict) else None), raw + + +def _ig_authorize_url(*, app_id: str, redirect_uri: str, scopes: str, state: str) -> str: + q = urllib.parse.urlencode( + { + "client_id": app_id, + "redirect_uri": redirect_uri, + "scope": scopes, + "response_type": "code", + "state": state, + } + ) + return f"https://www.instagram.com/oauth/authorize?{q}" + + +def _ig_exchange_code_for_short_token(*, app_id: str, app_secret: str, redirect_uri: str, code: str) -> tuple[dict[str, Any] | None, str | None]: + body = urllib.parse.urlencode( + { + "client_id": app_id, + "client_secret": app_secret, + "grant_type": "authorization_code", + "redirect_uri": redirect_uri, + "code": code, + } + ).encode("utf-8") + status, payload, raw = _http_json("POST", "https://api.instagram.com/oauth/access_token", data=body) + if status != 200 or payload is None: + return None, raw + return payload, None + + +def _ig_exchange_short_for_long(*, app_secret: str, short_token: str) -> tuple[dict[str, Any] | None, str | None]: + q = urllib.parse.urlencode( + { + "grant_type": "ig_exchange_token", + "client_secret": app_secret, + "access_token": short_token, + } + ) + status, payload, raw = _http_json("GET", f"https://graph.instagram.com/access_token?{q}") + if status != 200 or payload is None: + return None, raw + return payload, None + + +def _ig_subscribe_webhooks(*, api_version: str, access_token: str, subscribed_fields: str) -> tuple[dict[str, Any] | None, str | None]: + url = f"https://graph.instagram.com/{api_version}/me/subscribed_apps" + body = urllib.parse.urlencode({"subscribed_fields": subscribed_fields, "access_token": access_token}).encode("utf-8") + status, payload, raw = _http_json("POST", url, data=body) + if status != 200 or payload is None: + return None, raw + return payload, None + + +def _basic_dashboard_html() -> str: + return """ + + + + + IGDM Shadow Mode + + + +

IGDM Shadow Mode

+

Drafts are generated and stored, but never sent. Outgoing (echo) messages are captured later for comparison.

+

Instagram connection: checking…

+

Connect / Reconnect Instagram (Business Login)

+ + + + + + + + + + + + + + +
Time (CET)UserIntentDecisionDraftActual (captured)
+ + +
+
+ +
+
+
+
+
Inbound (user)
+ 

+          

+          

+            
Draft (not sent)

+            

+          

+        

+        

+        

+          

+            
Actual reply (captured)

+            

+          

+          

+            
Panel decision

+            

+          

+        

+      

+    
+
+    
+  
+"""
+
+
+class IgdmShadowHandler(BaseHTTPRequestHandler):
+    server_version = "igdm-shadow/0.1"
+
+    def do_GET(self) -> None:  # noqa: N802
+        parsed = urllib.parse.urlparse(self.path)
+        path = parsed.path or "/"
+        qs = urllib.parse.parse_qs(parsed.query or "")
+
+        if path == "/health":
+            _json(self, 200, {"ok": True, "service": "igdm-shadow"})
+            return
+
+        if path == "/meta/ig/connect":
+            app_id = self.server.cfg.get("meta_app_id") or ""
+            redirect_uri = self.server.cfg.get("ig_redirect_uri") or ""
+            scopes = self.server.cfg.get("ig_scopes") or ""
+            if not app_id or not redirect_uri:
+                _text(self, 500, "Missing META_APP_ID or IGDM_IG_REDIRECT_URI.")
+                return
+
+            state = secrets.token_urlsafe(32)
+            _safe_write_json(Path(self.server.cfg["ig_state_path"]), {"state": state, "created_at": int(time.time())})
+            _redirect(self, _ig_authorize_url(app_id=app_id, redirect_uri=redirect_uri, scopes=scopes, state=state))
+            return
+
+        if path == "/meta/ig/callback":
+            error = (qs.get("error") or [""])[0]
+            if error:
+                error_desc = (qs.get("error_description") or [""])[0]
+                _html(
+                    self,
+                    f"
Instagram connect failed
{html.escape(error)}\n{html.escape(error_desc)}
Back
",
+                )
+                return
+
+            code = (qs.get("code") or [""])[0]
+            state = (qs.get("state") or [""])[0]
+            if not code or not state:
+                _text(self, 400, "Missing code/state.")
+                return
+
+            state_obj = _read_json_file(Path(self.server.cfg["ig_state_path"])) or {}
+            state_expected = str(state_obj.get("state") or "")
+            created_at = int(state_obj.get("created_at") or 0)
+            if not state_expected or state != state_expected:
+                _text(self, 403, "Invalid state.")
+                return
+            if created_at and int(time.time()) - created_at > 15 * 60:
+                _text(self, 403, "State expired. Please retry connect.")
+                return
+
+            app_id = self.server.cfg.get("meta_app_id") or ""
+            redirect_uri = self.server.cfg.get("ig_redirect_uri") or ""
+            app_secret = self.server.cfg.get("app_secret") or ""
+            if not app_id or not redirect_uri:
+                _text(self, 500, "Missing META_APP_ID or IGDM_IG_REDIRECT_URI.")
+                return
+
+            short_payload, err = _ig_exchange_code_for_short_token(
+                app_id=app_id,
+                app_secret=app_secret,
+                redirect_uri=redirect_uri,
+                code=code,
+            )
+            if err or short_payload is None:
+                _html(self, f"
Instagram connect failed
{html.escape(err or '')}
Back
")
+                return
+
+            short_token = str(short_payload.get("access_token") or "")
+            ig_user_id = str(short_payload.get("user_id") or "")
+            if not short_token:
+                _html(self, "
Instagram connect failed
Missing short-lived token.
Back
")
+                return
+
+            long_payload, err = _ig_exchange_short_for_long(app_secret=app_secret, short_token=short_token)
+            if err or long_payload is None:
+                _html(self, f"
Instagram connect failed
{html.escape(err or '')}
Back
")
+                return
+
+            long_token = str(long_payload.get("access_token") or "")
+            expires_in_raw = long_payload.get("expires_in") or 0
+            expires_in = int(expires_in_raw) if str(expires_in_raw).isdigit() else 0
+            if not long_token:
+                _html(self, "
Instagram connect failed
Missing long-lived token.
Back
")
+                return
+
+            subscribed_fields = self.server.cfg.get("ig_subscribed_fields") or "messages"
+            sub_payload, sub_err = _ig_subscribe_webhooks(
+                api_version=self.server.cfg.get("graph_api_version") or "v24.0",
+                access_token=long_token,
+                subscribed_fields=subscribed_fields,
+            )
+
+            token_obj: dict[str, Any] = {
+                "created_at": int(time.time()),
+                "ig_user_id": ig_user_id or None,
+                "access_token": long_token,
+                "expires_in": expires_in or None,
+                "subscribed_fields": subscribed_fields,
+                "subscribed_at": int(time.time()) if sub_payload else None,
+                "subscribe_ok": bool(sub_payload),
+                "subscribe_error": sub_err if sub_err else None,
+                "subscribe_response": sub_payload if sub_payload else None,
+            }
+            _safe_write_json(Path(self.server.cfg["ig_token_path"]), token_obj)
+
+            if sub_payload is None:
+                _html(
+                    self,
+                    f"
Instagram connected, but webhook subscribe failed
{html.escape(sub_err or '')}
Back to IGDM
",
+                )
+                return
+
+            _html(self, "
Instagram connected
Webhook subscription enabled.
Back to IGDM
")
+            return
+
+        if path == "/meta/webhook":
+            expected = self.server.cfg["verify_token"]
+            mode = (qs.get("hub.mode") or [""])[0]
+            token = (qs.get("hub.verify_token") or [""])[0]
+            challenge = (qs.get("hub.challenge") or [""])[0]
+            if mode == "subscribe" and token and challenge:
+                if token == expected:
+                    _text(self, 200, challenge)
+                else:
+                    _text(self, 403, "Forbidden")
+                return
+            _text(self, 400, "Bad Request")
+            return
+
+        if path in ("/igdm", "/igdm/"):
+            _html(self, _basic_dashboard_html())
+            return
+
+        if path == "/api/igdm/items":
+            try:
+                limit = int((qs.get("limit") or ["100"])[0])
+            except Exception:
+                limit = 100
+            items = self.server.store.list_items(limit=limit)
+            _json(self, 200, {"ok": True, "count": len(items), "items": items})
+            return
+
+        if path == "/api/igdm/meta/ig/status":
+            token_obj = _read_json_file(Path(self.server.cfg["ig_token_path"])) or {}
+            connected = bool(token_obj.get("access_token"))
+            _json(
+                self,
+                200,
+                {
+                    "ok": True,
+                    "connected": connected,
+                    "ig_user_id": token_obj.get("ig_user_id"),
+                    "expires_in": token_obj.get("expires_in"),
+                    "subscribed_fields": token_obj.get("subscribed_fields"),
+                    "subscribed_at": token_obj.get("subscribed_at"),
+                    "subscribe_ok": token_obj.get("subscribe_ok"),
+                    "subscribe_error": token_obj.get("subscribe_error"),
+                },
+            )
+            return
+
+        if path.startswith("/api/igdm/item/"):
+            trace_id = path.split("/api/igdm/item/", 1)[1]
+            item = self.server.store.get_item(trace_id)
+            if not item:
+                _json(self, 404, {"ok": False, "error": "not_found"})
+                return
+            _json(self, 200, {"ok": True, **item})
+            return
+
+        _text(self, 404, "Not Found")
+
+    def do_POST(self) -> None:  # noqa: N802
+        parsed = urllib.parse.urlparse(self.path)
+        path = parsed.path or "/"
+
+        if path == "/api/igdm/meta/ig/subscribe":
+            token_obj = _read_json_file(Path(self.server.cfg["ig_token_path"])) or {}
+            token = str(token_obj.get("access_token") or "")
+            if not token:
+                _json(self, 400, {"ok": False, "error": "not_connected"})
+                return
+            subscribed_fields = self.server.cfg.get("ig_subscribed_fields") or "messages"
+            sub_payload, sub_err = _ig_subscribe_webhooks(
+                api_version=self.server.cfg.get("graph_api_version") or "v24.0",
+                access_token=token,
+                subscribed_fields=subscribed_fields,
+            )
+            token_obj.update(
+                {
+                    "subscribed_fields": subscribed_fields,
+                    "subscribed_at": int(time.time()) if sub_payload else None,
+                    "subscribe_ok": bool(sub_payload),
+                    "subscribe_error": sub_err if sub_err else None,
+                    "subscribe_response": sub_payload if sub_payload else None,
+                }
+            )
+            _safe_write_json(Path(self.server.cfg["ig_token_path"]), token_obj)
+            if sub_payload is None:
+                _json(self, 502, {"ok": False, "error": "subscribe_failed", "details": sub_err or ""})
+                return
+            _json(self, 200, {"ok": True, "result": sub_payload})
+            return
+
+        if path != "/meta/webhook":
+            _text(self, 404, "Not Found")
+            return
+
+        raw, payload = _read_json_body(self)
+        if payload is None:
+            _text(self, 400, "Bad Request")
+            return
+
+        signature = self.headers.get("X-Hub-Signature-256", "") or ""
+        if not meta_signature_is_valid(raw, signature, self.server.cfg["app_secret"]):
+            _text(self, 403, "Forbidden")
+            return
+
+        entries = payload.get("entry") or []
+        events_total = 0
+        inbound_total = 0
+        outbound_total = 0
+        drafts_total = 0
+        linked_total = 0
+
+        for entry in entries if isinstance(entries, list) else []:
+            messaging_events = entry.get("messaging") or []
+            if not isinstance(messaging_events, list):
+                continue
+            events_total += len(messaging_events)
+
+            for event in messaging_events:
+                if not isinstance(event, dict):
+                    continue
+                msg = event.get("message") or {}
+                if not isinstance(msg, dict):
+                    continue
+
+                sender_id = str((event.get("sender") or {}).get("id") or "").strip() or None
+                recipient_id = str((event.get("recipient") or {}).get("id") or "").strip() or None
+                ts_ms = int(event.get("timestamp") or 0) if str(event.get("timestamp") or "").isdigit() else None
+                is_echo = bool(msg.get("is_echo"))
+                mid = str(msg.get("mid") or "").strip() or None
+                text = msg.get("text")
+                if text is not None:
+                    text = str(text)
+
+                # Normalize "user id" as the non-page side of the thread.
+                user_id = (recipient_id if is_echo else sender_id) or ""
+                direction = "outbound" if is_echo else "inbound"
+
+                if not user_id:
+                    continue
+
+                event_id = self.server.store.insert_event(
+                    user_id=user_id,
+                    sender_id=sender_id,
+                    recipient_id=recipient_id,
+                    mid=mid,
+                    ts_ms=ts_ms,
+                    direction=direction,
+                    is_echo=is_echo,
+                    text=text,
+                    raw_event=event,
+                )
+
+                if is_echo:
+                    outbound_total += 1
+                    matched = self.server.store.link_outbound_to_latest_draft(user_id=user_id, outbound_event_id=event_id)
+                    if matched:
+                        linked_total += 1
+                    continue
+
+                inbound_total += 1
+                if not mid or not text:
+                    continue
+
+                try:
+                    _triage, _draft, _panel = process_inbound_message(
+                        store=self.server.store,
+                        reply_lib=self.server.reply_lib,
+                        user_id=user_id,
+                        mid=mid,
+                        ts_ms=ts_ms,
+                        text=text,
+                        inbound_event_id=event_id,
+                    )
+                    drafts_total += 1
+                except Exception as e:
+                    print(f"igdm: process error mid={mid} user={user_id}: {e}", file=sys.stderr, flush=True)
+
+        # Meta requires a fast 200 OK response.
+        _text(self, 200, "OK")
+
+        # Log summary after responding.
+        try:
+            print(
+                json.dumps(
+                    {
+                        "event": "igdm_webhook",
+                        "events": events_total,
+                        "inbound": inbound_total,
+                        "outbound": outbound_total,
+                        "drafts": drafts_total,
+                        "linked": linked_total,
+                    }
+                ),
+                flush=True,
+            )
+        except Exception:
+            pass
+
+    def log_message(self, fmt: str, *args: Any) -> None:  # noqa: A003
+        # Keep logs quiet; we print structured summaries ourselves.
+        return
+
+
+class IgdmShadowServer(ThreadingHTTPServer):
+    def __init__(self, server_address: tuple[str, int], handler: type[BaseHTTPRequestHandler], *, cfg: dict[str, str]):
+        super().__init__(server_address, handler)
+        self.cfg = cfg
+        self.store = IgdmStore(cfg["db_path"])
+        self.reply_lib = ReplyLibrary.load(cfg["reply_library"])
+        self._ready = threading.Event()
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(description="IGDM Shadow Mode server (Meta webhook + draft-only comparisons).")
+    ap.add_argument("--host", default=_env("IGDM_LISTEN_HOST", "127.0.0.1"))
+    ap.add_argument("--port", type=int, default=int(_env("IGDM_LISTEN_PORT", "5051")))
+    ap.add_argument("--db", default=_env("IGDM_DB_PATH", "/opt/if-emotion/data/igdm/igdm.sqlite"))
+    ap.add_argument(
+        "--reply-library",
+        default=_env("IGDM_REPLY_LIBRARY_PATH", "/opt/igdm-shadow/reply_library/top20_ready_answers.json"),
+    )
+    ap.add_argument("--verify-token", default=_env("META_VERIFY_TOKEN", ""))
+    ap.add_argument("--app-secret", default=_env("META_APP_SECRET", ""))
+    ap.add_argument("--meta-app-id", default=_env("META_APP_ID", ""))
+    ap.add_argument("--graph-api-version", default=_env("META_GRAPH_API_VERSION", "v24.0"))
+    ap.add_argument("--ig-redirect-uri", default=_env("IGDM_IG_REDIRECT_URI", ""))
+    ap.add_argument(
+        "--ig-scopes",
+        default=_env("IGDM_IG_SCOPES", "instagram_business_basic,instagram_business_manage_messages"),
+    )
+    ap.add_argument("--ig-subscribed-fields", default=_env("IGDM_IG_SUBSCRIBED_FIELDS", "messages"))
+    ap.add_argument("--ig-token-path", default=_env("IGDM_IG_TOKEN_PATH", "/opt/if-emotion/data/igdm/ig_token.json"))
+    ap.add_argument("--ig-state-path", default=_env("IGDM_IG_STATE_PATH", "/opt/if-emotion/data/igdm/ig_oauth_state.json"))
+    args = ap.parse_args(argv)
+
+    if not args.verify_token:
+        print("Missing META_VERIFY_TOKEN.", file=sys.stderr)
+        return 2
+    if not args.app_secret:
+        print("Missing META_APP_SECRET.", file=sys.stderr)
+        return 2
+    if not args.meta_app_id:
+        print("Missing META_APP_ID (needed for Instagram Business Login connect).", file=sys.stderr, flush=True)
+
+    reply_path = Path(args.reply_library)
+    if not reply_path.exists():
+        print(f"Missing reply library file: {reply_path}", file=sys.stderr)
+        return 2
+
+    Path(args.db).parent.mkdir(parents=True, exist_ok=True)
+
+    cfg = {
+        "db_path": str(args.db),
+        "reply_library": str(reply_path),
+        "verify_token": str(args.verify_token),
+        "app_secret": str(args.app_secret),
+        "meta_app_id": str(args.meta_app_id),
+        "graph_api_version": str(args.graph_api_version),
+        "ig_redirect_uri": str(args.ig_redirect_uri),
+        "ig_scopes": str(args.ig_scopes),
+        "ig_subscribed_fields": str(args.ig_subscribed_fields),
+        "ig_token_path": str(args.ig_token_path),
+        "ig_state_path": str(args.ig_state_path),
+    }
+
+    httpd = IgdmShadowServer((args.host, int(args.port)), IgdmShadowHandler, cfg=cfg)
+    print(json.dumps({"event": "igdm_start", "host": args.host, "port": args.port, "db": args.db}), flush=True)
+    httpd.serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
diff --git a/sergio_instagram_messaging/meta_setup_igdm.py b/sergio_instagram_messaging/meta_setup_igdm.py
new file mode 100644
index 0000000..f35ce03
--- /dev/null
+++ b/sergio_instagram_messaging/meta_setup_igdm.py
@@ -0,0 +1,220 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import urllib.parse
+import urllib.request
+from pathlib import Path
+from typing import Any
+
+from .export_meta_ig_history import DEFAULT_CREDS_PATH, GraphApiClient, GraphApiConfig, GraphApiError, load_dotenv
+from .meta_device_login import DEFAULT_STATE_PATH, cmd_poll, cmd_start
+
+
+RECOMMENDED_SCOPE = (
+    "pages_show_list,"
+    "pages_read_engagement,"
+    "business_management,"
+    "instagram_basic,"
+    "instagram_manage_messages,"
+    "pages_manage_metadata,"
+    "pages_messaging"
+)
+
+
+def _redact(obj: Any) -> Any:
+    if isinstance(obj, dict):
+        return {
+            k: ("" if k in {"access_token", "token"} else _redact(v))
+            for k, v in obj.items()
+        }
+    if isinstance(obj, list):
+        return [_redact(v) for v in obj]
+    return obj
+
+
+def _write_dotenv_kv(path: str, updates: dict[str, str]) -> None:
+    p = Path(path)
+    lines: list[str] = []
+    if p.exists():
+        lines = p.read_text(errors="replace").splitlines()
+
+    seen: set[str] = set()
+    out: list[str] = []
+    for raw in lines:
+        line = raw.rstrip("\n")
+        stripped = line.strip()
+        if not stripped or stripped.startswith("#") or "=" not in stripped:
+            out.append(line)
+            continue
+        key, _value = stripped.split("=", 1)
+        key = key.strip()
+        if key in updates:
+            out.append(f"{key}={updates[key]}")
+            seen.add(key)
+        else:
+            out.append(line)
+
+    for key, value in updates.items():
+        if key in seen:
+            continue
+        out.append(f"{key}={value}")
+
+    tmp = p.with_name(p.name + ".tmp")
+    tmp.write_text("\n".join(out).rstrip("\n") + "\n")
+    tmp.chmod(0o600)
+    tmp.replace(p)
+
+
+def _require(env: dict[str, str], key: str) -> str:
+    value = (env.get(key) or "").strip()
+    if not value:
+        raise KeyError(f"Missing required config: {key}")
+    return value
+
+
+def _optional(env: dict[str, str], key: str, default: str) -> str:
+    value = (env.get(key) or "").strip()
+    return value or default
+
+
+def _discover_page_token(*, user_token: str, version: str, page_id: str) -> str:
+    client = GraphApiClient(GraphApiConfig(access_token=user_token, version=version))
+    payload = client.get_json(
+        "/me/accounts",
+        {
+            "fields": "id,name,access_token",
+            "limit": "200",
+        },
+    )
+    data = payload.get("data") or []
+    if not isinstance(data, list) or not data:
+        raise SystemExit("No pages returned from /me/accounts for this user token.")
+
+    for page in data:
+        if not isinstance(page, dict):
+            continue
+        if str(page.get("id") or "").strip() != page_id:
+            continue
+        token = str(page.get("access_token") or "").strip()
+        if not token:
+            raise SystemExit("Selected page did not include an access_token in /me/accounts response.")
+        return token
+
+    raise SystemExit(f"Page id not found in /me/accounts: {page_id}")
+
+
+def _subscribe_page(*, page_id: str, page_token: str, version: str, fields: str) -> dict[str, Any]:
+    url = f"https://graph.facebook.com/{version}/{page_id}/subscribed_apps"
+    body = urllib.parse.urlencode(
+        {
+            "subscribed_fields": fields,
+            "access_token": page_token,
+        }
+    ).encode("utf-8")
+    req = urllib.request.Request(url, data=body, method="POST")
+    req.add_header("Content-Type", "application/x-www-form-urlencoded")
+    try:
+        with urllib.request.urlopen(req, timeout=60) as r:
+            return json.loads(r.read().decode("utf-8"))
+    except urllib.error.HTTPError as e:
+        raw = e.read().decode("utf-8", errors="replace")
+        try:
+            payload = json.loads(raw)
+        except Exception:
+            payload = {"raw": raw[:1000]}
+        raise GraphApiError({"status": e.code, "error": payload})
+
+
+def cmd_wizard(
+    *,
+    creds_path: str,
+    state_path: str,
+    scope: str,
+    page_id: str,
+    subscribed_fields: str,
+    max_wait_s: int,
+) -> int:
+    env = load_dotenv(creds_path)
+    version = _optional(env, "META_GRAPH_API_VERSION", "v24.0")
+
+    print("Starting Meta device login (no tokens printed).")
+    cmd_start(creds_path=creds_path, state_path=state_path, scope=scope)
+    print("\nWaiting for authorization...")
+    cmd_poll(
+        creds_path=creds_path,
+        state_path=state_path,
+        write_page_token=False,
+        target_ig_user_id=None,
+        max_wait_s=max_wait_s,
+    )
+
+    env = load_dotenv(creds_path)
+    user_token = (env.get("META_USER_ACCESS_TOKEN_LONG") or env.get("META_USER_ACCESS_TOKEN") or "").strip()
+    if not user_token:
+        raise SystemExit("Device login completed but META_USER_ACCESS_TOKEN(_LONG) was not written to creds file.")
+
+    page_token = _discover_page_token(user_token=user_token, version=version, page_id=page_id)
+    _write_dotenv_kv(creds_path, {"META_PAGE_ID": page_id, "META_PAGE_ACCESS_TOKEN": page_token})
+    print("Saved META_PAGE_ID + META_PAGE_ACCESS_TOKEN to creds file (no values printed).")
+
+    try:
+        resp = _subscribe_page(
+            page_id=page_id,
+            page_token=page_token,
+            version=version,
+            fields=subscribed_fields,
+        )
+        print(f"Subscribed Page to fields={subscribed_fields}: {json.dumps(_redact(resp), ensure_ascii=False)}")
+    except GraphApiError as e:
+        payload = e.payload if hasattr(e, "payload") else (e.args[0] if e.args else {"error": str(e)})
+        print(json.dumps(_redact(payload), ensure_ascii=False), file=sys.stderr)
+        return 3
+
+    print("Next: send a real DM to @socialmediatorr and refresh https://emo-social.infrafabric.io/igdm")
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(
+        description=(
+            "One-command wizard to enable IGDM shadow mode delivery.\n\n"
+            "What it does:\n"
+            "1) Starts Meta device login (prints URL + user code)\n"
+            "2) Polls until you authorize\n"
+            "3) Saves a fresh Page token into the creds file (no values printed)\n"
+            "4) Subscribes the Page to messages webhooks\n"
+        )
+    )
+    ap.add_argument("--creds", default=DEFAULT_CREDS_PATH)
+    ap.add_argument("--state", default=DEFAULT_STATE_PATH)
+    ap.add_argument("--scope", default=RECOMMENDED_SCOPE)
+    ap.add_argument("--page-id", default="918150581384278")
+    ap.add_argument("--subscribed-fields", default="messages")
+    ap.add_argument("--max-wait-s", type=int, default=900)
+    args = ap.parse_args(argv)
+
+    try:
+        return cmd_wizard(
+            creds_path=str(args.creds),
+            state_path=str(args.state),
+            scope=str(args.scope),
+            page_id=str(args.page_id),
+            subscribed_fields=str(args.subscribed_fields),
+            max_wait_s=int(args.max_wait_s),
+        )
+    except KeyError as e:
+        print(str(e), file=sys.stderr)
+        return 2
+    except SystemExit as e:
+        print(str(e), file=sys.stderr)
+        return 2
+    except Exception as e:
+        print(f"Error: {e}", file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
+
diff --git a/sergio_instagram_messaging/meta_subscribe_page.py b/sergio_instagram_messaging/meta_subscribe_page.py
index b388a56..35c6e00 100644
--- a/sergio_instagram_messaging/meta_subscribe_page.py
+++ b/sergio_instagram_messaging/meta_subscribe_page.py
@@ -62,7 +62,7 @@ def main(argv: list[str] | None = None) -> int:
     ap = argparse.ArgumentParser(
         description=(
             "Subscribe the Facebook Page to this Meta app's webhooks.\n\n"
-            "Note: this requires a Page access token that includes the pages_manage_metadata permission."
+            "Note: subscribing to the messages field typically requires a Page access token that includes the pages_messaging permission."
         )
     )
     ap.set_defaults(cmd=None)
@@ -103,4 +103,3 @@ def main(argv: list[str] | None = None) -> int:
 
 if __name__ == "__main__":
     raise SystemExit(main())
-