VibecoderMcSwaggins commited on
Commit
1d32642
Β·
1 Parent(s): 89f1173

docs: organize root files and add P1 synthesis fallback bug report

Browse files

MOVED to proper locations:
- SPEC_12_NARRATIVE_SYNTHESIS.md β†’ docs/specs/
- BRAINSTORM_EMBEDDINGS_META.md β†’ docs/brainstorming/

KEPT in root (per project convention):
- TOOL_ANALYSIS_CRITICAL.md
- CLAUDE.md, AGENTS.md, GEMINI.md, README.md

NEW bug report:
- P1_NARRATIVE_SYNTHESIS_FALLBACK.md - Documents why SPEC_12's LLM
synthesis silently falls back to template output. Root cause is
exception handling that catches all errors without user feedback.
LLM synthesis works locally with API keys; HF deployment needs secrets.

Updated ACTIVE_BUGS.md index with new P1 bug.

BRAINSTORM_EMBEDDINGS_META.md β†’ docs/brainstorming/BRAINSTORM_EMBEDDINGS_META.md RENAMED
File without changes
docs/bugs/ACTIVE_BUGS.md CHANGED
@@ -1,6 +1,6 @@
1
  # Active Bugs
2
 
3
- > Last updated: 2025-11-29
4
 
5
  ## P0 - Blocker
6
 
@@ -8,6 +8,22 @@
8
 
9
  ---
10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  ## P3 - Architecture/Enhancement
12
 
13
  ### ~~P3 - Missing Structured Cognitive Memory~~ FIXED (Phase 1)
 
1
  # Active Bugs
2
 
3
+ > Last updated: 2025-11-30
4
 
5
  ## P0 - Blocker
6
 
 
8
 
9
  ---
10
 
11
+ ## P1 - Important
12
+
13
+ ### P1 - Narrative Synthesis Falls Back to Template (NEW)
14
+ **File:** `P1_NARRATIVE_SYNTHESIS_FALLBACK.md`
15
+ **Related:** SPEC_12 (implemented but falling back)
16
+
17
+ **Problem:** Users see bullet-point template output instead of LLM-generated narrative prose.
18
+ **Root Cause:** Any exception in LLM synthesis triggers silent fallback to template.
19
+ **Impact:** Core value proposition (synthesized reports) not delivered.
20
+ **Fix Options:**
21
+ 1. Surface errors to user instead of silent fallback
22
+ 2. Configure HuggingFace Spaces secrets with API keys
23
+ 3. Add synthesis status indicator in UI
24
+
25
+ ---
26
+
27
  ## P3 - Architecture/Enhancement
28
 
29
  ### ~~P3 - Missing Structured Cognitive Memory~~ FIXED (Phase 1)
docs/bugs/P1_NARRATIVE_SYNTHESIS_FALLBACK.md ADDED
@@ -0,0 +1,185 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # P1: Narrative Synthesis Falls Back to Template (SPEC_12 Not Taking Effect)
2
+
3
+ **Status**: Open
4
+ **Priority**: P1 - Major UX degradation
5
+ **Affects**: Simple mode, all deployments
6
+ **Root Cause**: LLM synthesis silently failing β†’ template fallback
7
+ **Related**: SPEC_12 (implemented but not functioning)
8
+
9
+ ---
10
+
11
+ ## Problem Statement
12
+
13
+ SPEC_12 implemented LLM-based narrative synthesis, but users still see **template-formatted bullet points** instead of **prose paragraphs**:
14
+
15
+ ### What Users See (Template Fallback)
16
+
17
+ ```markdown
18
+ ## Sexual Health Analysis
19
+
20
+ ### Question
21
+ what medication for the best boners?
22
+
23
+ ### Drug Candidates
24
+ - **tadalafil**
25
+ - **sildenafil**
26
+
27
+ ### Key Findings
28
+ - Tadalafil improves erectile function
29
+
30
+ ### Assessment
31
+ - **Mechanism Score**: 4/10
32
+ - **Clinical Evidence Score**: 6/10
33
+ ```
34
+
35
+ ### What They Should See (LLM Synthesis)
36
+
37
+ ```markdown
38
+ ### Executive Summary
39
+
40
+ Sildenafil demonstrates clinically meaningful efficacy for erectile dysfunction,
41
+ with strong evidence from multiple RCTs demonstrating improved erectile function...
42
+
43
+ ### Background
44
+
45
+ Erectile dysfunction (ED) is a common male sexual health disorder...
46
+
47
+ ### Evidence Synthesis
48
+
49
+ **Mechanism of Action**
50
+ Sildenafil works by inhibiting phosphodiesterase type 5 (PDE5)...
51
+ ```
52
+
53
+ ---
54
+
55
+ ## Root Cause Analysis
56
+
57
+ ### Location: `src/orchestrators/simple.py:555-564`
58
+
59
+ ```python
60
+ try:
61
+ agent = Agent(model=get_model(), output_type=str, system_prompt=system_prompt)
62
+ result = await agent.run(user_prompt)
63
+ narrative = result.output
64
+ except Exception as e: # ← SILENT FALLBACK
65
+ logger.warning("LLM synthesis failed, using template fallback", error=str(e))
66
+ return self._generate_template_synthesis(query, evidence, assessment)
67
+ ```
68
+
69
+ **The Problem**: When ANY exception occurs during LLM synthesis, it silently falls back to template. Users see janky bullet points with no indication that the LLM call failed.
70
+
71
+ ### Why Synthesis Fails
72
+
73
+ | Cause | Symptom | Frequency |
74
+ |-------|---------|-----------|
75
+ | No API key in deployment | HuggingFace Spaces | HIGH |
76
+ | API rate limiting | Heavy usage | MEDIUM |
77
+ | Token overflow | Long evidence lists | MEDIUM |
78
+ | Model mismatch | Wrong model ID | LOW |
79
+ | Network timeout | Slow connections | LOW |
80
+
81
+ ---
82
+
83
+ ## Evidence: LLM Synthesis WORKS When Configured
84
+
85
+ Local test with API key:
86
+ ```python
87
+ # This works perfectly:
88
+ agent = Agent(model=get_model(), output_type=str, system_prompt=system_prompt)
89
+ result = await agent.run(user_prompt)
90
+ print(result.output) # β†’ Beautiful narrative prose!
91
+ ```
92
+
93
+ Output:
94
+ ```
95
+ ### Executive Summary
96
+
97
+ Sildenafil demonstrates clinically meaningful efficacy for erectile dysfunction,
98
+ with one study (Smith, 2020; N=100) reporting improved erectile function...
99
+ ```
100
+
101
+ ---
102
+
103
+ ## Impact
104
+
105
+ | Metric | Current | Expected |
106
+ |--------|---------|----------|
107
+ | Report quality | 3/10 (metadata dump) | 9/10 (professional prose) |
108
+ | User satisfaction | Low | High |
109
+ | Clinical utility | Limited | High |
110
+
111
+ The ENTIRE VALUE PROPOSITION of the research agent is the synthesized report. Template output defeats the purpose.
112
+
113
+ ---
114
+
115
+ ## Fix Options
116
+
117
+ ### Option A: Surface Error to User (RECOMMENDED)
118
+
119
+ When LLM synthesis fails, don't silently fall back. Show the user what went wrong:
120
+
121
+ ```python
122
+ except Exception as e:
123
+ logger.error("LLM synthesis failed", error=str(e), exc_info=True)
124
+
125
+ # Show error in report instead of silent fallback
126
+ error_note = f"""
127
+ ⚠️ **Note**: AI narrative synthesis unavailable.
128
+ Showing structured summary instead.
129
+
130
+ _Technical: {type(e).__name__}: {str(e)[:100]}_
131
+ """
132
+ template = self._generate_template_synthesis(query, evidence, assessment)
133
+ return f"{error_note}\n\n{template}"
134
+ ```
135
+
136
+ ### Option B: HuggingFace Secrets Configuration
137
+
138
+ For HuggingFace Spaces deployment, add secrets:
139
+ - `OPENAI_API_KEY` β†’ Required for synthesis
140
+ - `ANTHROPIC_API_KEY` β†’ Alternative provider
141
+
142
+ ### Option C: Graceful Degradation with Explanation
143
+
144
+ Add a banner explaining synthesis status:
145
+ - βœ… "AI-synthesized narrative report" (when LLM works)
146
+ - ⚠️ "Structured summary (AI synthesis unavailable)" (fallback)
147
+
148
+ ---
149
+
150
+ ## Diagnostic Steps
151
+
152
+ To determine why synthesis is failing in production:
153
+
154
+ 1. **Check logs for warning**: `"LLM synthesis failed, using template fallback"`
155
+ 2. **Check API key**: Is `OPENAI_API_KEY` set in environment?
156
+ 3. **Check model**: Is `gpt-5` accessible with current API tier?
157
+ 4. **Check rate limits**: Is the account quota exhausted?
158
+
159
+ ---
160
+
161
+ ## Acceptance Criteria
162
+
163
+ - [ ] Users see narrative prose reports (not bullet points) when API key is configured
164
+ - [ ] When synthesis fails, user sees clear indication (not silent fallback)
165
+ - [ ] HuggingFace Spaces deployment has proper secrets configured
166
+ - [ ] Logging captures the specific exception for debugging
167
+
168
+ ---
169
+
170
+ ## Files to Modify
171
+
172
+ | File | Change |
173
+ |------|--------|
174
+ | `src/orchestrators/simple.py:555-580` | Add error surfacing in fallback |
175
+ | `src/app.py` | Add synthesis status indicator to UI |
176
+ | HuggingFace Spaces Settings | Add `OPENAI_API_KEY` secret |
177
+
178
+ ---
179
+
180
+ ## Test Plan
181
+
182
+ 1. Run locally with API key β†’ Should get narrative prose
183
+ 2. Run locally WITHOUT API key β†’ Should get template WITH error message
184
+ 3. Deploy to HuggingFace with secrets β†’ Should get narrative prose
185
+ 4. Deploy to HuggingFace WITHOUT secrets β†’ Should get template WITH warning
SPEC_12_NARRATIVE_SYNTHESIS.md β†’ docs/specs/SPEC_12_NARRATIVE_SYNTHESIS.md RENAMED
File without changes