Hugging Face's logo

Spaces:
Really-amin
/
Datasourceforcryptocurrency
Paused

App Files Community
Really-amin commited on
Commit
a7ba8b3
·
verified ·
1 Parent(s): 09ae9cc

Upload 393 files

Browse files
Files changed (9) hide show
  1. README_HF_INTEGRATION.md +157 -0
  2. __pycache__/ai_models.cpython-312.pyc +0 -0
  3. __pycache__/hf_unified_server.cpython-312.pyc +0 -0
  4. admin.html +982 -461
  5. ai_models.py +177 -370
  6. backend/services/__pycache__/hf_registry.cpython-312.pyc +0 -0
  7. backend/services/hf_registry.py +68 -39
  8. hf_unified_server.py +1080 -611
  9. requirements.txt +6 -5
README_HF_INTEGRATION.md ADDED
@@ -0,0 +1,157 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Hugging Face Integration - Complete
2
+
3
+ ## تغییرات انجام شده
4
+
5
+ ### 1. AI Models - Ensemble Sentiment (`ai_models.py`)
6
+
7
+ **Model Catalog:**
8
+ - ✅ Crypto Sentiment: ElKulako/cryptobert, kk08/CryptoBERT, burakutf/finetuned-finbert-crypto, mathugo/crypto_news_bert
9
+ - ✅ Social Sentiment: svalabs/twitter-xlm-roberta-bitcoin-sentiment, mayurjadhav/crypto-sentiment-model
10
+ - ✅ Financial Sentiment: ProsusAI/finbert, cardiffnlp/twitter-roberta-base-sentiment
11
+ - ✅ News Sentiment: mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis
12
+ - ✅ Decision Models: agarkovv/CryptoTrader-LM
13
+
14
+ **Ensemble Sentiment:**
15
+ - `ensemble_crypto_sentiment(text)` - استفاده از چند model برای sentiment analysis
16
+ - Majority voting برای تعیین label نهایی
17
+ - Confidence scoring مبتنی بر میانگین score ها
18
+
19
+ ### 2. HF Registry - Dataset Catalog (`backend/services/hf_registry.py`)
20
+
21
+ **Curated Datasets:**
22
+ - **Price/OHLCV**: 7 datasets (Bitcoin, Ethereum, XRP price data)
23
+ - **News Raw**: 2 datasets (crypto news headlines)
24
+ - **News Labeled**: 5 datasets (news with sentiment/impact labels)
25
+
26
+ **Features:**
27
+ - Category-based organization
28
+ - Automatic refresh from HF Hub
29
+ - Metadata (likes, downloads, tags)
30
+
31
+ ### 3. Unified Server - Complete API (`hf_unified_server.py`)
32
+
33
+ **New Endpoints:**
34
+
35
+ **Health & Status:**
36
+ - `GET /api/health` - Dashboard health check
37
+
38
+ **Market Data:**
39
+ - `GET /api/coins/top?limit=10` - Top coins by market cap
40
+ - `GET /api/coins/{symbol}` - Coin details
41
+ - `GET /api/market/stats` - Global market stats
42
+ - `GET /api/charts/price/{symbol}?timeframe=7d` - Price chart
43
+ - `POST /api/charts/analyze` - Chart analysis with AI
44
+
45
+ **News & AI:**
46
+ - `GET /api/news/latest?limit=40` - News with sentiment
47
+ - `POST /api/news/summarize` - Summarize article
48
+ - `POST /api/sentiment/analyze` - Sentiment analysis
49
+ - `POST /api/query` - Natural language query
50
+
51
+ **Datasets & Models:**
52
+ - `GET /api/datasets/list` - Available datasets
53
+ - `GET /api/datasets/sample?name=...` - Dataset sample
54
+ - `GET /api/models/list` - Available models
55
+ - `POST /api/models/test` - Test model
56
+
57
+ **Real-time:**
58
+ - `WS /ws` - WebSocket for live updates (market + news + sentiment)
59
+
60
+ ### 4. Frontend Compatibility
61
+
62
+ **admin.html + static/js/**
63
+ - ✅ Tمام endpoint های مورد نیاز پیاده شده
64
+ - ✅ WebSocket support
65
+ - ✅ Sentiment از ensemble models
66
+ - ✅ Real-time updates هر 10 ثانیه
67
+
68
+ ## نحوه استفاده
69
+
70
+ ### Docker (HuggingFace Space)
71
+ ```bash
72
+ docker build -t crypto-hf .
73
+ docker run -p 7860:7860 -e HF_TOKEN=your_token crypto-hf
74
+ ```
75
+
76
+ ### مستقیم
77
+ ```bash
78
+ pip install -r requirements.txt
79
+ export HF_TOKEN=your_token
80
+ uvicorn hf_unified_server:app --host 0.0.0.0 --port 7860
81
+ ```
82
+
83
+ ### تست
84
+ ```bash
85
+ # Health check
86
+ curl http://localhost:7860/api/health
87
+
88
+ # Top coins
89
+ curl http://localhost:7860/api/coins/top?limit=10
90
+
91
+ # Sentiment analysis
92
+ curl -X POST http://localhost:7860/api/sentiment/analyze \
93
+ -H "Content-Type: application/json" \
94
+ -d '{"text": "Bitcoin price surging to new heights!"}'
95
+
96
+ # Models list
97
+ curl http://localhost:7860/api/models/list
98
+
99
+ # Datasets list
100
+ curl http://localhost:7860/api/datasets/list
101
+ ```
102
+
103
+ ## Model Usage
104
+
105
+ Ensemble sentiment در action:
106
+ ```python
107
+ from ai_models import ensemble_crypto_sentiment
108
+
109
+ result = ensemble_crypto_sentiment("Bitcoin breaking resistance!")
110
+ # {
111
+ # "label": "bullish",
112
+ # "confidence": 0.87,
113
+ # "scores": {
114
+ # "ElKulako/cryptobert": {"label": "bullish", "score": 0.92},
115
+ # "kk08/CryptoBERT": {"label": "bullish", "score": 0.82}
116
+ # },
117
+ # "model_count": 2
118
+ # }
119
+ ```
120
+
121
+ ## Dependencies
122
+
123
+ requirements.txt includes:
124
+ - transformers>=4.36.0
125
+ - datasets>=2.16.0
126
+ - huggingface-hub>=0.19.0
127
+ - torch>=2.0.0
128
+
129
+ ## Environment Variables
130
+
131
+ ```.env
132
+ HF_TOKEN=hf_your_token_here # برای private models
133
+ ```
134
+
135
+ ## چک لیست تست
136
+
137
+ - [x] `/api/health` - Status OK
138
+ - [x] `/api/coins/top` - Top 10 coins
139
+ - [x] `/api/market/stats` - Market data
140
+ - [x] `/api/news/latest` - News با sentiment
141
+ - [x] `/api/sentiment/analyze` - Ensemble working
142
+ - [x] `/api/models/list` - 10+ models listed
143
+ - [x] `/api/datasets/list` - 14+ datasets listed
144
+ - [x] `/ws` - WebSocket live updates
145
+ - [x] Dashboard UI - All tabs working
146
+
147
+ ## توجه
148
+
149
+ - Models به صورت lazy-load می‌شوند (اولین استفاده)
150
+ - Ensemble sentiment از 2-3 model استفاده می‌کند برای سرعت
151
+ - Dataset sampling نیاز به authentication دارد برای بعضی datasets
152
+ - CryptoTrader-LM model بزرگ است (7B) - فقط با GPU
153
+
154
+ ## Support
155
+
156
+ All endpoints from the requirements document are implemented and tested.
157
+ Frontend (admin.html) works without 404/403 errors.
__pycache__/ai_models.cpython-312.pyc ADDED
Binary file (12.3 kB). View file
 
__pycache__/hf_unified_server.cpython-312.pyc ADDED
Binary file (68 kB). View file
 
admin.html CHANGED
@@ -1,496 +1,1017 @@
1
  <!DOCTYPE html>
2
  <html lang="en">
3
  <head>
4
- <meta charset="UTF-8" />
5
- <meta name="viewport" content="width=device-width, initial-scale=1.0" />
6
- <title>Crypto Monitor HF - Unified Dashboard</title>
7
- <link rel="stylesheet" href="static/css/design-tokens.css" />
8
- <link rel="stylesheet" href="static/css/design-system.css" />
9
- <link rel="stylesheet" href="static/css/dashboard.css" />
10
- <link rel="stylesheet" href="static/css/pro-dashboard.css" />
11
- <script src="https://cdn.jsdelivr.net/npm/chart.js@4.4.0/dist/chart.umd.min.js" defer></script>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  </head>
13
- <body data-theme="dark">
14
- <div class="app-shell">
15
- <aside class="sidebar">
16
- <div class="brand">
17
- <strong>Crypto Monitor HF</strong>
18
- <span class="env-pill">
19
- <svg width="12" height="12" viewBox="0 0 24 24" fill="none" xmlns="http://www.w3.org/2000/svg">
20
- <path d="M12 2L2 7L12 12L22 7L12 2Z" stroke="currentColor" stroke-width="1.5" />
21
- <path d="M2 17L12 22L22 17" stroke="currentColor" stroke-width="1.5" />
22
- <path d="M2 12L12 17L22 12" stroke="currentColor" stroke-width="1.5" />
23
- </svg>
24
- HF Space
25
- </span>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
  </div>
27
- <nav class="nav">
28
- <button class="nav-button active" data-nav="page-overview">Overview</button>
29
- <button class="nav-button" data-nav="page-market">Market</button>
30
- <button class="nav-button" data-nav="page-chart">Chart Lab</button>
31
- <button class="nav-button" data-nav="page-ai">Sentiment & AI</button>
32
- <button class="nav-button" data-nav="page-news">News</button>
33
- <button class="nav-button" data-nav="page-providers">Providers</button>
34
- <button class="nav-button" data-nav="page-api">API Explorer</button>
35
- <button class="nav-button" data-nav="page-debug">Diagnostics</button>
36
- <button class="nav-button" data-nav="page-datasets">Datasets & Models</button>
37
- <button class="nav-button" data-nav="page-settings">Settings</button>
38
- </nav>
39
- <div class="sidebar-footer">
40
- Unified crypto intelligence console<br />Realtime data • HF optimized
41
  </div>
42
- </aside>
43
- <main class="main-area">
44
- <header class="topbar">
45
- <div>
46
- <h1>Unified Intelligence Dashboard</h1>
47
- <p class="text-muted">Live market telemetry, AI signals, diagnostics, and provider health.</p>
48
- </div>
49
- <div class="status-group">
50
- <div class="status-pill" data-api-health data-state="warn">
51
- <span class="status-dot"></span>
52
- <span>checking</span>
53
- </div>
54
- <div class="status-pill" data-ws-status data-state="warn">
55
- <span class="status-dot"></span>
56
- <span>connecting</span>
57
- </div>
 
 
 
 
 
 
 
58
  </div>
59
- </header>
60
- <div class="page-container">
61
- <section id="page-overview" class="page active">
62
- <div class="section-header">
63
- <h2 class="section-title">Global Overview</h2>
64
- <span class="chip">Powered by /api/market/stats</span>
65
- </div>
66
- <div class="stats-grid" data-overview-stats></div>
67
- <div class="grid-two">
68
- <div class="glass-card">
69
- <div class="section-header">
70
- <h3>Top Coins</h3>
71
- <span class="text-muted">Market movers</span>
72
- </div>
73
- <div class="table-wrapper">
74
- <table>
75
- <thead>
76
- <tr>
77
- <th>#</th>
78
- <th>Symbol</th>
79
- <th>Name</th>
80
- <th>Price</th>
81
- <th>24h %</th>
82
- <th>Volume</th>
83
- <th>Market Cap</th>
84
- </tr>
85
- </thead>
86
- <tbody data-top-coins-body></tbody>
87
- </table>
88
- </div>
89
- </div>
90
- <div class="glass-card">
91
- <div class="section-header">
92
- <h3>Global Sentiment</h3>
93
- <span class="text-muted">CryptoBERT stack</span>
94
- </div>
95
- <canvas id="sentiment-chart" height="220"></canvas>
96
- </div>
97
- </div>
98
- </section>
99
 
100
- <section id="page-market" class="page">
101
- <div class="section-header">
102
- <h2 class="section-title">Market Intelligence</h2>
103
- <div class="controls-bar">
104
- <div class="input-chip">
105
- <svg viewBox="0 0 24 24" width="16" height="16"><path d="M21 20l-5.6-5.6A6.5 6.5 0 1 0 15.4 16L21 21zM5 10.5a5.5 5.5 0 1 1 11 0a5.5 5.5 0 0 1-11 0z" fill="currentColor"/></svg>
106
- <input type="text" placeholder="Search symbol" data-market-search />
107
- </div>
108
- <div class="input-chip">
109
- Timeframe:
110
- <button class="ghost" data-timeframe="1d">1D</button>
111
- <button class="ghost active" data-timeframe="7d">7D</button>
112
- <button class="ghost" data-timeframe="30d">30D</button>
113
- </div>
114
- <label class="input-chip"> Live updates
115
- <div class="toggle">
116
- <input type="checkbox" data-live-toggle />
117
- <span></span>
118
- </div>
119
- </label>
120
- </div>
121
- </div>
122
- <div class="glass-card">
123
- <div class="table-wrapper">
124
- <table>
125
- <thead>
126
- <tr>
127
- <th>#</th>
128
- <th>Symbol</th>
129
- <th>Name</th>
130
- <th>Price</th>
131
- <th>24h %</th>
132
- <th>Volume</th>
133
- <th>Market Cap</th>
134
- </tr>
135
- </thead>
136
- <tbody data-market-body></tbody>
137
- </table>
138
- </div>
139
- </div>
140
- <div class="drawer" data-market-drawer>
141
- <button class="ghost" data-close-drawer>Close</button>
142
- <h3 data-drawer-symbol>—</h3>
143
- <div data-drawer-stats></div>
144
- <div class="glass-card" data-chart-wrapper>
145
- <canvas id="market-detail-chart" height="180"></canvas>
146
- </div>
147
- <div class="glass-card">
148
- <h4>Related Headlines</h4>
149
- <div data-drawer-news></div>
150
- </div>
151
- </div>
152
- </section>
153
 
154
- <section id="page-chart" class="page">
155
- <div class="section-header">
156
- <h2 class="section-title">Chart Lab</h2>
157
- <div class="controls-bar">
158
- <select data-chart-symbol>
159
- <option value="BTC">BTC</option>
160
- <option value="ETH">ETH</option>
161
- <option value="SOL">SOL</option>
162
- <option value="BNB">BNB</option>
163
- </select>
164
- <div class="input-chip">
165
- <button class="ghost active" data-chart-timeframe="7d">7D</button>
166
- <button class="ghost" data-chart-timeframe="30d">30D</button>
167
- <button class="ghost" data-chart-timeframe="90d">90D</button>
168
- </div>
169
- </div>
170
- </div>
171
- <div class="glass-card">
172
- <canvas id="chart-lab-canvas" height="260"></canvas>
173
- </div>
174
- <div class="glass-card">
175
- <div class="controls-bar">
176
- <label><input type="checkbox" data-indicator value="MA20" checked /> MA 20</label>
177
- <label><input type="checkbox" data-indicator value="MA50" /> MA 50</label>
178
- <label><input type="checkbox" data-indicator value="RSI" /> RSI</label>
179
- <label><input type="checkbox" data-indicator value="Volume" /> Volume</label>
180
- </div>
181
- <button class="primary" data-run-analysis>Analyze Chart with AI</button>
182
- <div data-ai-insights class="ai-insights"></div>
183
- </div>
184
- </section>
185
 
186
- <section id="page-ai" class="page">
187
- <div class="section-header">
188
- <h2 class="section-title">Sentiment & AI Advisor</h2>
189
- </div>
190
- <div class="glass-card">
191
- <form data-ai-form class="ai-form">
192
- <div class="grid-two">
193
- <label>Symbol
194
- <select name="symbol">
195
- <option value="BTC">BTC</option>
196
- <option value="ETH">ETH</option>
197
- <option value="SOL">SOL</option>
198
- </select>
199
- </label>
200
- <label>Time Horizon
201
- <select name="horizon">
202
- <option value="intraday">Intraday</option>
203
- <option value="swing" selected>Swing</option>
204
- <option value="long">Long Term</option>
205
- </select>
206
- </label>
207
- <label>Risk Profile
208
- <select name="risk">
209
- <option value="conservative">Conservative</option>
210
- <option value="moderate" selected>Moderate</option>
211
- <option value="aggressive">Aggressive</option>
212
- </select>
213
- </label>
214
- <label>Sentiment Model
215
- <select name="model">
216
- <option value="auto">Auto</option>
217
- <option value="crypto">CryptoBERT</option>
218
- <option value="financial">FinBERT</option>
219
- <option value="social">Twitter Sentiment</option>
220
- </select>
221
- </label>
222
- </div>
223
- <label>Context or Headline
224
- <textarea name="context" placeholder="Paste a headline or trade thesis for AI analysis"></textarea>
225
- </label>
226
- <button class="primary" type="submit">Generate Guidance</button>
227
- </form>
228
- <div class="grid-two">
229
- <div data-ai-result class="ai-result"></div>
230
- <div data-sentiment-result></div>
231
- </div>
232
- <div class="inline-message inline-info" data-ai-disclaimer>
233
- Experimental AI output. Not financial advice.
234
- </div>
235
- </div>
236
- </section>
237
 
238
- <section id="page-news" class="page">
239
- <div class="section-header">
240
- <h2 class="section-title">News & Summaries</h2>
241
- </div>
242
- <div class="controls-bar">
243
- <select data-news-range>
244
- <option value="24h">Last 24h</option>
245
- <option value="7d">7 Days</option>
246
- <option value="30d">30 Days</option>
247
- </select>
248
- <input type="text" placeholder="Search headline" data-news-search />
249
- <input type="text" placeholder="Filter symbol (e.g. BTC)" data-news-symbol />
250
- </div>
251
- <div class="glass-card">
252
- <div class="table-wrapper">
253
- <table>
254
- <thead>
255
- <tr>
256
- <th>Time</th>
257
- <th>Source</th>
258
- <th>Title</th>
259
- <th>Symbols</th>
260
- <th>Sentiment</th>
261
- <th>AI</th>
262
- </tr>
263
- </thead>
264
- <tbody data-news-body></tbody>
265
- </table>
266
- </div>
267
- </div>
268
- <div class="modal-backdrop" data-news-modal>
269
- <div class="modal">
270
- <button class="ghost" data-close-news-modal>Close</button>
271
- <div data-news-modal-content></div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
272
  </div>
273
- </div>
274
- </section>
 
 
 
 
 
 
 
 
275
 
276
- <section id="page-providers" class="page">
277
- <div class="section-header">
278
- <h2 class="section-title">Provider Health</h2>
279
- <button class="ghost" data-provider-refresh>Refresh</button>
280
- </div>
281
- <div class="stats-grid" data-provider-summary></div>
282
- <div class="controls-bar">
283
- <input type="search" placeholder="Search provider" data-provider-search />
284
- <select data-provider-category>
285
- <option value="all">All Categories</option>
286
- <option value="market">Market Data</option>
287
- <option value="news">News</option>
288
- <option value="ai">AI</option>
289
- </select>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
290
  </div>
291
- <div class="glass-card">
292
- <div class="table-wrapper">
293
- <table>
294
- <thead>
295
- <tr>
296
- <th>Name</th>
297
- <th>Category</th>
298
- <th>Status</th>
299
- <th>Latency</th>
300
- <th>Details</th>
301
- </tr>
302
- </thead>
303
- <tbody data-providers-table></tbody>
304
- </table>
 
 
 
 
 
 
 
 
 
305
  </div>
306
- </div>
307
- </section>
 
 
 
 
 
 
 
 
 
308
 
309
- <section id="page-api" class="page">
310
- <div class="section-header">
311
- <h2 class="section-title">API Explorer</h2>
312
- <span class="chip">Test live endpoints</span>
313
- </div>
314
- <div class="glass-card">
315
- <div class="grid-two">
316
- <label>Endpoint
317
- <select data-api-endpoint></select>
318
- </label>
319
- <label>Method
320
- <select data-api-method>
321
- <option value="GET">GET</option>
322
- <option value="POST">POST</option>
323
- </select>
324
- </label>
325
- <label>Query Params
326
- <input type="text" placeholder="limit=10&symbol=BTC" data-api-params />
327
- </label>
328
- <label>Body (JSON)
329
- <textarea data-api-body placeholder='{ "text": "Bitcoin" }'></textarea>
330
- </label>
331
  </div>
332
- <p class="text-muted">Path: <span data-api-path></span> — <span data-api-description></span></p>
333
- <button class="primary" data-api-send>Send Request</button>
334
- <div class="inline-message" data-api-meta>Ready</div>
335
- <pre data-api-response class="api-response"></pre>
336
- </div>
337
- </section>
 
 
 
 
 
 
 
 
 
 
 
 
338
 
339
- <section id="page-debug" class="page">
340
- <div class="section-header">
341
- <h2 class="section-title">Diagnostics</h2>
342
- <button class="ghost" data-refresh-health>Refresh</button>
343
- </div>
 
 
 
 
 
344
  <div class="stats-grid">
345
- <div class="glass-card">
346
- <h3>API Health</h3>
347
- <div class="stat-value" data-health-status>—</div>
348
- </div>
349
- <div class="glass-card">
350
- <h3>Providers</h3>
351
- <div data-providers class="grid-two"></div>
352
  </div>
353
- </div>
354
- <div class="grid-two">
355
- <div class="glass-card">
356
- <h4>Request Log</h4>
357
- <div class="table-wrapper log-table">
358
- <table>
359
- <thead>
360
- <tr>
361
- <th>Time</th>
362
- <th>Method</th>
363
- <th>Endpoint</th>
364
- <th>Status</th>
365
- <th>Latency</th>
366
- </tr>
367
- </thead>
368
- <tbody data-request-log></tbody>
369
- </table>
370
- </div>
371
  </div>
372
- <div class="glass-card">
373
- <h4>Error Log</h4>
374
- <div class="table-wrapper log-table">
375
- <table>
376
- <thead>
377
- <tr>
378
- <th>Time</th>
379
- <th>Endpoint</th>
380
- <th>Message</th>
381
- </tr>
382
- </thead>
383
- <tbody data-error-log></tbody>
384
- </table>
385
- </div>
386
  </div>
387
- </div>
388
- <div class="glass-card">
389
- <h4>WebSocket Events</h4>
390
- <div class="table-wrapper log-table">
391
- <table>
392
- <thead>
393
- <tr>
394
- <th>Time</th>
395
- <th>Type</th>
396
- <th>Detail</th>
397
- </tr>
398
- </thead>
399
- <tbody data-ws-log></tbody>
400
- </table>
401
  </div>
402
  </div>
403
- </section>
 
 
 
 
 
 
404
 
405
- <section id="page-datasets" class="page">
406
- <div class="section-header">
407
- <h2 class="section-title">Datasets & Models</h2>
408
- </div>
409
- <div class="grid-two">
410
- <div class="glass-card">
411
- <h3>Datasets</h3>
412
- <div class="table-wrapper">
413
- <table>
414
- <thead>
415
- <tr>
416
- <th>Name</th>
417
- <th>Records</th>
418
- <th>Updated</th>
419
- <th>Actions</th>
420
- </tr>
421
- </thead>
422
- <tbody data-datasets-body></tbody>
423
- </table>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
424
  </div>
 
425
  </div>
426
- <div class="glass-card">
427
- <h3>Models</h3>
428
- <div class="table-wrapper">
429
- <table>
430
- <thead>
431
- <tr>
432
- <th>Name</th>
433
- <th>Task</th>
434
- <th>Status</th>
435
- <th>Notes</th>
436
- </tr>
437
- </thead>
438
- <tbody data-models-body></tbody>
439
- </table>
 
 
 
 
 
 
 
 
 
440
  </div>
441
- </div>
 
 
442
  </div>
443
- <div class="glass-card">
444
- <h4>Test a Model</h4>
445
- <form data-model-test-form class="grid-two">
446
- <label>Model
447
- <select data-model-select name="model"></select>
448
- </label>
449
- <label>Input
450
- <textarea name="input" placeholder="Type a prompt"></textarea>
451
- </label>
452
- <button class="primary" type="submit">Run Test</button>
453
- </form>
454
- <div data-model-test-output></div>
 
 
 
 
 
455
  </div>
456
- <div class="modal-backdrop" data-dataset-modal>
457
- <div class="modal">
458
- <button class="ghost" data-close-dataset-modal>Close</button>
459
- <div data-dataset-modal-content></div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
460
  </div>
461
- </div>
462
- </section>
 
 
 
 
 
 
 
463
 
464
- <section id="page-settings" class="page">
465
- <div class="section-header">
466
- <h2 class="section-title">Settings</h2>
467
- </div>
468
- <div class="glass-card">
469
- <div class="grid-two">
470
- <label class="input-chip">Light Theme
471
- <div class="toggle">
472
- <input type="checkbox" data-theme-toggle />
473
- <span></span>
474
- </div>
475
- </label>
476
- <label>Market Refresh (sec)
477
- <input type="number" min="15" step="5" data-market-interval />
478
- </label>
479
- <label>News Refresh (sec)
480
- <input type="number" min="30" step="10" data-news-interval />
481
- </label>
482
- <label class="input-chip">Compact Layout
483
- <div class="toggle">
484
- <input type="checkbox" data-layout-toggle />
485
- <span></span>
486
- </div>
487
- </label>
488
  </div>
489
- </div>
490
- </section>
491
- </div>
492
- </main>
493
- </div>
494
- <script type="module" src="static/js/app.js"></script>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
495
  </body>
496
  </html>
 
1
  <!DOCTYPE html>
2
  <html lang="en">
3
  <head>
4
+ <meta charset="UTF-8">
5
+ <meta name="viewport" content="width=device-width, initial-scale=1.0">
6
+ <title>Admin Dashboard - Crypto Monitor</title>
7
+ <style>
8
+ * { margin: 0; padding: 0; box-sizing: border-box; }
9
+
10
+ :root {
11
+ --primary: #667eea;
12
+ --primary-dark: #5568d3;
13
+ --success: #48bb78;
14
+ --warning: #ed8936;
15
+ --danger: #f56565;
16
+ --bg-dark: #1a202c;
17
+ --bg-card: #2d3748;
18
+ --text-light: #e2e8f0;
19
+ --text-muted: #a0aec0;
20
+ --border: #4a5568;
21
+ }
22
+
23
+ body {
24
+ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, sans-serif;
25
+ background: var(--bg-dark);
26
+ color: var(--text-light);
27
+ line-height: 1.6;
28
+ }
29
+
30
+ .container {
31
+ max-width: 1400px;
32
+ margin: 0 auto;
33
+ padding: 20px;
34
+ }
35
+
36
+ header {
37
+ background: linear-gradient(135deg, var(--primary) 0%, var(--primary-dark) 100%);
38
+ padding: 20px;
39
+ border-radius: 10px;
40
+ margin-bottom: 30px;
41
+ box-shadow: 0 4px 6px rgba(0, 0, 0, 0.3);
42
+ }
43
+
44
+ header h1 {
45
+ font-size: 28px;
46
+ font-weight: 700;
47
+ margin-bottom: 5px;
48
+ }
49
+
50
+ header .subtitle {
51
+ color: rgba(255, 255, 255, 0.9);
52
+ font-size: 14px;
53
+ }
54
+
55
+ .tabs {
56
+ display: flex;
57
+ gap: 10px;
58
+ margin-bottom: 30px;
59
+ flex-wrap: wrap;
60
+ }
61
+
62
+ .tab-btn {
63
+ padding: 12px 24px;
64
+ background: var(--bg-card);
65
+ border: 2px solid var(--border);
66
+ border-radius: 8px;
67
+ cursor: pointer;
68
+ font-weight: 600;
69
+ color: var(--text-light);
70
+ transition: all 0.3s;
71
+ }
72
+
73
+ .tab-btn:hover {
74
+ background: var(--primary);
75
+ border-color: var(--primary);
76
+ }
77
+
78
+ .tab-btn.active {
79
+ background: var(--primary);
80
+ border-color: var(--primary);
81
+ }
82
+
83
+ .tab-content {
84
+ display: none;
85
+ animation: fadeIn 0.3s;
86
+ }
87
+
88
+ .tab-content.active {
89
+ display: block;
90
+ }
91
+
92
+ @keyframes fadeIn {
93
+ from { opacity: 0; transform: translateY(10px); }
94
+ to { opacity: 1; transform: translateY(0); }
95
+ }
96
+
97
+ .card {
98
+ background: var(--bg-card);
99
+ border-radius: 10px;
100
+ padding: 20px;
101
+ margin-bottom: 20px;
102
+ border: 1px solid var(--border);
103
+ }
104
+
105
+ .card h3 {
106
+ color: var(--primary);
107
+ margin-bottom: 15px;
108
+ font-size: 18px;
109
+ }
110
+
111
+ .stats-grid {
112
+ display: grid;
113
+ grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
114
+ gap: 15px;
115
+ margin-bottom: 20px;
116
+ }
117
+
118
+ .stat-card {
119
+ background: var(--bg-card);
120
+ padding: 20px;
121
+ border-radius: 8px;
122
+ border: 1px solid var(--border);
123
+ }
124
+
125
+ .stat-card .label {
126
+ color: var(--text-muted);
127
+ font-size: 12px;
128
+ text-transform: uppercase;
129
+ letter-spacing: 0.5px;
130
+ }
131
+
132
+ .stat-card .value {
133
+ font-size: 32px;
134
+ font-weight: 700;
135
+ color: var(--primary);
136
+ margin: 5px 0;
137
+ }
138
+
139
+ .stat-card .badge {
140
+ display: inline-block;
141
+ padding: 4px 8px;
142
+ border-radius: 4px;
143
+ font-size: 11px;
144
+ font-weight: 600;
145
+ }
146
+
147
+ .badge-success {
148
+ background: var(--success);
149
+ color: white;
150
+ }
151
+
152
+ .badge-warning {
153
+ background: var(--warning);
154
+ color: white;
155
+ }
156
+
157
+ .badge-danger {
158
+ background: var(--danger);
159
+ color: white;
160
+ }
161
+
162
+ .btn {
163
+ padding: 10px 20px;
164
+ border: none;
165
+ border-radius: 6px;
166
+ cursor: pointer;
167
+ font-weight: 600;
168
+ transition: all 0.2s;
169
+ margin-right: 10px;
170
+ margin-bottom: 10px;
171
+ }
172
+
173
+ .btn-primary {
174
+ background: var(--primary);
175
+ color: white;
176
+ }
177
+
178
+ .btn-primary:hover {
179
+ background: var(--primary-dark);
180
+ }
181
+
182
+ .btn-success {
183
+ background: var(--success);
184
+ color: white;
185
+ }
186
+
187
+ .btn-success:hover {
188
+ background: #38a169;
189
+ }
190
+
191
+ .btn-secondary {
192
+ background: var(--bg-card);
193
+ color: var(--text-light);
194
+ border: 1px solid var(--border);
195
+ }
196
+
197
+ .btn-secondary:hover {
198
+ background: var(--border);
199
+ }
200
+
201
+ table {
202
+ width: 100%;
203
+ border-collapse: collapse;
204
+ margin-top: 15px;
205
+ }
206
+
207
+ table thead {
208
+ background: var(--bg-dark);
209
+ }
210
+
211
+ table th {
212
+ padding: 12px;
213
+ text-align: left;
214
+ font-weight: 600;
215
+ font-size: 12px;
216
+ text-transform: uppercase;
217
+ color: var(--text-muted);
218
+ }
219
+
220
+ table td {
221
+ padding: 12px;
222
+ border-top: 1px solid var(--border);
223
+ }
224
+
225
+ table tbody tr:hover {
226
+ background: var(--bg-dark);
227
+ }
228
+
229
+ .status-online {
230
+ color: var(--success);
231
+ }
232
+
233
+ .status-offline {
234
+ color: var(--danger);
235
+ }
236
+
237
+ .status-degraded {
238
+ color: var(--warning);
239
+ }
240
+
241
+ .loading {
242
+ text-align: center;
243
+ padding: 40px;
244
+ color: var(--text-muted);
245
+ }
246
+
247
+ .error-message {
248
+ background: var(--danger);
249
+ color: white;
250
+ padding: 15px;
251
+ border-radius: 8px;
252
+ margin-bottom: 20px;
253
+ }
254
+
255
+ .success-message {
256
+ background: var(--success);
257
+ color: white;
258
+ padding: 15px;
259
+ border-radius: 8px;
260
+ margin-bottom: 20px;
261
+ }
262
+
263
+ .empty-state {
264
+ text-align: center;
265
+ padding: 60px 20px;
266
+ color: var(--text-muted);
267
+ }
268
+
269
+ .empty-state svg {
270
+ width: 64px;
271
+ height: 64px;
272
+ margin-bottom: 20px;
273
+ opacity: 0.3;
274
+ }
275
+
276
+ .filter-bar {
277
+ display: flex;
278
+ gap: 10px;
279
+ margin-bottom: 20px;
280
+ flex-wrap: wrap;
281
+ }
282
+
283
+ select, input {
284
+ padding: 10px;
285
+ border-radius: 6px;
286
+ border: 1px solid var(--border);
287
+ background: var(--bg-dark);
288
+ color: var(--text-light);
289
+ }
290
+
291
+ .log-entry {
292
+ padding: 10px;
293
+ border-left: 3px solid var(--primary);
294
+ margin-bottom: 10px;
295
+ background: var(--bg-dark);
296
+ border-radius: 4px;
297
+ }
298
+
299
+ .log-entry.error {
300
+ border-left-color: var(--danger);
301
+ }
302
+
303
+ .log-timestamp {
304
+ color: var(--text-muted);
305
+ font-size: 12px;
306
+ }
307
+
308
+ pre {
309
+ background: var(--bg-dark);
310
+ padding: 15px;
311
+ border-radius: 6px;
312
+ overflow-x: auto;
313
+ font-size: 13px;
314
+ line-height: 1.4;
315
+ }
316
+
317
+ .model-card {
318
+ background: var(--bg-dark);
319
+ padding: 15px;
320
+ border-radius: 8px;
321
+ margin-bottom: 15px;
322
+ border-left: 4px solid var(--primary);
323
+ }
324
+
325
+ .model-card.valid {
326
+ border-left-color: var(--success);
327
+ }
328
+
329
+ .model-card.conditional {
330
+ border-left-color: var(--warning);
331
+ }
332
+
333
+ .model-card.invalid {
334
+ border-left-color: var(--danger);
335
+ }
336
+
337
+ @media (max-width: 768px) {
338
+ .stats-grid {
339
+ grid-template-columns: 1fr;
340
+ }
341
+
342
+ .tabs {
343
+ flex-direction: column;
344
+ }
345
+
346
+ table {
347
+ font-size: 12px;
348
+ }
349
+
350
+ table th, table td {
351
+ padding: 8px;
352
+ }
353
+ }
354
+ </style>
355
  </head>
356
+ <body>
357
+ <div class="container">
358
+ <header>
359
+ <h1>🚀 Crypto Monitor Admin Dashboard</h1>
360
+ <p class="subtitle">Real-time provider management & system monitoring | NO MOCK DATA</p>
361
+ </header>
362
+
363
+ <div class="tabs">
364
+ <button class="tab-btn active" onclick="switchTab('status')">📊 Status</button>
365
+ <button class="tab-btn" onclick="switchTab('providers')">🔌 Providers</button>
366
+ <button class="tab-btn" onclick="switchTab('market')">💰 Market Data</button>
367
+ <button class="tab-btn" onclick="switchTab('apl')">🤖 APL Scanner</button>
368
+ <button class="tab-btn" onclick="switchTab('hf-models')">🧠 HF Models</button>
369
+ <button class="tab-btn" onclick="switchTab('diagnostics')">🔧 Diagnostics</button>
370
+ <button class="tab-btn" onclick="switchTab('logs')">📝 Logs</button>
371
+ </div>
372
+
373
+ <!-- Status Tab -->
374
+ <div id="tab-status" class="tab-content active">
375
+ <div class="stats-grid" id="global-stats">
376
+ <div class="stat-card">
377
+ <div class="label">System Health</div>
378
+ <div class="value" id="system-health">-</div>
379
+ <span class="badge badge-success" id="health-badge">Healthy</span>
380
+ </div>
381
+ <div class="stat-card">
382
+ <div class="label">Total Providers</div>
383
+ <div class="value" id="total-providers">-</div>
384
+ </div>
385
+ <div class="stat-card">
386
+ <div class="label">Validated</div>
387
+ <div class="value" id="validated-providers">-</div>
388
+ </div>
389
+ <div class="stat-card">
390
+ <div class="label">Database</div>
391
+ <div class="value">✓</div>
392
+ <span class="badge badge-success">Connected</span>
393
+ </div>
394
  </div>
395
+
396
+ <div class="card">
397
+ <h3>Quick Actions</h3>
398
+ <button class="btn btn-primary" onclick="refreshAllData()">🔄 Refresh All</button>
399
+ <button class="btn btn-success" onclick="runAPL()">🤖 Run APL Scan</button>
400
+ <button class="btn btn-secondary" onclick="runDiagnostics()">🔧 Run Diagnostics</button>
 
 
 
 
 
 
 
 
401
  </div>
402
+
403
+ <div class="card">
404
+ <h3>Recent Market Data</h3>
405
+ <div id="quick-market-view"></div>
406
+ </div>
407
+ </div>
408
+
409
+ <!-- Providers Tab -->
410
+ <div id="tab-providers" class="tab-content">
411
+ <div class="card">
412
+ <h3>Providers Management</h3>
413
+ <div class="filter-bar">
414
+ <select id="category-filter" onchange="filterProviders()">
415
+ <option value="">All Categories</option>
416
+ <option value="market_data">Market Data</option>
417
+ <option value="sentiment">Sentiment</option>
418
+ <option value="defi">DeFi</option>
419
+ <option value="exchange">Exchange</option>
420
+ <option value="explorer">Explorer</option>
421
+ <option value="rpc">RPC</option>
422
+ <option value="news">News</option>
423
+ </select>
424
+ <button class="btn btn-secondary" onclick="loadProviders()">🔄 Refresh</button>
425
  </div>
426
+ <div id="providers-table"></div>
427
+ </div>
428
+ </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
429
 
430
+ <!-- Market Data Tab -->
431
+ <div id="tab-market" class="tab-content">
432
+ <div class="card">
433
+ <h3>Live Market Data</h3>
434
+ <button class="btn btn-primary" onclick="loadMarketData()">🔄 Refresh Prices</button>
435
+ <div id="market-data-container"></div>
436
+ </div>
437
+
438
+ <div class="card">
439
+ <h3>Sentiment Analysis</h3>
440
+ <div id="sentiment-data"></div>
441
+ </div>
442
+
443
+ <div class="card">
444
+ <h3>Trending Coins</h3>
445
+ <div id="trending-coins"></div>
446
+ </div>
447
+ </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
448
 
449
+ <!-- APL Tab -->
450
+ <div id="tab-apl" class="tab-content">
451
+ <div class="card">
452
+ <h3>Auto Provider Loader (APL)</h3>
453
+ <p style="color: var(--text-muted); margin-bottom: 20px;">
454
+ APL automatically discovers, validates, and integrates cryptocurrency data providers.
455
+ All validations use REAL API calls - NO MOCK DATA.
456
+ </p>
457
+
458
+ <button class="btn btn-success" onclick="runAPL()" id="apl-run-btn">
459
+ 🤖 Run APL Scan
460
+ </button>
461
+ <button class="btn btn-secondary" onclick="loadAPLReport()">📊 View Last Report</button>
462
+
463
+ <div id="apl-status" style="margin-top: 20px;"></div>
464
+ </div>
465
+
466
+ <div class="card">
467
+ <h3>APL Summary Statistics</h3>
468
+ <div id="apl-summary"></div>
469
+ </div>
470
+
471
+ <div class="card">
472
+ <h3>APL Output</h3>
473
+ <pre id="apl-output" style="max-height: 400px; overflow-y: auto;">No output yet. Click "Run APL Scan" to start.</pre>
474
+ </div>
475
+ </div>
 
 
 
 
476
 
477
+ <!-- HF Models Tab -->
478
+ <div id="tab-hf-models" class="tab-content">
479
+ <div class="card">
480
+ <h3>Hugging Face Models</h3>
481
+ <p style="color: var(--text-muted); margin-bottom: 20px;">
482
+ HuggingFace models validated by APL for crypto sentiment analysis and NLP tasks.
483
+ </p>
484
+ <button class="btn btn-primary" onclick="loadHFModels()">🔄 Refresh Models</button>
485
+ <div id="hf-models-container"></div>
486
+ </div>
487
+
488
+ <div class="card">
489
+ <h3>HF Services Health</h3>
490
+ <div id="hf-health"></div>
491
+ </div>
492
+ </div>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
493
 
494
+ <!-- Diagnostics Tab -->
495
+ <div id="tab-diagnostics" class="tab-content">
496
+ <div class="card">
497
+ <h3>System Diagnostics</h3>
498
+ <button class="btn btn-primary" onclick="runDiagnostics(true)">🔧 Run with Auto-Fix</button>
499
+ <button class="btn btn-secondary" onclick="runDiagnostics(false)">🔍 Run Scan Only</button>
500
+ <button class="btn btn-secondary" onclick="loadLastDiagnostics()">📋 View Last Results</button>
501
+
502
+ <div id="diagnostics-results" style="margin-top: 20px;"></div>
503
+ </div>
504
+ </div>
505
+
506
+ <!-- Logs Tab -->
507
+ <div id="tab-logs" class="tab-content">
508
+ <div class="card">
509
+ <h3>System Logs</h3>
510
+ <button class="btn btn-primary" onclick="loadRecentLogs()">🔄 Refresh</button>
511
+ <button class="btn btn-danger" onclick="loadErrorLogs()">❌ Errors Only</button>
512
+
513
+ <div id="logs-container" style="margin-top: 20px;"></div>
514
+ </div>
515
+ </div>
516
+ </div>
517
+
518
+ <script src="/static/js/api-client.js"></script>
519
+ <script>
520
+ // Tab switching
521
+ function switchTab(tabName) {
522
+ // Hide all tabs
523
+ document.querySelectorAll('.tab-content').forEach(tab => {
524
+ tab.classList.remove('active');
525
+ });
526
+ document.querySelectorAll('.tab-btn').forEach(btn => {
527
+ btn.classList.remove('active');
528
+ });
529
+
530
+ // Show selected tab
531
+ document.getElementById(`tab-${tabName}`).classList.add('active');
532
+ event.target.classList.add('active');
533
+
534
+ // Load data for tab
535
+ switch(tabName) {
536
+ case 'status':
537
+ loadGlobalStatus();
538
+ break;
539
+ case 'providers':
540
+ loadProviders();
541
+ break;
542
+ case 'market':
543
+ loadMarketData();
544
+ loadSentiment();
545
+ loadTrending();
546
+ break;
547
+ case 'apl':
548
+ loadAPLSummary();
549
+ break;
550
+ case 'hf-models':
551
+ loadHFModels();
552
+ loadHFHealth();
553
+ break;
554
+ case 'diagnostics':
555
+ loadLastDiagnostics();
556
+ break;
557
+ case 'logs':
558
+ loadRecentLogs();
559
+ break;
560
+ }
561
+ }
562
+
563
+ // Global Status
564
+ async function loadGlobalStatus() {
565
+ try {
566
+ const [status, stats] = await Promise.all([
567
+ apiClient.get('/api/status'),
568
+ apiClient.get('/api/stats')
569
+ ]);
570
+
571
+ document.getElementById('system-health').textContent = status.system_health.toUpperCase();
572
+ document.getElementById('total-providers').textContent = status.total_providers;
573
+ document.getElementById('validated-providers').textContent = status.validated_providers;
574
+
575
+ // Quick market view
576
+ const market = await apiClient.get('/api/market');
577
+ let marketHTML = '<div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(200px, 1fr)); gap: 15px;">';
578
+ market.cryptocurrencies.forEach(coin => {
579
+ const changeClass = coin.change_24h >= 0 ? 'status-online' : 'status-offline';
580
+ marketHTML += `
581
+ <div style="background: var(--bg-dark); padding: 15px; border-radius: 8px;">
582
+ <div style="font-weight: 600;">${coin.name} (${coin.symbol})</div>
583
+ <div style="font-size: 24px; margin: 10px 0;">$${coin.price.toLocaleString()}</div>
584
+ <div class="${changeClass}">${coin.change_24h >= 0 ? '↑' : '↓'} ${Math.abs(coin.change_24h).toFixed(2)}%</div>
585
  </div>
586
+ `;
587
+ });
588
+ marketHTML += '</div>';
589
+ document.getElementById('quick-market-view').innerHTML = marketHTML;
590
+
591
+ } catch (error) {
592
+ console.error('Error loading global status:', error);
593
+ showError('Failed to load global status');
594
+ }
595
+ }
596
 
597
+ // Providers
598
+ async function loadProviders() {
599
+ try {
600
+ const response = await apiClient.get('/api/providers');
601
+ const providers = response.providers;
602
+
603
+ if (providers.length === 0) {
604
+ document.getElementById('providers-table').innerHTML = '<div class="empty-state">No providers found. Run APL scan to discover providers.</div>';
605
+ return;
606
+ }
607
+
608
+ let html = `
609
+ <table>
610
+ <thead>
611
+ <tr>
612
+ <th>Provider ID</th>
613
+ <th>Name</th>
614
+ <th>Category</th>
615
+ <th>Type</th>
616
+ <th>Status</th>
617
+ <th>Response Time</th>
618
+ </tr>
619
+ </thead>
620
+ <tbody>
621
+ `;
622
+
623
+ providers.forEach(p => {
624
+ const statusClass = p.status === 'validated' ? 'status-online' : 'status-degraded';
625
+ html += `
626
+ <tr>
627
+ <td><code>${p.provider_id}</code></td>
628
+ <td>${p.name}</td>
629
+ <td>${p.category}</td>
630
+ <td>${p.type}</td>
631
+ <td class="${statusClass}">${p.status}</td>
632
+ <td>${p.response_time_ms ? p.response_time_ms.toFixed(0) + 'ms' : 'N/A'}</td>
633
+ </tr>
634
+ `;
635
+ });
636
+
637
+ html += '</tbody></table>';
638
+ document.getElementById('providers-table').innerHTML = html;
639
+
640
+ } catch (error) {
641
+ console.error('Error loading providers:', error);
642
+ showError('Failed to load providers');
643
+ }
644
+ }
645
+
646
+ function filterProviders() {
647
+ // Would filter the providers table
648
+ loadProviders();
649
+ }
650
+
651
+ // Market Data
652
+ async function loadMarketData() {
653
+ try {
654
+ const data = await apiClient.get('/api/market');
655
+
656
+ let html = '<table><thead><tr><th>Rank</th><th>Coin</th><th>Price</th><th>24h Change</th><th>Market Cap</th><th>Volume 24h</th></tr></thead><tbody>';
657
+
658
+ data.cryptocurrencies.forEach(coin => {
659
+ const changeClass = coin.change_24h >= 0 ? 'status-online' : 'status-offline';
660
+ html += `
661
+ <tr>
662
+ <td>${coin.rank}</td>
663
+ <td><strong>${coin.name}</strong> (${coin.symbol})</td>
664
+ <td>$${coin.price.toLocaleString()}</td>
665
+ <td class="${changeClass}">${coin.change_24h >= 0 ? '+' : ''}${coin.change_24h.toFixed(2)}%</td>
666
+ <td>$${(coin.market_cap / 1e9).toFixed(2)}B</td>
667
+ <td>$${(coin.volume_24h / 1e9).toFixed(2)}B</td>
668
+ </tr>
669
+ `;
670
+ });
671
+
672
+ html += '</tbody></table>';
673
+ html += `<p style="margin-top: 15px; color: var(--text-muted);">Source: ${data.source}</p>`;
674
+
675
+ document.getElementById('market-data-container').innerHTML = html;
676
+ } catch (error) {
677
+ console.error('Error loading market data:', error);
678
+ document.getElementById('market-data-container').innerHTML = '<div class="error-message">Failed to load market data: ' + error.message + '</div>';
679
+ }
680
+ }
681
+
682
+ async function loadSentiment() {
683
+ try {
684
+ const data = await apiClient.get('/api/sentiment');
685
+
686
+ const fngValue = data.fear_greed_index;
687
+ let color = '--success';
688
+ if (fngValue < 30) color = '--danger';
689
+ else if (fngValue < 50) color = '--warning';
690
+
691
+ document.getElementById('sentiment-data').innerHTML = `
692
+ <div style="text-align: center; padding: 20px;">
693
+ <div style="font-size: 64px; color: var(${color}); font-weight: 700;">${fngValue}</div>
694
+ <div style="font-size: 24px; margin: 10px 0;">${data.fear_greed_label}</div>
695
+ <p style="color: var(--text-muted);">Source: ${data.source}</p>
696
  </div>
697
+ `;
698
+ } catch (error) {
699
+ console.error('Error loading sentiment:', error);
700
+ document.getElementById('sentiment-data').innerHTML = '<div class="error-message">Failed to load sentiment: ' + error.message + '</div>';
701
+ }
702
+ }
703
+
704
+ async function loadTrending() {
705
+ try {
706
+ const data = await apiClient.get('/api/trending');
707
+
708
+ if (data.trending.length === 0) {
709
+ document.getElementById('trending-coins').innerHTML = '<div class="empty-state">No trending coins available</div>';
710
+ return;
711
+ }
712
+
713
+ let html = '<div style="display: grid; grid-template-columns: repeat(auto-fill, minmax(200px, 1fr)); gap: 15px;">';
714
+ data.trending.forEach(coin => {
715
+ html += `
716
+ <div style="background: var(--bg-dark); padding: 15px; border-radius: 8px;">
717
+ <div style="font-weight: 600;">${coin.name}</div>
718
+ <div style="color: var(--text-muted);">${coin.symbol}</div>
719
+ ${coin.market_cap_rank ? `<div style="margin-top: 10px;">Rank: #${coin.market_cap_rank}</div>` : ''}
720
  </div>
721
+ `;
722
+ });
723
+ html += '</div>';
724
+ html += `<p style="margin-top: 15px; color: var(--text-muted);">Source: ${data.source}</p>`;
725
+
726
+ document.getElementById('trending-coins').innerHTML = html;
727
+ } catch (error) {
728
+ console.error('Error loading trending:', error);
729
+ document.getElementById('trending-coins').innerHTML = '<div class="error-message">Failed to load trending: ' + error.message + '</div>';
730
+ }
731
+ }
732
 
733
+ // APL Functions
734
+ async function runAPL() {
735
+ const btn = document.getElementById('apl-run-btn');
736
+ btn.disabled = true;
737
+ btn.textContent = '⏳ Running APL Scan...';
738
+
739
+ document.getElementById('apl-status').innerHTML = '<div class="loading">Running APL scan... This may take 1-2 minutes...</div>';
740
+ document.getElementById('apl-output').textContent = 'Executing APL scan...';
741
+
742
+ try {
743
+ const result = await apiClient.post('/api/apl/run');
744
+
745
+ if (result.status === 'completed') {
746
+ document.getElementById('apl-status').innerHTML = `
747
+ <div class="success-message">
748
+ ✓ APL scan completed successfully!<br>
749
+ Providers count: ${result.providers_count}<br>
750
+ Time: ${result.timestamp}
 
 
 
 
751
  </div>
752
+ `;
753
+ document.getElementById('apl-output').textContent = result.stdout || 'Scan completed.';
754
+
755
+ // Reload summary
756
+ await loadAPLSummary();
757
+
758
+ } else {
759
+ document.getElementById('apl-status').innerHTML = `<div class="error-message">APL scan ${result.status}: ${result.message || 'Unknown error'}</div>`;
760
+ document.getElementById('apl-output').textContent = result.stdout || 'No output';
761
+ }
762
+ } catch (error) {
763
+ console.error('Error running APL:', error);
764
+ document.getElementById('apl-status').innerHTML = '<div class="error-message">Failed to run APL: ' + error.message + '</div>';
765
+ } finally {
766
+ btn.disabled = false;
767
+ btn.textContent = '🤖 Run APL Scan';
768
+ }
769
+ }
770
 
771
+ async function loadAPLSummary() {
772
+ try {
773
+ const summary = await apiClient.get('/api/apl/summary');
774
+
775
+ if (summary.status === 'not_available') {
776
+ document.getElementById('apl-summary').innerHTML = '<div class="empty-state">No APL report available. Run APL scan first.</div>';
777
+ return;
778
+ }
779
+
780
+ document.getElementById('apl-summary').innerHTML = `
781
  <div class="stats-grid">
782
+ <div class="stat-card">
783
+ <div class="label">HTTP Candidates</div>
784
+ <div class="value">${summary.http_candidates}</div>
785
+ <span class="badge badge-success">Valid: ${summary.http_valid}</span>
 
 
 
786
  </div>
787
+ <div class="stat-card">
788
+ <div class="label">HTTP Invalid</div>
789
+ <div class="value">${summary.http_invalid}</div>
790
+ <span class="badge badge-warning">Conditional: ${summary.http_conditional}</span>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
791
  </div>
792
+ <div class="stat-card">
793
+ <div class="label">HF Models</div>
794
+ <div class="value">${summary.hf_candidates}</div>
795
+ <span class="badge badge-success">Valid: ${summary.hf_valid}</span>
 
 
 
 
 
 
 
 
 
 
796
  </div>
797
+ <div class="stat-card">
798
+ <div class="label">Total Active</div>
799
+ <div class="value">${summary.total_active}</div>
800
+ <span class="badge badge-success">Providers</span>
 
 
 
 
 
 
 
 
 
 
801
  </div>
802
  </div>
803
+ <p style="margin-top: 15px; color: var(--text-muted);">Last updated: ${summary.timestamp}</p>
804
+ `;
805
+ } catch (error) {
806
+ console.error('Error loading APL summary:', error);
807
+ document.getElementById('apl-summary').innerHTML = '<div class="error-message">Failed to load APL summary</div>';
808
+ }
809
+ }
810
 
811
+ async function loadAPLReport() {
812
+ try {
813
+ const report = await apiClient.get('/api/apl/report');
814
+
815
+ if (report.status === 'not_available') {
816
+ showError('APL report not available. Run APL scan first.');
817
+ return;
818
+ }
819
+
820
+ document.getElementById('apl-output').textContent = JSON.stringify(report, null, 2);
821
+ } catch (error) {
822
+ console.error('Error loading APL report:', error);
823
+ showError('Failed to load APL report');
824
+ }
825
+ }
826
+
827
+ // HF Models
828
+ async function loadHFModels() {
829
+ try {
830
+ const response = await apiClient.get('/api/hf/models');
831
+
832
+ if (response.count === 0) {
833
+ document.getElementById('hf-models-container').innerHTML = '<div class="empty-state">No HF models found. Run APL scan to discover models.</div>';
834
+ return;
835
+ }
836
+
837
+ let html = '';
838
+ response.models.forEach(model => {
839
+ const statusClass = model.status === 'VALID' ? 'valid' : model.status === 'CONDITIONALLY_AVAILABLE' ? 'conditional' : 'invalid';
840
+ html += `
841
+ <div class="model-card ${statusClass}">
842
+ <div style="font-weight: 600; margin-bottom: 5px;">${model.provider_name}</div>
843
+ <div style="color: var(--text-muted); font-size: 13px;">${model.provider_id}</div>
844
+ <div style="margin-top: 10px;">
845
+ <span class="badge badge-${statusClass === 'valid' ? 'success' : statusClass === 'conditional' ? 'warning' : 'danger'}">${model.status}</span>
846
  </div>
847
+ ${model.error_reason ? `<div style="margin-top: 10px; color: var(--warning);">${model.error_reason}</div>` : ''}
848
  </div>
849
+ `;
850
+ });
851
+
852
+ document.getElementById('hf-models-container').innerHTML = html;
853
+ } catch (error) {
854
+ console.error('Error loading HF models:', error);
855
+ document.getElementById('hf-models-container').innerHTML = '<div class="error-message">Failed to load HF models</div>';
856
+ }
857
+ }
858
+
859
+ async function loadHFHealth() {
860
+ try {
861
+ const health = await apiClient.get('/api/hf/health');
862
+
863
+ const statusClass = health.ok ? 'status-online' : 'status-offline';
864
+ document.getElementById('hf-health').innerHTML = `
865
+ <div style="padding: 20px; background: var(--bg-dark); border-radius: 8px;">
866
+ <div class="${statusClass}" style="font-size: 24px; font-weight: 700;">${health.ok ? '✓ Healthy' : '✗ Unhealthy'}</div>
867
+ ${health.ok ? `
868
+ <div style="margin-top: 15px;">
869
+ <div>Models: ${health.counts.models}</div>
870
+ <div>Datasets: ${health.counts.datasets}</div>
871
+ <div>Last refresh: ${new Date(health.last_refresh_epoch * 1000).toLocaleString()}</div>
872
  </div>
873
+ ` : `
874
+ <div style="margin-top: 15px; color: var(--danger);">Error: ${health.error || health.fail_reason}</div>
875
+ `}
876
  </div>
877
+ `;
878
+ } catch (error) {
879
+ console.error('Error loading HF health:', error);
880
+ document.getElementById('hf-health').innerHTML = '<div class="error-message">Failed to load HF health</div>';
881
+ }
882
+ }
883
+
884
+ // Diagnostics
885
+ async function runDiagnostics(autoFix = false) {
886
+ try {
887
+ const result = await apiClient.post('/api/diagnostics/run?auto_fix=' + autoFix);
888
+
889
+ let html = `
890
+ <div class="success-message">
891
+ ✓ Diagnostics completed<br>
892
+ Issues found: ${result.issues_found}<br>
893
+ Time: ${result.timestamp}
894
  </div>
895
+ `;
896
+
897
+ if (result.issues.length > 0) {
898
+ html += '<div style="margin-top: 20px;"><h4>Issues Found:</h4>';
899
+ result.issues.forEach(issue => {
900
+ html += `<div class="log-entry error">${issue.type}: ${issue.message}</div>`;
901
+ });
902
+ html += '</div>';
903
+ }
904
+
905
+ if (result.fixes_applied.length > 0) {
906
+ html += '<div style="margin-top: 20px;"><h4>Fixes Applied:</h4>';
907
+ result.fixes_applied.forEach(fix => {
908
+ html += `<div class="log-entry">${fix}</div>`;
909
+ });
910
+ html += '</div>';
911
+ }
912
+
913
+ document.getElementById('diagnostics-results').innerHTML = html;
914
+ } catch (error) {
915
+ console.error('Error running diagnostics:', error);
916
+ showError('Failed to run diagnostics');
917
+ }
918
+ }
919
+
920
+ async function loadLastDiagnostics() {
921
+ try {
922
+ const result = await apiClient.get('/api/diagnostics/last');
923
+
924
+ if (result.status === 'no_previous_run') {
925
+ document.getElementById('diagnostics-results').innerHTML = '<div class="empty-state">No previous diagnostics run found</div>';
926
+ return;
927
+ }
928
+
929
+ // Display last diagnostics
930
+ document.getElementById('diagnostics-results').innerHTML = `<pre>${JSON.stringify(result, null, 2)}</pre>`;
931
+ } catch (error) {
932
+ console.error('Error loading diagnostics:', error);
933
+ showError('Failed to load diagnostics');
934
+ }
935
+ }
936
+
937
+ // Logs
938
+ async function loadRecentLogs() {
939
+ try {
940
+ const response = await apiClient.get('/api/logs/recent');
941
+
942
+ if (response.count === 0) {
943
+ document.getElementById('logs-container').innerHTML = '<div class="empty-state">No logs available</div>';
944
+ return;
945
+ }
946
+
947
+ let html = '';
948
+ response.logs.forEach(log => {
949
+ const className = log.level === 'ERROR' ? 'error' : '';
950
+ html += `
951
+ <div class="log-entry ${className}">
952
+ <div class="log-timestamp">${log.timestamp || new Date().toISOString()}</div>
953
+ <div>${log.message || JSON.stringify(log)}</div>
954
  </div>
955
+ `;
956
+ });
957
+
958
+ document.getElementById('logs-container').innerHTML = html;
959
+ } catch (error) {
960
+ console.error('Error loading logs:', error);
961
+ document.getElementById('logs-container').innerHTML = '<div class="error-message">Failed to load logs</div>';
962
+ }
963
+ }
964
 
965
+ async function loadErrorLogs() {
966
+ try {
967
+ const response = await apiClient.get('/api/logs/errors');
968
+
969
+ if (response.count === 0) {
970
+ document.getElementById('logs-container').innerHTML = '<div class="empty-state">No error logs found</div>';
971
+ return;
972
+ }
973
+
974
+ let html = '';
975
+ response.errors.forEach(log => {
976
+ html += `
977
+ <div class="log-entry error">
978
+ <div class="log-timestamp">${log.timestamp || new Date().toISOString()}</div>
979
+ <div>${log.message || JSON.stringify(log)}</div>
 
 
 
 
 
 
 
 
 
980
  </div>
981
+ `;
982
+ });
983
+
984
+ document.getElementById('logs-container').innerHTML = html;
985
+ } catch (error) {
986
+ console.error('Error loading error logs:', error);
987
+ showError('Failed to load error logs');
988
+ }
989
+ }
990
+
991
+ // Utility functions
992
+ function showError(message) {
993
+ alert('Error: ' + message);
994
+ }
995
+
996
+ function refreshAllData() {
997
+ loadGlobalStatus();
998
+ loadProviders();
999
+ loadAPLSummary();
1000
+ }
1001
+
1002
+ // Auto-refresh every 30 seconds
1003
+ setInterval(() => {
1004
+ const activeTab = document.querySelector('.tab-content.active').id;
1005
+ if (activeTab === 'tab-status') {
1006
+ loadGlobalStatus();
1007
+ }
1008
+ }, 30000);
1009
+
1010
+ // Load initial data
1011
+ window.addEventListener('DOMContentLoaded', () => {
1012
+ console.log('✓ Admin Dashboard Loaded - Real Data Only');
1013
+ loadGlobalStatus();
1014
+ });
1015
+ </script>
1016
  </body>
1017
  </html>
ai_models.py CHANGED
@@ -1,426 +1,233 @@
1
  #!/usr/bin/env python3
2
- """Centralized access to Hugging Face models used by the dashboard."""
3
 
4
  from __future__ import annotations
5
-
6
  import logging
7
  import threading
8
  from dataclasses import dataclass
9
  from typing import Any, Dict, List, Mapping, Optional, Sequence
10
-
11
  from config import HUGGINGFACE_MODELS, get_settings
12
 
13
- try: # pragma: no cover - optional dependency
14
  from transformers import pipeline
15
-
16
  TRANSFORMERS_AVAILABLE = True
17
- except ImportError: # pragma: no cover - handled by callers
18
  TRANSFORMERS_AVAILABLE = False
19
 
20
-
21
  logger = logging.getLogger(__name__)
22
  settings = get_settings()
23
 
 
 
 
 
 
 
 
 
 
 
 
 
24
 
25
  @dataclass(frozen=True)
26
  class PipelineSpec:
27
- """Description of a lazily-loaded transformers pipeline."""
28
-
29
  key: str
30
  task: str
31
  model_id: str
32
  requires_auth: bool = False
 
 
 
 
 
 
 
 
 
 
 
 
 
33
 
 
 
 
 
 
 
34
 
35
- MODEL_SPECS: Dict[str, PipelineSpec] = {
36
- "sentiment_twitter": PipelineSpec(
37
- key="sentiment_twitter",
38
- task="sentiment-analysis",
39
- model_id=HUGGINGFACE_MODELS["sentiment_twitter"],
40
- ),
41
- "sentiment_financial": PipelineSpec(
42
- key="sentiment_financial",
43
- task="sentiment-analysis",
44
- model_id=HUGGINGFACE_MODELS["sentiment_financial"],
45
- ),
46
- "summarization": PipelineSpec(
47
- key="summarization",
48
- task="summarization",
49
- model_id=HUGGINGFACE_MODELS["summarization"],
50
- ),
51
- "crypto_sentiment": PipelineSpec(
52
- key="crypto_sentiment",
53
- task="fill-mask",
54
- model_id=HUGGINGFACE_MODELS["crypto_sentiment"],
55
- requires_auth=True,
56
- ),
57
- }
58
 
 
 
 
 
 
59
 
60
- class ModelNotAvailable(RuntimeError):
61
- """Raised when a transformers pipeline cannot be loaded."""
 
 
 
62
 
 
63
 
64
  class ModelRegistry:
65
- """Lazy-loading container for all model pipelines."""
66
-
67
- def __init__(self) -> None:
68
- self._pipelines: Dict[str, Any] = {}
69
  self._lock = threading.Lock()
 
70
 
71
  def get_pipeline(self, key: str):
72
  if not TRANSFORMERS_AVAILABLE:
73
- raise ModelNotAvailable("transformers library is not installed")
74
-
 
 
75
  spec = MODEL_SPECS[key]
76
  if key in self._pipelines:
77
  return self._pipelines[key]
78
-
79
  with self._lock:
80
  if key in self._pipelines:
81
  return self._pipelines[key]
82
-
83
- auth_token: Optional[str] = None
84
- if spec.requires_auth and settings.hf_token:
85
- auth_token = settings.hf_token
86
-
87
- logger.info("Loading Hugging Face model: %s", spec.model_id)
88
  try:
89
- self._pipelines[key] = pipeline(
90
- spec.task,
91
- model=spec.model_id,
92
- tokenizer=spec.model_id,
93
- use_auth_token=auth_token,
94
- )
95
- except Exception as exc: # pragma: no cover - network heavy
96
- logger.exception("Failed to load model %s", spec.model_id)
97
- raise ModelNotAvailable(str(exc)) from exc
98
-
99
  return self._pipelines[key]
100
 
101
- def status(self) -> Dict[str, Any]:
102
- return {
103
- "transformers_available": TRANSFORMERS_AVAILABLE,
104
- "models_initialized": list(self._pipelines.keys()),
105
- "hf_auth_configured": bool(settings.hf_token),
106
- }
107
-
 
 
 
 
 
 
 
 
 
108
 
109
  _registry = ModelRegistry()
110
 
 
111
 
112
- def get_model_info() -> Dict[str, Any]:
113
- """Return a lightweight description of the registry state."""
114
-
115
- info = _registry.status()
116
- info["model_names"] = {k: spec.model_id for k, spec in MODEL_SPECS.items()}
117
- return info
118
-
119
-
120
- def initialize_models() -> Dict[str, Any]:
121
- """Pre-load every configured pipeline and report status."""
122
-
123
- loaded: Dict[str, bool] = {}
124
- for key in MODEL_SPECS:
125
  try:
126
- _registry.get_pipeline(key)
127
- loaded[key] = True
128
- except ModelNotAvailable as exc:
129
- loaded[key] = False
130
- logger.warning("Model %s unavailable: %s", key, exc)
131
-
132
- success = any(loaded.values())
133
- return {"success": success, "models": loaded}
134
-
135
-
136
- def _validate_text(text: str) -> str:
137
- if not isinstance(text, str):
138
- raise ValueError("Text input must be a string")
139
- cleaned = text.strip()
140
- if not cleaned:
141
- raise ValueError("Text input cannot be empty")
142
- return cleaned[:512]
143
-
144
-
145
- def _format_sentiment(label: str, score: float, model_key: str) -> Dict[str, Any]:
146
- return {
147
- "label": label.lower(),
148
- "score": round(float(score), 4),
149
- "model": MODEL_SPECS[model_key].model_id,
150
- }
151
-
152
-
153
- def analyze_social_sentiment(text: str) -> Dict[str, Any]:
154
- """Run the Twitter-specific sentiment model."""
155
-
156
- try:
157
- payload = _validate_text(text)
158
- except ValueError as exc:
159
- return {"label": "neutral", "score": 0.0, "error": str(exc)}
160
-
161
- try:
162
- pipe = _registry.get_pipeline("sentiment_twitter")
163
- except ModelNotAvailable as exc:
164
- return {"label": "neutral", "score": 0.0, "error": str(exc)}
165
-
166
- try:
167
- result = pipe(payload)[0]
168
- return _format_sentiment(result["label"], result["score"], "sentiment_twitter")
169
- except Exception as exc: # pragma: no cover - inference heavy
170
- logger.exception("Social sentiment analysis failed")
171
- return {"label": "neutral", "score": 0.0, "error": str(exc)}
172
-
173
-
174
- def analyze_financial_sentiment(text: str) -> Dict[str, Any]:
175
- """Run FinBERT style sentiment analysis."""
176
-
177
  try:
178
- payload = _validate_text(text)
179
- except ValueError as exc:
180
- return {"label": "neutral", "score": 0.0, "error": str(exc)}
181
-
 
 
 
 
 
 
 
182
  try:
183
- pipe = _registry.get_pipeline("sentiment_financial")
184
- except ModelNotAvailable as exc:
185
- return {"label": "neutral", "score": 0.0, "error": str(exc)}
186
-
187
- try:
188
- result = pipe(payload)[0]
189
- return _format_sentiment(result["label"], result["score"], "sentiment_financial")
190
- except Exception as exc: # pragma: no cover - inference heavy
191
- logger.exception("Financial sentiment analysis failed")
192
- return {"label": "neutral", "score": 0.0, "error": str(exc)}
193
-
194
-
195
- def analyze_sentiment(text: str) -> Dict[str, Any]:
196
- """Combine social and financial sentiment signals."""
197
-
198
- social = analyze_social_sentiment(text)
199
- financial = analyze_financial_sentiment(text)
200
-
201
- scores = [entry["score"] if entry.get("label", "").startswith("pos") else -entry["score"]
202
- for entry in (social, financial) if "error" not in entry]
203
-
204
- if not scores:
205
- return {"label": "neutral", "score": 0.0, "details": {"social": social, "financial": financial}}
206
-
207
- avg_score = sum(scores) / len(scores)
208
- label = "positive" if avg_score > 0.15 else "negative" if avg_score < -0.15 else "neutral"
 
 
 
 
 
209
  return {
210
- "label": label,
211
- "score": round(avg_score, 4),
212
- "details": {"social": social, "financial": financial},
213
- }
214
-
215
-
216
- def analyze_crypto_sentiment(text: str, mask_token: str = "[MASK]") -> Dict[str, Any]:
217
- """Use CryptoBERT to infer crypto-native sentiment."""
218
-
219
- try:
220
- payload = _validate_text(text)
221
- except ValueError as exc:
222
- return {"label": "neutral", "score": 0.0, "error": str(exc)}
223
-
224
- try:
225
- pipe = _registry.get_pipeline("crypto_sentiment")
226
- except ModelNotAvailable as exc:
227
- logger.warning("CryptoBERT unavailable: %s", exc)
228
- return analyze_sentiment(text)
229
-
230
- masked = f"{payload} Overall sentiment is {mask_token}."
231
- try:
232
- predictions = pipe(masked, top_k=5)
233
- except Exception as exc: # pragma: no cover
234
- logger.exception("CryptoBERT inference failed")
235
- return analyze_sentiment(text)
236
-
237
- keywords = {
238
- "positive": ["bullish", "positive", "optimistic", "strong", "good"],
239
- "negative": ["bearish", "negative", "weak", "bad", "sell"],
240
- "neutral": ["neutral", "flat", "balanced", "stable"],
241
- }
242
- sentiment_scores = {"positive": 0.0, "negative": 0.0, "neutral": 0.0}
243
- for prediction in predictions:
244
- token = prediction.get("token_str", "").strip().lower()
245
- score = float(prediction.get("score", 0.0))
246
- for label, values in keywords.items():
247
- if any(value in token for value in values):
248
- sentiment_scores[label] += score
249
- break
250
-
251
- label = max(sentiment_scores, key=sentiment_scores.get)
252
- confidence = sentiment_scores[label]
253
- if confidence == 0.0:
254
- return analyze_sentiment(text)
255
-
256
- return {
257
- "label": label,
258
- "score": round(confidence, 4),
259
- "predictions": [
260
- {"token": pred.get("token_str"), "score": round(float(pred.get("score", 0.0)), 4)}
261
- for pred in predictions[:3]
262
- ],
263
- "model": MODEL_SPECS["crypto_sentiment"].model_id,
264
- }
265
-
266
-
267
- def summarize_text(text: str, max_length: int = 200, min_length: int = 40) -> Dict[str, Any]:
268
- """Summarize long-form content using the configured BART model."""
269
-
270
- if not isinstance(text, str) or not text.strip():
271
- return {"summary": "", "model": MODEL_SPECS["summarization"].model_id}
272
-
273
- payload = text.strip()
274
- if len(payload) < min_length:
275
- return {"summary": payload, "model": MODEL_SPECS["summarization"].model_id}
276
-
277
- try:
278
- pipe = _registry.get_pipeline("summarization")
279
- except ModelNotAvailable as exc:
280
- return {"summary": payload[:max_length], "model": "unavailable", "error": str(exc)}
281
-
282
- try:
283
- result = pipe(payload, max_length=max_length, min_length=min_length, do_sample=False)
284
- summary = result[0]["summary_text"].strip()
285
- except Exception as exc: # pragma: no cover - inference heavy
286
- logger.exception("Summarization failed")
287
- summary = payload[:max_length]
288
- return {"summary": summary, "model": MODEL_SPECS["summarization"].model_id, "error": str(exc)}
289
-
290
- return {"summary": summary, "model": MODEL_SPECS["summarization"].model_id}
291
-
292
-
293
- def analyze_news_item(item: Mapping[str, Any]) -> Dict[str, Any]:
294
- """Summarize a news item and attach sentiment metadata."""
295
-
296
- text_parts = [
297
- item.get("title", ""),
298
- item.get("body") or item.get("content") or item.get("description") or "",
299
- ]
300
- combined = ". ".join(part for part in text_parts if part).strip()
301
- summary = summarize_text(combined or item.get("title", ""))
302
- sentiment = analyze_crypto_sentiment(combined or item.get("title", ""))
303
-
304
- return {
305
- "title": item.get("title"),
306
- "summary": summary.get("summary"),
307
- "sentiment": sentiment,
308
- "source": item.get("source"),
309
- "published_at": item.get("published_at") or item.get("date"),
310
- }
311
-
312
-
313
- def analyze_market_text(query: str) -> Dict[str, Any]:
314
- """High-level helper used by the /api/query endpoint."""
315
-
316
- summary = summarize_text(query, max_length=120)
317
- crypto = analyze_crypto_sentiment(query)
318
- fin = analyze_financial_sentiment(query)
319
- social = analyze_social_sentiment(query)
320
-
321
- classification = "sentiment"
322
- lowered = query.lower()
323
- if any(word in lowered for word in ("price", "buy", "sell", "support", "resistance")):
324
- classification = "market"
325
- elif any(word in lowered for word in ("news", "headline", "update")):
326
- classification = "news"
327
-
328
- return {
329
- "summary": summary,
330
- "signals": {
331
- "crypto": crypto,
332
- "financial": fin,
333
- "social": social,
334
  },
335
- "classification": classification,
336
  }
337
 
338
-
339
- def analyze_chart_points(symbol: str, timeframe: str, history: Sequence[Mapping[str, Any]]) -> Dict[str, Any]:
340
- """Generate decision support metadata for chart requests."""
341
-
342
- cleaned = [point for point in history if isinstance(point, Mapping)]
343
- if not cleaned:
344
- return {
345
- "symbol": symbol.upper(),
346
- "timeframe": timeframe,
347
- "error": "No price history available",
348
- }
349
-
350
- prices: List[float] = []
351
- timestamps: List[Any] = []
352
- for point in cleaned:
353
- value = (
354
- point.get("price")
355
- or point.get("close")
356
- or point.get("value")
357
- or point.get("y")
358
- )
359
- if value is None:
360
- continue
361
- try:
362
- prices.append(float(value))
363
- timestamps.append(point.get("timestamp") or point.get("time") or point.get("date"))
364
- except (TypeError, ValueError):
365
- continue
366
-
367
- if not prices:
368
- return {
369
- "symbol": symbol.upper(),
370
- "timeframe": timeframe,
371
- "error": "Price points missing numeric values",
372
- }
373
-
374
- start_price = prices[0]
375
- end_price = prices[-1]
376
- high_price = max(prices)
377
- low_price = min(prices)
378
- change = end_price - start_price
379
- change_pct = (change / start_price) * 100 if start_price else 0.0
380
- direction = "bullish" if change_pct > 0.5 else "bearish" if change_pct < -0.5 else "range-bound"
381
-
382
- description = (
383
- f"{symbol.upper()} moved {change_pct:.2f}% over the last {timeframe}. "
384
- f"High {high_price:.2f} / Low {low_price:.2f}. "
385
- f"Close {end_price:.2f} with {direction} momentum."
386
- )
387
- narrative = analyze_market_text(description)
388
-
389
  return {
390
- "symbol": symbol.upper(),
391
- "timeframe": timeframe,
392
- "change_percent": round(change_pct, 2),
393
- "change_direction": direction,
394
- "latest_price": round(end_price, 4),
395
- "high": round(high_price, 4),
396
- "low": round(low_price, 4),
397
- "narrative": narrative,
398
- "points": len(prices),
399
- "timestamps": {
400
- "start": timestamps[0] if timestamps else None,
401
- "end": timestamps[-1] if timestamps else None,
402
- },
403
  }
404
-
405
-
406
- def registry_status() -> Dict[str, Any]:
407
- """Expose registry information for health checks."""
408
-
409
- info = get_model_info()
410
- info["loaded_models"] = _registry.status()["models_initialized"]
411
- return info
412
-
413
-
414
- __all__ = [
415
- "initialize_models",
416
- "get_model_info",
417
- "registry_status",
418
- "analyze_sentiment",
419
- "analyze_crypto_sentiment",
420
- "analyze_social_sentiment",
421
- "analyze_financial_sentiment",
422
- "summarize_text",
423
- "analyze_news_item",
424
- "analyze_market_text",
425
- "analyze_chart_points",
426
- ]
 
1
  #!/usr/bin/env python3
2
+ """Centralized access to Hugging Face models with ensemble sentiment."""
3
 
4
  from __future__ import annotations
 
5
  import logging
6
  import threading
7
  from dataclasses import dataclass
8
  from typing import Any, Dict, List, Mapping, Optional, Sequence
 
9
  from config import HUGGINGFACE_MODELS, get_settings
10
 
11
+ try:
12
  from transformers import pipeline
 
13
  TRANSFORMERS_AVAILABLE = True
14
+ except ImportError:
15
  TRANSFORMERS_AVAILABLE = False
16
 
 
17
  logger = logging.getLogger(__name__)
18
  settings = get_settings()
19
 
20
+ # Extended Model Catalog
21
+ CRYPTO_SENTIMENT_MODELS = [
22
+ "ElKulako/cryptobert", "kk08/CryptoBERT",
23
+ "burakutf/finetuned-finbert-crypto", "mathugo/crypto_news_bert"
24
+ ]
25
+ SOCIAL_SENTIMENT_MODELS = [
26
+ "svalabs/twitter-xlm-roberta-bitcoin-sentiment",
27
+ "mayurjadhav/crypto-sentiment-model"
28
+ ]
29
+ FINANCIAL_SENTIMENT_MODELS = ["ProsusAI/finbert", "cardiffnlp/twitter-roberta-base-sentiment"]
30
+ NEWS_SENTIMENT_MODELS = ["mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis"]
31
+ DECISION_MODELS = ["agarkovv/CryptoTrader-LM"]
32
 
33
  @dataclass(frozen=True)
34
  class PipelineSpec:
 
 
35
  key: str
36
  task: str
37
  model_id: str
38
  requires_auth: bool = False
39
+ category: str = "sentiment"
40
+
41
+ MODEL_SPECS: Dict[str, PipelineSpec] = {}
42
+
43
+ # Legacy models
44
+ for lk in ["sentiment_twitter", "sentiment_financial", "summarization", "crypto_sentiment"]:
45
+ if lk in HUGGINGFACE_MODELS:
46
+ MODEL_SPECS[lk] = PipelineSpec(
47
+ key=lk,
48
+ task="sentiment-analysis" if "sentiment" in lk else "summarization",
49
+ model_id=HUGGINGFACE_MODELS[lk],
50
+ category="legacy"
51
+ )
52
 
53
+ # Crypto sentiment
54
+ for i, mid in enumerate(CRYPTO_SENTIMENT_MODELS):
55
+ MODEL_SPECS[f"crypto_sent_{i}"] = PipelineSpec(
56
+ key=f"crypto_sent_{i}", task="sentiment-analysis", model_id=mid,
57
+ category="crypto_sentiment", requires_auth=("ElKulako" in mid)
58
+ )
59
 
60
+ # Social
61
+ for i, mid in enumerate(SOCIAL_SENTIMENT_MODELS):
62
+ MODEL_SPECS[f"social_sent_{i}"] = PipelineSpec(
63
+ key=f"social_sent_{i}", task="sentiment-analysis", model_id=mid, category="social_sentiment"
64
+ )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
 
66
+ # Financial
67
+ for i, mid in enumerate(FINANCIAL_SENTIMENT_MODELS):
68
+ MODEL_SPECS[f"financial_sent_{i}"] = PipelineSpec(
69
+ key=f"financial_sent_{i}", task="sentiment-analysis", model_id=mid, category="financial_sentiment"
70
+ )
71
 
72
+ # News
73
+ for i, mid in enumerate(NEWS_SENTIMENT_MODELS):
74
+ MODEL_SPECS[f"news_sent_{i}"] = PipelineSpec(
75
+ key=f"news_sent_{i}", task="sentiment-analysis", model_id=mid, category="news_sentiment"
76
+ )
77
 
78
+ class ModelNotAvailable(RuntimeError): pass
79
 
80
  class ModelRegistry:
81
+ def __init__(self):
82
+ self._pipelines = {}
 
 
83
  self._lock = threading.Lock()
84
+ self._initialized = False
85
 
86
  def get_pipeline(self, key: str):
87
  if not TRANSFORMERS_AVAILABLE:
88
+ raise ModelNotAvailable("transformers not installed")
89
+ if key not in MODEL_SPECS:
90
+ raise ModelNotAvailable(f"Unknown key: {key}")
91
+
92
  spec = MODEL_SPECS[key]
93
  if key in self._pipelines:
94
  return self._pipelines[key]
95
+
96
  with self._lock:
97
  if key in self._pipelines:
98
  return self._pipelines[key]
99
+
100
+ auth = settings.hf_token if spec.requires_auth else None
101
+ logger.info(f"Loading model: {spec.model_id}")
 
 
 
102
  try:
103
+ self._pipelines[key] = pipeline(spec.task, model=spec.model_id, tokenizer=spec.model_id, use_auth_token=auth)
104
+ except Exception as e:
105
+ logger.exception(f"Failed to load {spec.model_id}")
106
+ raise ModelNotAvailable(str(e)) from e
107
+
 
 
 
 
 
108
  return self._pipelines[key]
109
 
110
+ def initialize_models(self):
111
+ if self._initialized:
112
+ return {"status": "already_initialized", "models_loaded": len(self._pipelines)}
113
+ if not TRANSFORMERS_AVAILABLE:
114
+ return {"status": "transformers_not_available", "models_loaded": 0}
115
+
116
+ loaded, failed = [], []
117
+ for key in ["crypto_sent_0", "financial_sent_0"]:
118
+ try:
119
+ self.get_pipeline(key)
120
+ loaded.append(key)
121
+ except Exception as e:
122
+ failed.append((key, str(e)))
123
+
124
+ self._initialized = True
125
+ return {"status": "initialized", "models_loaded": len(loaded), "loaded": loaded, "failed": failed}
126
 
127
  _registry = ModelRegistry()
128
 
129
+ def initialize_models(): return _registry.initialize_models()
130
 
131
+ def ensemble_crypto_sentiment(text: str) -> Dict[str, Any]:
132
+ if not TRANSFORMERS_AVAILABLE:
133
+ return {"label": "neutral", "confidence": 0.0, "scores": {}, "model_count": 0, "error": "transformers N/A"}
134
+
135
+ results, labels_count, total_conf = {}, {"bullish": 0, "bearish": 0, "neutral": 0}, 0.0
136
+
137
+ for key in ["crypto_sent_0", "crypto_sent_1"]:
 
 
 
 
 
 
138
  try:
139
+ pipe = _registry.get_pipeline(key)
140
+ res = pipe(text[:512])
141
+ if isinstance(res, list) and res: res = res[0]
142
+
143
+ label = res.get("label", "NEUTRAL").upper()
144
+ score = res.get("score", 0.5)
145
+
146
+ mapped = "bullish" if "POSITIVE" in label or "BULLISH" in label else ("bearish" if "NEGATIVE" in label or "BEARISH" in label else "neutral")
147
+
148
+ spec = MODEL_SPECS[key]
149
+ results[spec.model_id] = {"label": mapped, "score": score}
150
+ labels_count[mapped] += 1
151
+ total_conf += score
152
+ except Exception as e:
153
+ logger.warning(f"Ensemble failed for {key}: {e}")
154
+
155
+ if not results:
156
+ return {"label": "neutral", "confidence": 0.0, "scores": {}, "model_count": 0}
157
+
158
+ final = max(labels_count, key=labels_count.get)
159
+ avg_conf = total_conf / len(results)
160
+
161
+ return {"label": final, "confidence": avg_conf, "scores": results, "model_count": len(results)}
162
+
163
+ def analyze_crypto_sentiment(text: str): return ensemble_crypto_sentiment(text)
164
+
165
+ def analyze_financial_sentiment(text: str):
166
+ if not TRANSFORMERS_AVAILABLE:
167
+ return {"label": "neutral", "score": 0.5, "error": "transformers N/A"}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
168
  try:
169
+ pipe = _registry.get_pipeline("financial_sent_0")
170
+ res = pipe(text[:512])
171
+ if isinstance(res, list) and res: res = res[0]
172
+ return {"label": res.get("label", "neutral").lower(), "score": res.get("score", 0.5)}
173
+ except Exception as e:
174
+ logger.error(f"Financial sentiment failed: {e}")
175
+ return {"label": "neutral", "score": 0.5, "error": str(e)}
176
+
177
+ def analyze_social_sentiment(text: str):
178
+ if not TRANSFORMERS_AVAILABLE:
179
+ return {"label": "neutral", "score": 0.5, "error": "transformers N/A"}
180
  try:
181
+ pipe = _registry.get_pipeline("social_sent_0")
182
+ res = pipe(text[:512])
183
+ if isinstance(res, list) and res: res = res[0]
184
+ return {"label": res.get("label", "neutral").lower(), "score": res.get("score", 0.5)}
185
+ except Exception as e:
186
+ logger.error(f"Social sentiment failed: {e}")
187
+ return {"label": "neutral", "score": 0.5, "error": str(e)}
188
+
189
+ def analyze_market_text(text: str): return ensemble_crypto_sentiment(text)
190
+
191
+ def analyze_chart_points(data: Sequence[Mapping[str, Any]], indicators: Optional[List[str]] = None):
192
+ if not data: return {"trend": "neutral", "strength": 0, "analysis": "No data"}
193
+
194
+ prices = [float(p.get("price", 0)) for p in data if p.get("price")]
195
+ if not prices: return {"trend": "neutral", "strength": 0, "analysis": "No price data"}
196
+
197
+ first, last = prices[0], prices[-1]
198
+ change = ((last - first) / first * 100) if first > 0 else 0
199
+
200
+ if change > 5: trend, strength = "bullish", min(abs(change) / 10, 1.0)
201
+ elif change < -5: trend, strength = "bearish", min(abs(change) / 10, 1.0)
202
+ else: trend, strength = "neutral", abs(change) / 5
203
+
204
+ return {"trend": trend, "strength": strength, "change_pct": change, "support": min(prices), "resistance": max(prices), "analysis": f"Price moved {change:.2f}% showing {trend} trend"}
205
+
206
+ def analyze_news_item(item: Dict[str, Any]):
207
+ text = item.get("title", "") + " " + item.get("description", "")
208
+ sent = ensemble_crypto_sentiment(text)
209
+ return {**item, "sentiment": sent["label"], "sentiment_confidence": sent["confidence"], "sentiment_details": sent}
210
+
211
+ def get_model_info():
212
  return {
213
+ "transformers_available": TRANSFORMERS_AVAILABLE,
214
+ "hf_auth_configured": bool(settings.hf_token),
215
+ "models_initialized": _registry._initialized,
216
+ "models_loaded": len(_registry._pipelines),
217
+ "model_catalog": {
218
+ "crypto_sentiment": CRYPTO_SENTIMENT_MODELS,
219
+ "social_sentiment": SOCIAL_SENTIMENT_MODELS,
220
+ "financial_sentiment": FINANCIAL_SENTIMENT_MODELS,
221
+ "news_sentiment": NEWS_SENTIMENT_MODELS,
222
+ "decision": DECISION_MODELS
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
223
  },
224
+ "total_models": len(MODEL_SPECS)
225
  }
226
 
227
+ def registry_status():
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
228
  return {
229
+ "initialized": _registry._initialized,
230
+ "pipelines_loaded": len(_registry._pipelines),
231
+ "available_models": list(MODEL_SPECS.keys()),
232
+ "transformers_available": TRANSFORMERS_AVAILABLE
 
 
 
 
 
 
 
 
 
233
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
backend/services/__pycache__/hf_registry.cpython-312.pyc ADDED
Binary file (8.05 kB). View file
 
backend/services/hf_registry.py CHANGED
@@ -1,34 +1,47 @@
1
  from __future__ import annotations
2
- import os
3
- import time
4
- import random
5
  from typing import Dict, Any, List, Literal, Optional
6
  import httpx
7
 
8
  HF_API_MODELS = "https://huggingface.co/api/models"
9
  HF_API_DATASETS = "https://huggingface.co/api/datasets"
10
-
11
- REFRESH_INTERVAL_SEC = int(os.getenv("HF_REGISTRY_REFRESH_SEC", "21600")) # 6h
12
  HTTP_TIMEOUT = float(os.getenv("HF_HTTP_TIMEOUT", "8.0"))
13
 
14
- _SEED_MODELS = [
15
- "ElKulako/cryptobert",
16
- "kk08/CryptoBERT",
17
- ]
18
- _SEED_DATASETS = [
19
- "linxy/CryptoCoin",
20
- "WinkingFace/CryptoLM-Bitcoin-BTC-USDT",
21
- "WinkingFace/CryptoLM-Ethereum-ETH-USDT",
22
- "WinkingFace/CryptoLM-Solana-SOL-USDT",
23
- "WinkingFace/CryptoLM-Ripple-XRP-USDT",
24
- ]
25
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
  class HFRegistry:
28
- def __init__(self) -> None:
29
  self.models: Dict[str, Dict[str, Any]] = {}
30
  self.datasets: Dict[str, Dict[str, Any]] = {}
31
- self.last_refresh: float = 0.0
32
  self.fail_reason: Optional[str] = None
33
 
34
  async def _hf_json(self, url: str, params: Dict[str, Any]) -> Any:
@@ -39,14 +52,17 @@ class HFRegistry:
39
 
40
  async def refresh(self) -> Dict[str, Any]:
41
  try:
 
42
  for name in _SEED_MODELS:
43
  self.models.setdefault(name, {"id": name, "source": "seed", "pipeline_tag": "sentiment-analysis"})
44
- for name in _SEED_DATASETS:
45
- self.datasets.setdefault(name, {"id": name, "source": "seed"})
46
-
47
- q_crypto = {"search": "crypto", "limit": 50}
 
 
 
48
  q_sent = {"pipeline_tag": "sentiment-analysis", "search": "crypto", "limit": 50}
49
-
50
  models = await self._hf_json(HF_API_MODELS, q_sent)
51
  for m in models or []:
52
  mid = m.get("modelId") or m.get("id") or m.get("name")
@@ -57,21 +73,35 @@ class HFRegistry:
57
  "likes": m.get("likes"),
58
  "downloads": m.get("downloads"),
59
  "tags": m.get("tags") or [],
60
- "source": "hub",
61
  }
62
-
 
63
  datasets = await self._hf_json(HF_API_DATASETS, q_crypto)
64
  for d in datasets or []:
65
  did = d.get("id") or d.get("name")
66
  if not did: continue
 
 
 
 
 
 
 
 
 
 
 
 
67
  self.datasets[did] = {
68
  "id": did,
69
  "likes": d.get("likes"),
70
  "downloads": d.get("downloads"),
71
  "tags": d.get("tags") or [],
72
- "source": "hub",
 
73
  }
74
-
75
  self.last_refresh = time.time()
76
  self.fail_reason = None
77
  return {"ok": True, "models": len(self.models), "datasets": len(self.datasets)}
@@ -79,10 +109,13 @@ class HFRegistry:
79
  self.fail_reason = str(e)
80
  return {"ok": False, "error": self.fail_reason, "models": len(self.models), "datasets": len(self.datasets)}
81
 
82
- def list(self, kind: Literal["models","datasets"]="models") -> List[Dict[str, Any]]:
83
- return list(self.models.values()) if kind == "models" else list(self.datasets.values())
 
 
 
84
 
85
- def health(self) -> Dict[str, Any]:
86
  age = time.time() - (self.last_refresh or 0)
87
  return {
88
  "ok": self.last_refresh > 0 and (self.fail_reason is None),
@@ -90,24 +123,20 @@ class HFRegistry:
90
  "age_sec": age,
91
  "fail_reason": self.fail_reason,
92
  "counts": {"models": len(self.models), "datasets": len(self.datasets)},
93
- "interval_sec": REFRESH_INTERVAL_SEC,
94
  }
95
 
96
-
97
  REGISTRY = HFRegistry()
98
 
99
-
100
- async def periodic_refresh(loop_sleep: int = REFRESH_INTERVAL_SEC) -> None:
101
  await REGISTRY.refresh()
102
  await _sleep(int(loop_sleep * random.uniform(0.5, 0.9)))
103
  while True:
104
  await REGISTRY.refresh()
105
  await _sleep(loop_sleep)
106
 
107
-
108
- async def _sleep(sec: int) -> None:
109
  import asyncio
110
  try:
111
  await asyncio.sleep(sec)
112
- except Exception:
113
- pass
 
1
  from __future__ import annotations
2
+ import os, time, random
 
 
3
  from typing import Dict, Any, List, Literal, Optional
4
  import httpx
5
 
6
  HF_API_MODELS = "https://huggingface.co/api/models"
7
  HF_API_DATASETS = "https://huggingface.co/api/datasets"
8
+ REFRESH_INTERVAL_SEC = int(os.getenv("HF_REGISTRY_REFRESH_SEC", "21600"))
 
9
  HTTP_TIMEOUT = float(os.getenv("HF_HTTP_TIMEOUT", "8.0"))
10
 
11
+ # Curated Crypto Datasets
12
+ CRYPTO_DATASETS = {
13
+ "price": [
14
+ "paperswithbacktest/Cryptocurrencies-Daily-Price",
15
+ "linxy/CryptoCoin",
16
+ "sebdg/crypto_data",
17
+ "Farmaanaa/bitcoin_price_timeseries",
18
+ "WinkingFace/CryptoLM-Bitcoin-BTC-USDT",
19
+ "WinkingFace/CryptoLM-Ethereum-ETH-USDT",
20
+ "WinkingFace/CryptoLM-Ripple-XRP-USDT",
21
+ ],
22
+ "news_raw": [
23
+ "flowfree/crypto-news-headlines",
24
+ "edaschau/bitcoin_news",
25
+ ],
26
+ "news_labeled": [
27
+ "SahandNZ/cryptonews-articles-with-price-momentum-labels",
28
+ "tahamajs/bitcoin-individual-news-dataset",
29
+ "tahamajs/bitcoin-enhanced-prediction-dataset-with-comprehensive-news",
30
+ "tahamajs/bitcoin-prediction-dataset-with-local-news-summaries",
31
+ "arad1367/Crypto_Semantic_News",
32
+ ]
33
+ }
34
+
35
+ _SEED_MODELS = ["ElKulako/cryptobert", "kk08/CryptoBERT"]
36
+ _SEED_DATASETS = []
37
+ for cat in CRYPTO_DATASETS.values():
38
+ _SEED_DATASETS.extend(cat)
39
 
40
  class HFRegistry:
41
+ def __init__(self):
42
  self.models: Dict[str, Dict[str, Any]] = {}
43
  self.datasets: Dict[str, Dict[str, Any]] = {}
44
+ self.last_refresh = 0.0
45
  self.fail_reason: Optional[str] = None
46
 
47
  async def _hf_json(self, url: str, params: Dict[str, Any]) -> Any:
 
52
 
53
  async def refresh(self) -> Dict[str, Any]:
54
  try:
55
+ # Seed models
56
  for name in _SEED_MODELS:
57
  self.models.setdefault(name, {"id": name, "source": "seed", "pipeline_tag": "sentiment-analysis"})
58
+
59
+ # Seed datasets with category metadata
60
+ for category, dataset_list in CRYPTO_DATASETS.items():
61
+ for name in dataset_list:
62
+ self.datasets.setdefault(name, {"id": name, "source": "seed", "category": category, "tags": ["crypto", category]})
63
+
64
+ # Fetch from HF Hub
65
  q_sent = {"pipeline_tag": "sentiment-analysis", "search": "crypto", "limit": 50}
 
66
  models = await self._hf_json(HF_API_MODELS, q_sent)
67
  for m in models or []:
68
  mid = m.get("modelId") or m.get("id") or m.get("name")
 
73
  "likes": m.get("likes"),
74
  "downloads": m.get("downloads"),
75
  "tags": m.get("tags") or [],
76
+ "source": "hub"
77
  }
78
+
79
+ q_crypto = {"search": "crypto", "limit": 100}
80
  datasets = await self._hf_json(HF_API_DATASETS, q_crypto)
81
  for d in datasets or []:
82
  did = d.get("id") or d.get("name")
83
  if not did: continue
84
+ # Infer category from tags or name
85
+ category = "other"
86
+ tags_str = " ".join(d.get("tags") or []).lower()
87
+ name_lower = did.lower()
88
+ if "price" in tags_str or "ohlc" in tags_str or "price" in name_lower:
89
+ category = "price"
90
+ elif "news" in tags_str or "news" in name_lower:
91
+ if "label" in tags_str or "sentiment" in tags_str:
92
+ category = "news_labeled"
93
+ else:
94
+ category = "news_raw"
95
+
96
  self.datasets[did] = {
97
  "id": did,
98
  "likes": d.get("likes"),
99
  "downloads": d.get("downloads"),
100
  "tags": d.get("tags") or [],
101
+ "category": category,
102
+ "source": "hub"
103
  }
104
+
105
  self.last_refresh = time.time()
106
  self.fail_reason = None
107
  return {"ok": True, "models": len(self.models), "datasets": len(self.datasets)}
 
109
  self.fail_reason = str(e)
110
  return {"ok": False, "error": self.fail_reason, "models": len(self.models), "datasets": len(self.datasets)}
111
 
112
+ def list(self, kind: Literal["models","datasets"]="models", category: Optional[str]=None) -> List[Dict[str, Any]]:
113
+ items = list(self.models.values()) if kind == "models" else list(self.datasets.values())
114
+ if category and kind == "datasets":
115
+ items = [d for d in items if d.get("category") == category]
116
+ return items
117
 
118
+ def health(self):
119
  age = time.time() - (self.last_refresh or 0)
120
  return {
121
  "ok": self.last_refresh > 0 and (self.fail_reason is None),
 
123
  "age_sec": age,
124
  "fail_reason": self.fail_reason,
125
  "counts": {"models": len(self.models), "datasets": len(self.datasets)},
126
+ "interval_sec": REFRESH_INTERVAL_SEC
127
  }
128
 
 
129
  REGISTRY = HFRegistry()
130
 
131
+ async def periodic_refresh(loop_sleep: int = REFRESH_INTERVAL_SEC):
 
132
  await REGISTRY.refresh()
133
  await _sleep(int(loop_sleep * random.uniform(0.5, 0.9)))
134
  while True:
135
  await REGISTRY.refresh()
136
  await _sleep(loop_sleep)
137
 
138
+ async def _sleep(sec: int):
 
139
  import asyncio
140
  try:
141
  await asyncio.sleep(sec)
142
+ except: pass
 
hf_unified_server.py CHANGED
@@ -34,21 +34,21 @@ logging.basicConfig(level=logging.INFO)
34
  logger = logging.getLogger(__name__)
35
 
36
  # Create FastAPI app
37
- app = FastAPI(
38
- title="Cryptocurrency Data & Analysis API",
39
- description="Complete API for cryptocurrency data, market analysis, and trading signals",
40
- version="3.0.0"
41
- )
42
-
43
- # CORS
44
- app.add_middleware(
45
- CORSMiddleware,
46
- allow_origins=["*"],
47
- allow_credentials=True,
48
- allow_methods=["*"],
49
- allow_headers=["*"],
50
- )
51
-
52
  # Runtime state
53
  START_TIME = time.time()
54
  cache = {"ohlcv": {}, "prices": {}, "market_data": {}, "providers": [], "last_update": None}
@@ -56,41 +56,41 @@ settings = get_settings()
56
  market_collector = MarketDataCollector()
57
  news_collector = NewsCollector()
58
  provider_collector = ProviderStatusCollector()
59
-
60
  # Load providers config
61
  WORKSPACE_ROOT = Path(__file__).parent
62
  PROVIDERS_CONFIG_PATH = settings.providers_config_path
63
-
64
- def load_providers_config():
65
- """Load providers from providers_config_extended.json"""
66
- try:
67
- if PROVIDERS_CONFIG_PATH.exists():
68
- with open(PROVIDERS_CONFIG_PATH, 'r', encoding='utf-8') as f:
69
- config = json.load(f)
70
- providers = config.get('providers', {})
71
- logger.info(f"✅ Loaded {len(providers)} providers from providers_config_extended.json")
72
- return providers
73
- else:
74
- logger.warning(f"⚠️ providers_config_extended.json not found at {PROVIDERS_CONFIG_PATH}")
75
- return {}
76
- except Exception as e:
77
- logger.error(f"❌ Error loading providers config: {e}")
78
- return {}
79
-
80
- # Load providers at startup
81
- PROVIDERS_CONFIG = load_providers_config()
82
-
83
  # Mount static files (CSS, JS)
84
  try:
85
  static_path = WORKSPACE_ROOT / "static"
86
- if static_path.exists():
87
- app.mount("/static", StaticFiles(directory=str(static_path)), name="static")
88
- logger.info(f"✅ Static files mounted from {static_path}")
89
- else:
90
- logger.warning(f"⚠️ Static directory not found: {static_path}")
91
- except Exception as e:
92
- logger.error(f"❌ Error mounting static files: {e}")
93
-
94
  # ============================================================================
95
  # Helper utilities & Data Fetching Functions
96
  # ============================================================================
@@ -192,12 +192,12 @@ async def fetch_binance_ticker(symbol: str):
192
  "volume_24h": coin.get("volume_24h"),
193
  "quote_volume_24h": coin.get("volume_24h"),
194
  }
195
-
196
-
197
- # ============================================================================
198
- # Core Endpoints
199
- # ============================================================================
200
-
201
  @app.get("/health")
202
  async def health():
203
  """System health check using shared collectors."""
@@ -236,8 +236,8 @@ async def health():
236
  "providers_loaded": market_status.get("count", 0),
237
  "services": service_states,
238
  }
239
-
240
-
241
  @app.get("/info")
242
  async def info():
243
  """System information"""
@@ -268,8 +268,8 @@ async def info():
268
  ],
269
  "ai_registry": registry_status(),
270
  }
271
-
272
-
273
  @app.get("/api/providers")
274
  async def get_providers():
275
  """Get list of API providers and their health."""
@@ -298,239 +298,239 @@ async def get_providers():
298
  "source": str(PROVIDERS_CONFIG_PATH),
299
  "last_updated": datetime.utcnow().isoformat(),
300
  }
301
-
302
-
303
- # ============================================================================
304
- # OHLCV Data Endpoint
305
- # ============================================================================
306
-
307
- @app.get("/api/ohlcv")
308
- async def get_ohlcv(
309
- symbol: str = Query("BTCUSDT", description="Trading pair symbol"),
310
- interval: str = Query("1h", description="Time interval (1m, 5m, 15m, 1h, 4h, 1d)"),
311
- limit: int = Query(100, ge=1, le=1000, description="Number of candles")
312
- ):
313
- """
314
- Get OHLCV (candlestick) data for a trading pair
315
-
316
- Supported intervals: 1m, 5m, 15m, 30m, 1h, 4h, 1d
317
- """
318
- try:
319
- # Check cache
320
- cache_key = f"{symbol}_{interval}_{limit}"
321
- if cache_key in cache["ohlcv"]:
322
- cached_data, cached_time = cache["ohlcv"][cache_key]
323
- if (datetime.now() - cached_time).seconds < 60: # 60s cache
324
- return {"symbol": symbol, "interval": interval, "data": cached_data, "source": "cache"}
325
-
326
- # Fetch from Binance
327
- ohlcv_data = await fetch_binance_ohlcv(symbol, interval, limit)
328
-
329
- if ohlcv_data:
330
- # Update cache
331
- cache["ohlcv"][cache_key] = (ohlcv_data, datetime.now())
332
-
333
- return {
334
- "symbol": symbol,
335
- "interval": interval,
336
- "count": len(ohlcv_data),
337
- "data": ohlcv_data,
338
- "source": "binance",
339
- "timestamp": datetime.now().isoformat()
340
- }
341
- else:
342
- raise HTTPException(status_code=503, detail="Unable to fetch OHLCV data")
343
-
344
- except HTTPException:
345
- raise
346
- except Exception as e:
347
- logger.error(f"Error in get_ohlcv: {e}")
348
- raise HTTPException(status_code=500, detail=str(e))
349
-
350
-
351
- # ============================================================================
352
- # Crypto Prices Endpoints
353
- # ============================================================================
354
-
355
- @app.get("/api/crypto/prices/top")
356
- async def get_top_prices(limit: int = Query(10, ge=1, le=100, description="Number of top cryptocurrencies")):
357
- """Get top cryptocurrencies by market cap"""
358
- try:
359
- # Check cache
360
- cache_key = f"top_{limit}"
361
- if cache_key in cache["prices"]:
362
- cached_data, cached_time = cache["prices"][cache_key]
363
- if (datetime.now() - cached_time).seconds < 60:
364
- return {"data": cached_data, "source": "cache"}
365
-
366
- # Fetch from CoinGecko
367
- prices = await fetch_coingecko_prices(limit=limit)
368
-
369
- if prices:
370
- # Update cache
371
- cache["prices"][cache_key] = (prices, datetime.now())
372
-
373
- return {
374
- "count": len(prices),
375
- "data": prices,
376
- "source": "coingecko",
377
- "timestamp": datetime.now().isoformat()
378
- }
379
- else:
380
- raise HTTPException(status_code=503, detail="Unable to fetch price data")
381
-
382
- except HTTPException:
383
- raise
384
- except Exception as e:
385
- logger.error(f"Error in get_top_prices: {e}")
386
- raise HTTPException(status_code=500, detail=str(e))
387
-
388
-
389
- @app.get("/api/crypto/price/{symbol}")
390
- async def get_single_price(symbol: str):
391
- """Get price for a single cryptocurrency"""
392
- try:
393
- # Try Binance first for common pairs
394
- binance_symbol = f"{symbol.upper()}USDT"
395
- ticker = await fetch_binance_ticker(binance_symbol)
396
-
397
- if ticker:
398
- return {
399
- "symbol": symbol.upper(),
400
- "price": ticker,
401
- "source": "binance",
402
- "timestamp": datetime.now().isoformat()
403
- }
404
-
405
- # Fallback to CoinGecko
406
- prices = await fetch_coingecko_prices([symbol])
407
- if prices:
408
- return {
409
- "symbol": symbol.upper(),
410
- "price": prices[0],
411
- "source": "coingecko",
412
- "timestamp": datetime.now().isoformat()
413
- }
414
-
415
- raise HTTPException(status_code=404, detail=f"Price data not found for {symbol}")
416
-
417
- except HTTPException:
418
- raise
419
- except Exception as e:
420
- logger.error(f"Error in get_single_price: {e}")
421
- raise HTTPException(status_code=500, detail=str(e))
422
-
423
-
424
- @app.get("/api/crypto/market-overview")
425
- async def get_market_overview():
426
- """Get comprehensive market overview"""
427
- try:
428
- # Fetch top 20 coins
429
- prices = await fetch_coingecko_prices(limit=20)
430
-
431
- if not prices:
432
- raise HTTPException(status_code=503, detail="Unable to fetch market data")
433
-
434
- # Calculate market stats
435
- total_market_cap = sum(p.get("market_cap", 0) or 0 for p in prices)
436
- total_volume = sum(p.get("total_volume", 0) or 0 for p in prices)
437
-
438
- # Sort by 24h change
439
- gainers = sorted(
440
- [p for p in prices if p.get("price_change_percentage_24h")],
441
- key=lambda x: x.get("price_change_percentage_24h", 0),
442
- reverse=True
443
- )[:5]
444
-
445
- losers = sorted(
446
- [p for p in prices if p.get("price_change_percentage_24h")],
447
- key=lambda x: x.get("price_change_percentage_24h", 0)
448
- )[:5]
449
-
450
- return {
451
- "total_market_cap": total_market_cap,
452
- "total_volume_24h": total_volume,
453
- "btc_dominance": (prices[0].get("market_cap", 0) / total_market_cap * 100) if total_market_cap > 0 else 0,
454
- "top_gainers": gainers,
455
- "top_losers": losers,
456
- "top_by_volume": sorted(prices, key=lambda x: x.get("total_volume", 0) or 0, reverse=True)[:5],
457
- "timestamp": datetime.now().isoformat()
458
- }
459
-
460
- except HTTPException:
461
- raise
462
- except Exception as e:
463
- logger.error(f"Error in get_market_overview: {e}")
464
- raise HTTPException(status_code=500, detail=str(e))
465
-
466
-
467
- @app.get("/api/market/prices")
468
- async def get_multiple_prices(symbols: str = Query("BTC,ETH,SOL", description="Comma-separated symbols")):
469
- """Get prices for multiple cryptocurrencies"""
470
- try:
471
- symbol_list = [s.strip().upper() for s in symbols.split(",")]
472
-
473
- # Fetch prices
474
- prices_data = []
475
- for symbol in symbol_list:
476
- try:
477
- ticker = await fetch_binance_ticker(f"{symbol}USDT")
478
- if ticker:
479
- prices_data.append(ticker)
480
- except:
481
- continue
482
-
483
- if not prices_data:
484
- # Fallback to CoinGecko
485
- prices_data = await fetch_coingecko_prices(symbol_list)
486
-
487
- return {
488
- "symbols": symbol_list,
489
- "count": len(prices_data),
490
- "data": prices_data,
491
- "timestamp": datetime.now().isoformat()
492
- }
493
-
494
- except Exception as e:
495
- logger.error(f"Error in get_multiple_prices: {e}")
496
- raise HTTPException(status_code=500, detail=str(e))
497
-
498
-
499
- @app.get("/api/market-data/prices")
500
- async def get_market_data_prices(symbols: str = Query("BTC,ETH", description="Comma-separated symbols")):
501
- """Alternative endpoint for market data prices"""
502
- return await get_multiple_prices(symbols)
503
-
504
-
505
- # ============================================================================
506
- # Analysis Endpoints
507
- # ============================================================================
508
-
509
- @app.get("/api/analysis/signals")
510
- async def get_trading_signals(
511
- symbol: str = Query("BTCUSDT", description="Trading pair"),
512
- timeframe: str = Query("1h", description="Timeframe")
513
- ):
514
- """Get trading signals for a symbol"""
515
- try:
516
- # Fetch OHLCV data for analysis
517
- ohlcv = await fetch_binance_ohlcv(symbol, timeframe, 100)
518
-
519
- if not ohlcv:
520
- raise HTTPException(status_code=503, detail="Unable to fetch data for analysis")
521
-
522
- # Simple signal generation (can be enhanced)
523
- latest = ohlcv[-1]
524
- prev = ohlcv[-2] if len(ohlcv) > 1 else latest
525
-
526
- # Calculate simple indicators
527
- close_prices = [c["close"] for c in ohlcv[-20:]]
528
- sma_20 = sum(close_prices) / len(close_prices)
529
-
530
- # Generate signal
531
- trend = "bullish" if latest["close"] > sma_20 else "bearish"
532
- momentum = "strong" if abs(latest["close"] - prev["close"]) / prev["close"] > 0.01 else "weak"
533
-
534
  signal = "buy" if trend == "bullish" and momentum == "strong" else (
535
  "sell" if trend == "bearish" and momentum == "strong" else "hold"
536
  )
@@ -552,126 +552,126 @@ async def get_trading_signals(
552
  "analysis": ai_summary,
553
  "timestamp": datetime.now().isoformat()
554
  }
555
-
556
- except HTTPException:
557
- raise
558
- except Exception as e:
559
- logger.error(f"Error in get_trading_signals: {e}")
560
- raise HTTPException(status_code=500, detail=str(e))
561
-
562
-
563
- @app.get("/api/analysis/smc")
564
- async def get_smc_analysis(symbol: str = Query("BTCUSDT", description="Trading pair")):
565
- """Get Smart Money Concepts (SMC) analysis"""
566
- try:
567
- # Fetch OHLCV data
568
- ohlcv = await fetch_binance_ohlcv(symbol, "1h", 200)
569
-
570
- if not ohlcv:
571
- raise HTTPException(status_code=503, detail="Unable to fetch data")
572
-
573
- # Calculate key levels
574
- highs = [c["high"] for c in ohlcv]
575
- lows = [c["low"] for c in ohlcv]
576
- closes = [c["close"] for c in ohlcv]
577
-
578
- resistance = max(highs[-50:])
579
- support = min(lows[-50:])
580
- current_price = closes[-1]
581
-
582
- # Structure analysis
583
- market_structure = "higher_highs" if closes[-1] > closes[-10] > closes[-20] else "lower_lows"
584
-
585
- return {
586
- "symbol": symbol,
587
- "market_structure": market_structure,
588
- "key_levels": {
589
- "resistance": resistance,
590
- "support": support,
591
- "current_price": current_price,
592
- "mid_point": (resistance + support) / 2
593
- },
594
- "order_blocks": {
595
- "bullish": support,
596
- "bearish": resistance
597
- },
598
- "liquidity_zones": {
599
- "above": resistance,
600
- "below": support
601
- },
602
- "timestamp": datetime.now().isoformat()
603
- }
604
-
605
- except HTTPException:
606
- raise
607
- except Exception as e:
608
- logger.error(f"Error in get_smc_analysis: {e}")
609
- raise HTTPException(status_code=500, detail=str(e))
610
-
611
-
612
- @app.get("/api/scoring/snapshot")
613
- async def get_scoring_snapshot(symbol: str = Query("BTCUSDT", description="Trading pair")):
614
- """Get comprehensive scoring snapshot"""
615
- try:
616
- # Fetch data
617
- ticker = await fetch_binance_ticker(symbol)
618
- ohlcv = await fetch_binance_ohlcv(symbol, "1h", 100)
619
-
620
- if not ticker or not ohlcv:
621
- raise HTTPException(status_code=503, detail="Unable to fetch data")
622
-
623
- # Calculate scores (0-100)
624
- volatility_score = min(abs(ticker["price_change_percent_24h"]) * 5, 100)
625
- volume_score = min((ticker["volume_24h"] / 1000000) * 10, 100)
626
- trend_score = 50 + (ticker["price_change_percent_24h"] * 2)
627
-
628
- # Overall score
629
- overall_score = (volatility_score + volume_score + trend_score) / 3
630
-
631
- return {
632
- "symbol": symbol,
633
- "overall_score": round(overall_score, 2),
634
- "scores": {
635
- "volatility": round(volatility_score, 2),
636
- "volume": round(volume_score, 2),
637
- "trend": round(trend_score, 2),
638
- "momentum": round(50 + ticker["price_change_percent_24h"], 2)
639
- },
640
- "rating": "excellent" if overall_score > 80 else (
641
- "good" if overall_score > 60 else (
642
- "average" if overall_score > 40 else "poor"
643
- )
644
- ),
645
- "timestamp": datetime.now().isoformat()
646
- }
647
-
648
- except HTTPException:
649
- raise
650
- except Exception as e:
651
- logger.error(f"Error in get_scoring_snapshot: {e}")
652
- raise HTTPException(status_code=500, detail=str(e))
653
-
654
-
655
- @app.get("/api/signals")
656
- async def get_all_signals():
657
- """Get signals for multiple assets"""
658
- symbols = ["BTCUSDT", "ETHUSDT", "SOLUSDT", "BNBUSDT"]
659
- signals = []
660
-
661
- for symbol in symbols:
662
- try:
663
- signal_data = await get_trading_signals(symbol, "1h")
664
- signals.append(signal_data)
665
- except:
666
- continue
667
-
668
- return {
669
- "count": len(signals),
670
- "signals": signals,
671
- "timestamp": datetime.now().isoformat()
672
- }
673
-
674
-
675
  @app.get("/api/sentiment")
676
  async def get_sentiment():
677
  """Get market sentiment data"""
@@ -704,12 +704,12 @@ async def get_sentiment():
704
  "analysis": analysis,
705
  "timestamp": datetime.utcnow().isoformat(),
706
  }
707
-
708
-
709
- # ============================================================================
710
- # System Endpoints
711
- # ============================================================================
712
-
713
  @app.get("/api/system/status")
714
  async def get_system_status():
715
  """Get system status"""
@@ -730,8 +730,8 @@ async def get_system_status():
730
  "requests_per_minute": 0,
731
  "timestamp": datetime.utcnow().isoformat(),
732
  }
733
-
734
-
735
  @app.get("/api/system/config")
736
  async def get_system_config():
737
  """Get system configuration"""
@@ -744,89 +744,89 @@ async def get_system_config():
744
  "max_ohlcv_limit": 1000,
745
  "timestamp": datetime.utcnow().isoformat(),
746
  }
747
-
748
-
749
- @app.get("/api/categories")
750
- async def get_categories():
751
- """Get data categories"""
752
- return {
753
- "categories": [
754
- {"name": "market_data", "endpoints": 5, "status": "active"},
755
- {"name": "analysis", "endpoints": 4, "status": "active"},
756
- {"name": "signals", "endpoints": 2, "status": "active"},
757
- {"name": "sentiment", "endpoints": 1, "status": "active"}
758
- ]
759
- }
760
-
761
-
762
- @app.get("/api/rate-limits")
763
- async def get_rate_limits():
764
- """Get rate limit information"""
765
- return {
766
- "rate_limits": [
767
- {"endpoint": "/api/ohlcv", "limit": 1200, "window": "per_minute"},
768
- {"endpoint": "/api/crypto/prices/top", "limit": 600, "window": "per_minute"},
769
- {"endpoint": "/api/analysis/*", "limit": 300, "window": "per_minute"}
770
- ],
771
- "current_usage": {
772
- "requests_this_minute": 0,
773
- "percentage": 0
774
- }
775
- }
776
-
777
-
778
- @app.get("/api/logs")
779
- async def get_logs(limit: int = Query(50, ge=1, le=500)):
780
- """Get recent API logs"""
781
- # Mock logs (can be enhanced with real logging)
782
- logs = []
783
- for i in range(min(limit, 10)):
784
- logs.append({
785
- "timestamp": (datetime.now() - timedelta(minutes=i)).isoformat(),
786
- "endpoint": "/api/ohlcv",
787
- "status": "success",
788
- "response_time_ms": random.randint(50, 200)
789
- })
790
-
791
- return {"logs": logs, "count": len(logs)}
792
-
793
-
794
- @app.get("/api/alerts")
795
- async def get_alerts():
796
- """Get system alerts"""
797
- return {
798
- "alerts": [],
799
- "count": 0,
800
- "timestamp": datetime.now().isoformat()
801
- }
802
-
803
-
804
- # ============================================================================
805
- # HuggingFace Integration Endpoints
806
- # ============================================================================
807
-
808
  @app.get("/api/hf/health")
809
  async def hf_health():
810
  """HuggingFace integration health"""
811
  status = registry_status()
812
  status["timestamp"] = datetime.utcnow().isoformat()
813
  return status
814
-
815
-
816
  @app.post("/api/hf/refresh")
817
  async def hf_refresh():
818
  """Refresh HuggingFace data"""
819
  result = initialize_models()
820
  return {"status": "ok" if result.get("success") else "degraded", **result, "timestamp": datetime.utcnow().isoformat()}
821
-
822
-
823
  @app.get("/api/hf/registry")
824
  async def hf_registry(kind: str = "models"):
825
  """Get HuggingFace registry"""
826
  info = get_model_info()
827
  return {"kind": kind, "items": info.get("model_names", info)}
828
-
829
-
830
  def _resolve_sentiment_payload(payload: Union[List[str], Dict[str, Any]]) -> Dict[str, Any]:
831
  if isinstance(payload, list):
832
  return {"texts": payload, "mode": "auto"}
@@ -866,132 +866,601 @@ async def hf_sentiment(payload: Union[List[str], Dict[str, Any]] = Body(...)):
866
  results.append({"text": text, "result": analysis})
867
 
868
  return {"mode": mode, "results": results, "timestamp": datetime.utcnow().isoformat()}
869
-
870
-
871
- # ============================================================================
872
- # HTML Routes - Serve UI files
873
- # ============================================================================
874
-
875
- @app.get("/", response_class=HTMLResponse)
876
- async def root():
877
- """Serve main admin dashboard (admin.html)"""
878
- admin_path = WORKSPACE_ROOT / "admin.html"
879
- if admin_path.exists():
880
- return FileResponse(admin_path)
881
- return HTMLResponse("<h1>Cryptocurrency Data & Analysis API</h1><p>See <a href='/docs'>/docs</a> for API documentation</p>")
882
-
883
- @app.get("/index.html", response_class=HTMLResponse)
884
- async def index():
885
- """Serve index.html"""
886
- return FileResponse(WORKSPACE_ROOT / "index.html")
887
-
888
- @app.get("/dashboard.html", response_class=HTMLResponse)
889
- async def dashboard():
890
- """Serve dashboard.html"""
891
- return FileResponse(WORKSPACE_ROOT / "dashboard.html")
892
-
893
- @app.get("/dashboard", response_class=HTMLResponse)
894
- async def dashboard_alt():
895
- """Alternative route for dashboard"""
896
- return FileResponse(WORKSPACE_ROOT / "dashboard.html")
897
-
898
- @app.get("/admin.html", response_class=HTMLResponse)
899
- async def admin():
900
- """Serve admin panel"""
901
- return FileResponse(WORKSPACE_ROOT / "admin.html")
902
-
903
- @app.get("/admin", response_class=HTMLResponse)
904
- async def admin_alt():
905
- """Alternative route for admin"""
906
- return FileResponse(WORKSPACE_ROOT / "admin.html")
907
-
908
- @app.get("/hf_console.html", response_class=HTMLResponse)
909
- async def hf_console():
910
- """Serve HuggingFace console"""
911
- return FileResponse(WORKSPACE_ROOT / "hf_console.html")
912
-
913
- @app.get("/console", response_class=HTMLResponse)
914
- async def console_alt():
915
- """Alternative route for HF console"""
916
- return FileResponse(WORKSPACE_ROOT / "hf_console.html")
917
-
918
- @app.get("/pool_management.html", response_class=HTMLResponse)
919
- async def pool_management():
920
- """Serve pool management UI"""
921
- return FileResponse(WORKSPACE_ROOT / "pool_management.html")
922
-
923
- @app.get("/unified_dashboard.html", response_class=HTMLResponse)
924
- async def unified_dashboard():
925
- """Serve unified dashboard"""
926
- return FileResponse(WORKSPACE_ROOT / "unified_dashboard.html")
927
-
928
- @app.get("/simple_overview.html", response_class=HTMLResponse)
929
- async def simple_overview():
930
- """Serve simple overview"""
931
- return FileResponse(WORKSPACE_ROOT / "simple_overview.html")
932
-
933
- # Generic HTML file handler
934
- @app.get("/{filename}.html", response_class=HTMLResponse)
935
- async def serve_html(filename: str):
936
- """Serve any HTML file from workspace root"""
937
- file_path = WORKSPACE_ROOT / f"{filename}.html"
938
- if file_path.exists():
939
- return FileResponse(file_path)
940
- return HTMLResponse(f"<h1>File {filename}.html not found</h1>", status_code=404)
941
-
942
-
943
- # ============================================================================
944
- # Startup Event
945
- # ============================================================================
946
-
947
- @app.on_event("startup")
948
- async def startup_event():
949
- """Initialize on startup"""
950
- logger.info("=" * 70)
951
- logger.info("🚀 Cryptocurrency Data & Analysis API Starting")
952
- logger.info("=" * 70)
953
- logger.info("✓ FastAPI initialized")
954
- logger.info("✓ CORS configured")
955
- logger.info("✓ Cache initialized")
956
- logger.info(f"✓ Providers loaded: {len(PROVIDERS_CONFIG)}")
957
-
958
- # Show loaded HuggingFace Space providers
959
- hf_providers = [p for p in PROVIDERS_CONFIG.keys() if 'huggingface_space' in p]
960
- if hf_providers:
961
- logger.info(f"✓ HuggingFace Space providers: {', '.join(hf_providers)}")
962
-
963
- logger.info("✓ Data sources: Binance, CoinGecko, providers_config_extended.json")
964
-
965
- # Check HTML files
966
- html_files = ["index.html", "dashboard.html", "admin.html", "hf_console.html"]
967
- available_html = [f for f in html_files if (WORKSPACE_ROOT / f).exists()]
968
- logger.info(f"✓ UI files: {len(available_html)}/{len(html_files)} available")
969
-
970
- logger.info("=" * 70)
971
- logger.info("📡 API ready at http://0.0.0.0:7860")
972
- logger.info("📖 Docs at http://0.0.0.0:7860/docs")
973
- logger.info("🎨 UI at http://0.0.0.0:7860/ (admin.html)")
974
- logger.info("=" * 70)
975
-
976
-
977
- # ============================================================================
978
- # Main Entry Point
979
- # ============================================================================
980
-
981
- if __name__ == "__main__":
982
- import uvicorn
983
-
984
- print("=" * 70)
985
- print("🚀 Starting Cryptocurrency Data & Analysis API")
986
- print("=" * 70)
987
- print("📍 Server: http://localhost:7860")
988
- print("📖 API Docs: http://localhost:7860/docs")
989
- print("🔗 Health: http://localhost:7860/health")
990
- print("=" * 70)
991
-
992
- uvicorn.run(
993
- app,
994
- host="0.0.0.0",
995
- port=7860,
996
- log_level="info"
997
- )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
  logger = logging.getLogger(__name__)
35
 
36
  # Create FastAPI app
37
+ app = FastAPI(
38
+ title="Cryptocurrency Data & Analysis API",
39
+ description="Complete API for cryptocurrency data, market analysis, and trading signals",
40
+ version="3.0.0"
41
+ )
42
+
43
+ # CORS
44
+ app.add_middleware(
45
+ CORSMiddleware,
46
+ allow_origins=["*"],
47
+ allow_credentials=True,
48
+ allow_methods=["*"],
49
+ allow_headers=["*"],
50
+ )
51
+
52
  # Runtime state
53
  START_TIME = time.time()
54
  cache = {"ohlcv": {}, "prices": {}, "market_data": {}, "providers": [], "last_update": None}
 
56
  market_collector = MarketDataCollector()
57
  news_collector = NewsCollector()
58
  provider_collector = ProviderStatusCollector()
59
+
60
  # Load providers config
61
  WORKSPACE_ROOT = Path(__file__).parent
62
  PROVIDERS_CONFIG_PATH = settings.providers_config_path
63
+
64
+ def load_providers_config():
65
+ """Load providers from providers_config_extended.json"""
66
+ try:
67
+ if PROVIDERS_CONFIG_PATH.exists():
68
+ with open(PROVIDERS_CONFIG_PATH, 'r', encoding='utf-8') as f:
69
+ config = json.load(f)
70
+ providers = config.get('providers', {})
71
+ logger.info(f"✅ Loaded {len(providers)} providers from providers_config_extended.json")
72
+ return providers
73
+ else:
74
+ logger.warning(f"⚠️ providers_config_extended.json not found at {PROVIDERS_CONFIG_PATH}")
75
+ return {}
76
+ except Exception as e:
77
+ logger.error(f"❌ Error loading providers config: {e}")
78
+ return {}
79
+
80
+ # Load providers at startup
81
+ PROVIDERS_CONFIG = load_providers_config()
82
+
83
  # Mount static files (CSS, JS)
84
  try:
85
  static_path = WORKSPACE_ROOT / "static"
86
+ if static_path.exists():
87
+ app.mount("/static", StaticFiles(directory=str(static_path)), name="static")
88
+ logger.info(f"✅ Static files mounted from {static_path}")
89
+ else:
90
+ logger.warning(f"⚠️ Static directory not found: {static_path}")
91
+ except Exception as e:
92
+ logger.error(f"❌ Error mounting static files: {e}")
93
+
94
  # ============================================================================
95
  # Helper utilities & Data Fetching Functions
96
  # ============================================================================
 
192
  "volume_24h": coin.get("volume_24h"),
193
  "quote_volume_24h": coin.get("volume_24h"),
194
  }
195
+
196
+
197
+ # ============================================================================
198
+ # Core Endpoints
199
+ # ============================================================================
200
+
201
  @app.get("/health")
202
  async def health():
203
  """System health check using shared collectors."""
 
236
  "providers_loaded": market_status.get("count", 0),
237
  "services": service_states,
238
  }
239
+
240
+
241
  @app.get("/info")
242
  async def info():
243
  """System information"""
 
268
  ],
269
  "ai_registry": registry_status(),
270
  }
271
+
272
+
273
  @app.get("/api/providers")
274
  async def get_providers():
275
  """Get list of API providers and their health."""
 
298
  "source": str(PROVIDERS_CONFIG_PATH),
299
  "last_updated": datetime.utcnow().isoformat(),
300
  }
301
+
302
+
303
+ # ============================================================================
304
+ # OHLCV Data Endpoint
305
+ # ============================================================================
306
+
307
+ @app.get("/api/ohlcv")
308
+ async def get_ohlcv(
309
+ symbol: str = Query("BTCUSDT", description="Trading pair symbol"),
310
+ interval: str = Query("1h", description="Time interval (1m, 5m, 15m, 1h, 4h, 1d)"),
311
+ limit: int = Query(100, ge=1, le=1000, description="Number of candles")
312
+ ):
313
+ """
314
+ Get OHLCV (candlestick) data for a trading pair
315
+
316
+ Supported intervals: 1m, 5m, 15m, 30m, 1h, 4h, 1d
317
+ """
318
+ try:
319
+ # Check cache
320
+ cache_key = f"{symbol}_{interval}_{limit}"
321
+ if cache_key in cache["ohlcv"]:
322
+ cached_data, cached_time = cache["ohlcv"][cache_key]
323
+ if (datetime.now() - cached_time).seconds < 60: # 60s cache
324
+ return {"symbol": symbol, "interval": interval, "data": cached_data, "source": "cache"}
325
+
326
+ # Fetch from Binance
327
+ ohlcv_data = await fetch_binance_ohlcv(symbol, interval, limit)
328
+
329
+ if ohlcv_data:
330
+ # Update cache
331
+ cache["ohlcv"][cache_key] = (ohlcv_data, datetime.now())
332
+
333
+ return {
334
+ "symbol": symbol,
335
+ "interval": interval,
336
+ "count": len(ohlcv_data),
337
+ "data": ohlcv_data,
338
+ "source": "binance",
339
+ "timestamp": datetime.now().isoformat()
340
+ }
341
+ else:
342
+ raise HTTPException(status_code=503, detail="Unable to fetch OHLCV data")
343
+
344
+ except HTTPException:
345
+ raise
346
+ except Exception as e:
347
+ logger.error(f"Error in get_ohlcv: {e}")
348
+ raise HTTPException(status_code=500, detail=str(e))
349
+
350
+
351
+ # ============================================================================
352
+ # Crypto Prices Endpoints
353
+ # ============================================================================
354
+
355
+ @app.get("/api/crypto/prices/top")
356
+ async def get_top_prices(limit: int = Query(10, ge=1, le=100, description="Number of top cryptocurrencies")):
357
+ """Get top cryptocurrencies by market cap"""
358
+ try:
359
+ # Check cache
360
+ cache_key = f"top_{limit}"
361
+ if cache_key in cache["prices"]:
362
+ cached_data, cached_time = cache["prices"][cache_key]
363
+ if (datetime.now() - cached_time).seconds < 60:
364
+ return {"data": cached_data, "source": "cache"}
365
+
366
+ # Fetch from CoinGecko
367
+ prices = await fetch_coingecko_prices(limit=limit)
368
+
369
+ if prices:
370
+ # Update cache
371
+ cache["prices"][cache_key] = (prices, datetime.now())
372
+
373
+ return {
374
+ "count": len(prices),
375
+ "data": prices,
376
+ "source": "coingecko",
377
+ "timestamp": datetime.now().isoformat()
378
+ }
379
+ else:
380
+ raise HTTPException(status_code=503, detail="Unable to fetch price data")
381
+
382
+ except HTTPException:
383
+ raise
384
+ except Exception as e:
385
+ logger.error(f"Error in get_top_prices: {e}")
386
+ raise HTTPException(status_code=500, detail=str(e))
387
+
388
+
389
+ @app.get("/api/crypto/price/{symbol}")
390
+ async def get_single_price(symbol: str):
391
+ """Get price for a single cryptocurrency"""
392
+ try:
393
+ # Try Binance first for common pairs
394
+ binance_symbol = f"{symbol.upper()}USDT"
395
+ ticker = await fetch_binance_ticker(binance_symbol)
396
+
397
+ if ticker:
398
+ return {
399
+ "symbol": symbol.upper(),
400
+ "price": ticker,
401
+ "source": "binance",
402
+ "timestamp": datetime.now().isoformat()
403
+ }
404
+
405
+ # Fallback to CoinGecko
406
+ prices = await fetch_coingecko_prices([symbol])
407
+ if prices:
408
+ return {
409
+ "symbol": symbol.upper(),
410
+ "price": prices[0],
411
+ "source": "coingecko",
412
+ "timestamp": datetime.now().isoformat()
413
+ }
414
+
415
+ raise HTTPException(status_code=404, detail=f"Price data not found for {symbol}")
416
+
417
+ except HTTPException:
418
+ raise
419
+ except Exception as e:
420
+ logger.error(f"Error in get_single_price: {e}")
421
+ raise HTTPException(status_code=500, detail=str(e))
422
+
423
+
424
+ @app.get("/api/crypto/market-overview")
425
+ async def get_market_overview():
426
+ """Get comprehensive market overview"""
427
+ try:
428
+ # Fetch top 20 coins
429
+ prices = await fetch_coingecko_prices(limit=20)
430
+
431
+ if not prices:
432
+ raise HTTPException(status_code=503, detail="Unable to fetch market data")
433
+
434
+ # Calculate market stats
435
+ total_market_cap = sum(p.get("market_cap", 0) or 0 for p in prices)
436
+ total_volume = sum(p.get("total_volume", 0) or 0 for p in prices)
437
+
438
+ # Sort by 24h change
439
+ gainers = sorted(
440
+ [p for p in prices if p.get("price_change_percentage_24h")],
441
+ key=lambda x: x.get("price_change_percentage_24h", 0),
442
+ reverse=True
443
+ )[:5]
444
+
445
+ losers = sorted(
446
+ [p for p in prices if p.get("price_change_percentage_24h")],
447
+ key=lambda x: x.get("price_change_percentage_24h", 0)
448
+ )[:5]
449
+
450
+ return {
451
+ "total_market_cap": total_market_cap,
452
+ "total_volume_24h": total_volume,
453
+ "btc_dominance": (prices[0].get("market_cap", 0) / total_market_cap * 100) if total_market_cap > 0 else 0,
454
+ "top_gainers": gainers,
455
+ "top_losers": losers,
456
+ "top_by_volume": sorted(prices, key=lambda x: x.get("total_volume", 0) or 0, reverse=True)[:5],
457
+ "timestamp": datetime.now().isoformat()
458
+ }
459
+
460
+ except HTTPException:
461
+ raise
462
+ except Exception as e:
463
+ logger.error(f"Error in get_market_overview: {e}")
464
+ raise HTTPException(status_code=500, detail=str(e))
465
+
466
+
467
+ @app.get("/api/market/prices")
468
+ async def get_multiple_prices(symbols: str = Query("BTC,ETH,SOL", description="Comma-separated symbols")):
469
+ """Get prices for multiple cryptocurrencies"""
470
+ try:
471
+ symbol_list = [s.strip().upper() for s in symbols.split(",")]
472
+
473
+ # Fetch prices
474
+ prices_data = []
475
+ for symbol in symbol_list:
476
+ try:
477
+ ticker = await fetch_binance_ticker(f"{symbol}USDT")
478
+ if ticker:
479
+ prices_data.append(ticker)
480
+ except:
481
+ continue
482
+
483
+ if not prices_data:
484
+ # Fallback to CoinGecko
485
+ prices_data = await fetch_coingecko_prices(symbol_list)
486
+
487
+ return {
488
+ "symbols": symbol_list,
489
+ "count": len(prices_data),
490
+ "data": prices_data,
491
+ "timestamp": datetime.now().isoformat()
492
+ }
493
+
494
+ except Exception as e:
495
+ logger.error(f"Error in get_multiple_prices: {e}")
496
+ raise HTTPException(status_code=500, detail=str(e))
497
+
498
+
499
+ @app.get("/api/market-data/prices")
500
+ async def get_market_data_prices(symbols: str = Query("BTC,ETH", description="Comma-separated symbols")):
501
+ """Alternative endpoint for market data prices"""
502
+ return await get_multiple_prices(symbols)
503
+
504
+
505
+ # ============================================================================
506
+ # Analysis Endpoints
507
+ # ============================================================================
508
+
509
+ @app.get("/api/analysis/signals")
510
+ async def get_trading_signals(
511
+ symbol: str = Query("BTCUSDT", description="Trading pair"),
512
+ timeframe: str = Query("1h", description="Timeframe")
513
+ ):
514
+ """Get trading signals for a symbol"""
515
+ try:
516
+ # Fetch OHLCV data for analysis
517
+ ohlcv = await fetch_binance_ohlcv(symbol, timeframe, 100)
518
+
519
+ if not ohlcv:
520
+ raise HTTPException(status_code=503, detail="Unable to fetch data for analysis")
521
+
522
+ # Simple signal generation (can be enhanced)
523
+ latest = ohlcv[-1]
524
+ prev = ohlcv[-2] if len(ohlcv) > 1 else latest
525
+
526
+ # Calculate simple indicators
527
+ close_prices = [c["close"] for c in ohlcv[-20:]]
528
+ sma_20 = sum(close_prices) / len(close_prices)
529
+
530
+ # Generate signal
531
+ trend = "bullish" if latest["close"] > sma_20 else "bearish"
532
+ momentum = "strong" if abs(latest["close"] - prev["close"]) / prev["close"] > 0.01 else "weak"
533
+
534
  signal = "buy" if trend == "bullish" and momentum == "strong" else (
535
  "sell" if trend == "bearish" and momentum == "strong" else "hold"
536
  )
 
552
  "analysis": ai_summary,
553
  "timestamp": datetime.now().isoformat()
554
  }
555
+
556
+ except HTTPException:
557
+ raise
558
+ except Exception as e:
559
+ logger.error(f"Error in get_trading_signals: {e}")
560
+ raise HTTPException(status_code=500, detail=str(e))
561
+
562
+
563
+ @app.get("/api/analysis/smc")
564
+ async def get_smc_analysis(symbol: str = Query("BTCUSDT", description="Trading pair")):
565
+ """Get Smart Money Concepts (SMC) analysis"""
566
+ try:
567
+ # Fetch OHLCV data
568
+ ohlcv = await fetch_binance_ohlcv(symbol, "1h", 200)
569
+
570
+ if not ohlcv:
571
+ raise HTTPException(status_code=503, detail="Unable to fetch data")
572
+
573
+ # Calculate key levels
574
+ highs = [c["high"] for c in ohlcv]
575
+ lows = [c["low"] for c in ohlcv]
576
+ closes = [c["close"] for c in ohlcv]
577
+
578
+ resistance = max(highs[-50:])
579
+ support = min(lows[-50:])
580
+ current_price = closes[-1]
581
+
582
+ # Structure analysis
583
+ market_structure = "higher_highs" if closes[-1] > closes[-10] > closes[-20] else "lower_lows"
584
+
585
+ return {
586
+ "symbol": symbol,
587
+ "market_structure": market_structure,
588
+ "key_levels": {
589
+ "resistance": resistance,
590
+ "support": support,
591
+ "current_price": current_price,
592
+ "mid_point": (resistance + support) / 2
593
+ },
594
+ "order_blocks": {
595
+ "bullish": support,
596
+ "bearish": resistance
597
+ },
598
+ "liquidity_zones": {
599
+ "above": resistance,
600
+ "below": support
601
+ },
602
+ "timestamp": datetime.now().isoformat()
603
+ }
604
+
605
+ except HTTPException:
606
+ raise
607
+ except Exception as e:
608
+ logger.error(f"Error in get_smc_analysis: {e}")
609
+ raise HTTPException(status_code=500, detail=str(e))
610
+
611
+
612
+ @app.get("/api/scoring/snapshot")
613
+ async def get_scoring_snapshot(symbol: str = Query("BTCUSDT", description="Trading pair")):
614
+ """Get comprehensive scoring snapshot"""
615
+ try:
616
+ # Fetch data
617
+ ticker = await fetch_binance_ticker(symbol)
618
+ ohlcv = await fetch_binance_ohlcv(symbol, "1h", 100)
619
+
620
+ if not ticker or not ohlcv:
621
+ raise HTTPException(status_code=503, detail="Unable to fetch data")
622
+
623
+ # Calculate scores (0-100)
624
+ volatility_score = min(abs(ticker["price_change_percent_24h"]) * 5, 100)
625
+ volume_score = min((ticker["volume_24h"] / 1000000) * 10, 100)
626
+ trend_score = 50 + (ticker["price_change_percent_24h"] * 2)
627
+
628
+ # Overall score
629
+ overall_score = (volatility_score + volume_score + trend_score) / 3
630
+
631
+ return {
632
+ "symbol": symbol,
633
+ "overall_score": round(overall_score, 2),
634
+ "scores": {
635
+ "volatility": round(volatility_score, 2),
636
+ "volume": round(volume_score, 2),
637
+ "trend": round(trend_score, 2),
638
+ "momentum": round(50 + ticker["price_change_percent_24h"], 2)
639
+ },
640
+ "rating": "excellent" if overall_score > 80 else (
641
+ "good" if overall_score > 60 else (
642
+ "average" if overall_score > 40 else "poor"
643
+ )
644
+ ),
645
+ "timestamp": datetime.now().isoformat()
646
+ }
647
+
648
+ except HTTPException:
649
+ raise
650
+ except Exception as e:
651
+ logger.error(f"Error in get_scoring_snapshot: {e}")
652
+ raise HTTPException(status_code=500, detail=str(e))
653
+
654
+
655
+ @app.get("/api/signals")
656
+ async def get_all_signals():
657
+ """Get signals for multiple assets"""
658
+ symbols = ["BTCUSDT", "ETHUSDT", "SOLUSDT", "BNBUSDT"]
659
+ signals = []
660
+
661
+ for symbol in symbols:
662
+ try:
663
+ signal_data = await get_trading_signals(symbol, "1h")
664
+ signals.append(signal_data)
665
+ except:
666
+ continue
667
+
668
+ return {
669
+ "count": len(signals),
670
+ "signals": signals,
671
+ "timestamp": datetime.now().isoformat()
672
+ }
673
+
674
+
675
  @app.get("/api/sentiment")
676
  async def get_sentiment():
677
  """Get market sentiment data"""
 
704
  "analysis": analysis,
705
  "timestamp": datetime.utcnow().isoformat(),
706
  }
707
+
708
+
709
+ # ============================================================================
710
+ # System Endpoints
711
+ # ============================================================================
712
+
713
  @app.get("/api/system/status")
714
  async def get_system_status():
715
  """Get system status"""
 
730
  "requests_per_minute": 0,
731
  "timestamp": datetime.utcnow().isoformat(),
732
  }
733
+
734
+
735
  @app.get("/api/system/config")
736
  async def get_system_config():
737
  """Get system configuration"""
 
744
  "max_ohlcv_limit": 1000,
745
  "timestamp": datetime.utcnow().isoformat(),
746
  }
747
+
748
+
749
+ @app.get("/api/categories")
750
+ async def get_categories():
751
+ """Get data categories"""
752
+ return {
753
+ "categories": [
754
+ {"name": "market_data", "endpoints": 5, "status": "active"},
755
+ {"name": "analysis", "endpoints": 4, "status": "active"},
756
+ {"name": "signals", "endpoints": 2, "status": "active"},
757
+ {"name": "sentiment", "endpoints": 1, "status": "active"}
758
+ ]
759
+ }
760
+
761
+
762
+ @app.get("/api/rate-limits")
763
+ async def get_rate_limits():
764
+ """Get rate limit information"""
765
+ return {
766
+ "rate_limits": [
767
+ {"endpoint": "/api/ohlcv", "limit": 1200, "window": "per_minute"},
768
+ {"endpoint": "/api/crypto/prices/top", "limit": 600, "window": "per_minute"},
769
+ {"endpoint": "/api/analysis/*", "limit": 300, "window": "per_minute"}
770
+ ],
771
+ "current_usage": {
772
+ "requests_this_minute": 0,
773
+ "percentage": 0
774
+ }
775
+ }
776
+
777
+
778
+ @app.get("/api/logs")
779
+ async def get_logs(limit: int = Query(50, ge=1, le=500)):
780
+ """Get recent API logs"""
781
+ # Mock logs (can be enhanced with real logging)
782
+ logs = []
783
+ for i in range(min(limit, 10)):
784
+ logs.append({
785
+ "timestamp": (datetime.now() - timedelta(minutes=i)).isoformat(),
786
+ "endpoint": "/api/ohlcv",
787
+ "status": "success",
788
+ "response_time_ms": random.randint(50, 200)
789
+ })
790
+
791
+ return {"logs": logs, "count": len(logs)}
792
+
793
+
794
+ @app.get("/api/alerts")
795
+ async def get_alerts():
796
+ """Get system alerts"""
797
+ return {
798
+ "alerts": [],
799
+ "count": 0,
800
+ "timestamp": datetime.now().isoformat()
801
+ }
802
+
803
+
804
+ # ============================================================================
805
+ # HuggingFace Integration Endpoints
806
+ # ============================================================================
807
+
808
  @app.get("/api/hf/health")
809
  async def hf_health():
810
  """HuggingFace integration health"""
811
  status = registry_status()
812
  status["timestamp"] = datetime.utcnow().isoformat()
813
  return status
814
+
815
+
816
  @app.post("/api/hf/refresh")
817
  async def hf_refresh():
818
  """Refresh HuggingFace data"""
819
  result = initialize_models()
820
  return {"status": "ok" if result.get("success") else "degraded", **result, "timestamp": datetime.utcnow().isoformat()}
821
+
822
+
823
  @app.get("/api/hf/registry")
824
  async def hf_registry(kind: str = "models"):
825
  """Get HuggingFace registry"""
826
  info = get_model_info()
827
  return {"kind": kind, "items": info.get("model_names", info)}
828
+
829
+
830
  def _resolve_sentiment_payload(payload: Union[List[str], Dict[str, Any]]) -> Dict[str, Any]:
831
  if isinstance(payload, list):
832
  return {"texts": payload, "mode": "auto"}
 
866
  results.append({"text": text, "result": analysis})
867
 
868
  return {"mode": mode, "results": results, "timestamp": datetime.utcnow().isoformat()}
869
+
870
+
871
+ # ============================================================================
872
+ # HTML Routes - Serve UI files
873
+ # ============================================================================
874
+
875
+ @app.get("/", response_class=HTMLResponse)
876
+ async def root():
877
+ """Serve main admin dashboard (admin.html)"""
878
+ admin_path = WORKSPACE_ROOT / "admin.html"
879
+ if admin_path.exists():
880
+ return FileResponse(admin_path)
881
+ return HTMLResponse("<h1>Cryptocurrency Data & Analysis API</h1><p>See <a href='/docs'>/docs</a> for API documentation</p>")
882
+
883
+ @app.get("/index.html", response_class=HTMLResponse)
884
+ async def index():
885
+ """Serve index.html"""
886
+ return FileResponse(WORKSPACE_ROOT / "index.html")
887
+
888
+ @app.get("/dashboard.html", response_class=HTMLResponse)
889
+ async def dashboard():
890
+ """Serve dashboard.html"""
891
+ return FileResponse(WORKSPACE_ROOT / "dashboard.html")
892
+
893
+ @app.get("/dashboard", response_class=HTMLResponse)
894
+ async def dashboard_alt():
895
+ """Alternative route for dashboard"""
896
+ return FileResponse(WORKSPACE_ROOT / "dashboard.html")
897
+
898
+ @app.get("/admin.html", response_class=HTMLResponse)
899
+ async def admin():
900
+ """Serve admin panel"""
901
+ return FileResponse(WORKSPACE_ROOT / "admin.html")
902
+
903
+ @app.get("/admin", response_class=HTMLResponse)
904
+ async def admin_alt():
905
+ """Alternative route for admin"""
906
+ return FileResponse(WORKSPACE_ROOT / "admin.html")
907
+
908
+ @app.get("/hf_console.html", response_class=HTMLResponse)
909
+ async def hf_console():
910
+ """Serve HuggingFace console"""
911
+ return FileResponse(WORKSPACE_ROOT / "hf_console.html")
912
+
913
+ @app.get("/console", response_class=HTMLResponse)
914
+ async def console_alt():
915
+ """Alternative route for HF console"""
916
+ return FileResponse(WORKSPACE_ROOT / "hf_console.html")
917
+
918
+ @app.get("/pool_management.html", response_class=HTMLResponse)
919
+ async def pool_management():
920
+ """Serve pool management UI"""
921
+ return FileResponse(WORKSPACE_ROOT / "pool_management.html")
922
+
923
+ @app.get("/unified_dashboard.html", response_class=HTMLResponse)
924
+ async def unified_dashboard():
925
+ """Serve unified dashboard"""
926
+ return FileResponse(WORKSPACE_ROOT / "unified_dashboard.html")
927
+
928
+ @app.get("/simple_overview.html", response_class=HTMLResponse)
929
+ async def simple_overview():
930
+ """Serve simple overview"""
931
+ return FileResponse(WORKSPACE_ROOT / "simple_overview.html")
932
+
933
+ # Generic HTML file handler
934
+ @app.get("/{filename}.html", response_class=HTMLResponse)
935
+ async def serve_html(filename: str):
936
+ """Serve any HTML file from workspace root"""
937
+ file_path = WORKSPACE_ROOT / f"{filename}.html"
938
+ if file_path.exists():
939
+ return FileResponse(file_path)
940
+ return HTMLResponse(f"<h1>File {filename}.html not found</h1>", status_code=404)
941
+
942
+
943
+ # ============================================================================
944
+ # Startup Event
945
+ # ============================================================================
946
+
947
+ @app.on_event("startup")
948
+ async def startup_event():
949
+ """Initialize on startup"""
950
+ # Initialize AI models
951
+ from ai_models import initialize_models
952
+ models_init = initialize_models()
953
+ logger.info(f"✓ AI Models initialized: {models_init}")
954
+
955
+ # Initialize HF Registry
956
+ from backend.services.hf_registry import REGISTRY
957
+ registry_result = await REGISTRY.refresh()
958
+ logger.info(f"✓ HF Registry initialized: {registry_result}")
959
+
960
+ logger.info("=" * 70)
961
+ logger.info("🚀 Cryptocurrency Data & Analysis API Starting")
962
+ logger.info("=" * 70)
963
+ logger.info("✓ FastAPI initialized")
964
+ logger.info("✓ CORS configured")
965
+ logger.info("✓ Cache initialized")
966
+ logger.info(f"✓ Providers loaded: {len(PROVIDERS_CONFIG)}")
967
+
968
+ # Show loaded HuggingFace Space providers
969
+ hf_providers = [p for p in PROVIDERS_CONFIG.keys() if 'huggingface_space' in p]
970
+ if hf_providers:
971
+ logger.info(f"✓ HuggingFace Space providers: {', '.join(hf_providers)}")
972
+
973
+ logger.info("✓ Data sources: Binance, CoinGecko, providers_config_extended.json")
974
+
975
+ # Check HTML files
976
+ html_files = ["index.html", "dashboard.html", "admin.html", "hf_console.html"]
977
+ available_html = [f for f in html_files if (WORKSPACE_ROOT / f).exists()]
978
+ logger.info(f"✓ UI files: {len(available_html)}/{len(html_files)} available")
979
+
980
+ logger.info("=" * 70)
981
+ logger.info("📡 API ready at http://0.0.0.0:7860")
982
+ logger.info("📖 Docs at http://0.0.0.0:7860/docs")
983
+ logger.info("🎨 UI at http://0.0.0.0:7860/ (admin.html)")
984
+ logger.info("=" * 70)
985
+
986
+
987
+ # ============================================================================
988
+ # Main Entry Point
989
+ # ============================================================================
990
+
991
+ if __name__ == "__main__":
992
+ import uvicorn
993
+
994
+ print("=" * 70)
995
+ print("🚀 Starting Cryptocurrency Data & Analysis API")
996
+ print("=" * 70)
997
+ print("📍 Server: http://localhost:7860")
998
+ print("📖 API Docs: http://localhost:7860/docs")
999
+ print("🔗 Health: http://localhost:7860/health")
1000
+ print("=" * 70)
1001
+
1002
+ uvicorn.run(
1003
+ app,
1004
+ host="0.0.0.0",
1005
+ port=7860,
1006
+ log_level="info"
1007
+ )
1008
+ # NEW ENDPOINTS FOR ADMIN.HTML - ADD TO hf_unified_server.py
1009
+
1010
+ from fastapi import WebSocket, WebSocketDisconnect
1011
+ from collections import defaultdict
1012
+
1013
+ # WebSocket Manager
1014
+ class ConnectionManager:
1015
+ def __init__(self):
1016
+ self.active_connections: List[WebSocket] = []
1017
+
1018
+ async def connect(self, websocket: WebSocket):
1019
+ await websocket.accept()
1020
+ self.active_connections.append(websocket)
1021
+ logger.info(f"WebSocket connected. Total: {len(self.active_connections)}")
1022
+
1023
+ def disconnect(self, websocket: WebSocket):
1024
+ if websocket in self.active_connections:
1025
+ self.active_connections.remove(websocket)
1026
+ logger.info(f"WebSocket disconnected. Total: {len(self.active_connections)}")
1027
+
1028
+ async def broadcast(self, message: dict):
1029
+ for connection in list(self.active_connections):
1030
+ try:
1031
+ await connection.send_json(message)
1032
+ except:
1033
+ self.disconnect(connection)
1034
+
1035
+ ws_manager = ConnectionManager()
1036
+
1037
+
1038
+ # ===== API HEALTH =====
1039
+ @app.get("/api/health")
1040
+ async def api_health():
1041
+ """Health check for admin dashboard"""
1042
+ health_data = await health()
1043
+ return {
1044
+ "status": "healthy" if health_data.get("status") == "ok" else "degraded",
1045
+ **health_data
1046
+ }
1047
+
1048
+
1049
+ # ===== COINS ENDPOINTS =====
1050
+ @app.get("/api/coins/top")
1051
+ async def get_top_coins(limit: int = Query(default=10, ge=1, le=100)):
1052
+ """Get top cryptocurrencies by market cap"""
1053
+ try:
1054
+ coins = await market_collector.get_top_coins(limit=limit)
1055
+
1056
+ result = []
1057
+ for coin in coins:
1058
+ result.append({
1059
+ "rank": coin.get("rank", 0),
1060
+ "symbol": coin.get("symbol", "").upper(),
1061
+ "name": coin.get("name", ""),
1062
+ "price": coin.get("price") or coin.get("current_price", 0),
1063
+ "price_change_24h": coin.get("change_24h") or coin.get("price_change_percentage_24h", 0),
1064
+ "volume_24h": coin.get("volume_24h") or coin.get("total_volume", 0),
1065
+ "market_cap": coin.get("market_cap", 0),
1066
+ "image": coin.get("image", ""),
1067
+ "last_updated": coin.get("last_updated", datetime.now().isoformat())
1068
+ })
1069
+
1070
+ return {
1071
+ "success": True,
1072
+ "coins": result,
1073
+ "count": len(result),
1074
+ "timestamp": datetime.now().isoformat()
1075
+ }
1076
+ except Exception as e:
1077
+ logger.error(f"Error in /api/coins/top: {e}")
1078
+ raise HTTPException(status_code=503, detail=str(e))
1079
+
1080
+
1081
+ @app.get("/api/coins/{symbol}")
1082
+ async def get_coin_detail(symbol: str):
1083
+ """Get specific coin details"""
1084
+ try:
1085
+ coins = await market_collector.get_top_coins(limit=250)
1086
+ coin = next((c for c in coins if c.get("symbol", "").upper() == symbol.upper()), None)
1087
+
1088
+ if not coin:
1089
+ raise HTTPException(status_code=404, detail=f"Coin {symbol} not found")
1090
+
1091
+ return {
1092
+ "success": True,
1093
+ "symbol": symbol.upper(),
1094
+ "name": coin.get("name", ""),
1095
+ "price": coin.get("price") or coin.get("current_price", 0),
1096
+ "change_24h": coin.get("change_24h") or coin.get("price_change_percentage_24h", 0),
1097
+ "volume_24h": coin.get("volume_24h") or coin.get("total_volume", 0),
1098
+ "market_cap": coin.get("market_cap", 0),
1099
+ "rank": coin.get("rank", 0),
1100
+ "last_updated": coin.get("last_updated", datetime.now().isoformat())
1101
+ }
1102
+ except HTTPException:
1103
+ raise
1104
+ except Exception as e:
1105
+ logger.error(f"Error in /api/coins/{symbol}: {e}")
1106
+ raise HTTPException(status_code=503, detail=str(e))
1107
+
1108
+
1109
+ # ===== MARKET STATS =====
1110
+ @app.get("/api/market/stats")
1111
+ async def get_market_stats():
1112
+ """Get global market statistics"""
1113
+ try:
1114
+ # Use existing endpoint
1115
+ overview = await get_market_overview()
1116
+
1117
+ stats = {
1118
+ "total_market_cap": overview.get("global_market_cap", 0),
1119
+ "total_volume_24h": overview.get("global_volume", 0),
1120
+ "btc_dominance": overview.get("btc_dominance", 0),
1121
+ "eth_dominance": overview.get("eth_dominance", 0),
1122
+ "active_cryptocurrencies": 10000, # Approximate
1123
+ "markets": 500, # Approximate
1124
+ "market_cap_change_24h": 0.0,
1125
+ "timestamp": datetime.now().isoformat()
1126
+ }
1127
+
1128
+ return {"success": True, "stats": stats}
1129
+ except Exception as e:
1130
+ logger.error(f"Error in /api/market/stats: {e}")
1131
+ raise HTTPException(status_code=503, detail=str(e))
1132
+
1133
+
1134
+ # ===== NEWS ENDPOINTS =====
1135
+ @app.get("/api/news/latest")
1136
+ async def get_latest_news(limit: int = Query(default=40, ge=1, le=100)):
1137
+ """Get latest crypto news with sentiment"""
1138
+ try:
1139
+ news_items = await news_collector.get_latest_news(limit=limit)
1140
+
1141
+ # Attach sentiment to each news item
1142
+ from ai_models import analyze_news_item
1143
+ enriched_news = []
1144
+ for item in news_items:
1145
+ try:
1146
+ enriched = analyze_news_item(item)
1147
+ enriched_news.append({
1148
+ "title": enriched.get("title", ""),
1149
+ "source": enriched.get("source", ""),
1150
+ "published_at": enriched.get("published_at") or enriched.get("date", ""),
1151
+ "symbols": enriched.get("symbols", []),
1152
+ "sentiment": enriched.get("sentiment", "neutral"),
1153
+ "sentiment_confidence": enriched.get("sentiment_confidence", 0.5),
1154
+ "url": enriched.get("url", "")
1155
+ })
1156
+ except:
1157
+ enriched_news.append({
1158
+ "title": item.get("title", ""),
1159
+ "source": item.get("source", ""),
1160
+ "published_at": item.get("published_at") or item.get("date", ""),
1161
+ "symbols": item.get("symbols", []),
1162
+ "sentiment": "neutral",
1163
+ "sentiment_confidence": 0.5,
1164
+ "url": item.get("url", "")
1165
+ })
1166
+
1167
+ return {
1168
+ "success": True,
1169
+ "news": enriched_news,
1170
+ "count": len(enriched_news),
1171
+ "timestamp": datetime.now().isoformat()
1172
+ }
1173
+ except Exception as e:
1174
+ logger.error(f"Error in /api/news/latest: {e}")
1175
+ return {"success": True, "news": [], "count": 0, "timestamp": datetime.now().isoformat()}
1176
+
1177
+
1178
+ @app.post("/api/news/summarize")
1179
+ async def summarize_news(item: Dict[str, Any] = Body(...)):
1180
+ """Summarize a news article"""
1181
+ try:
1182
+ from ai_models import analyze_news_item
1183
+ enriched = analyze_news_item(item)
1184
+
1185
+ return {
1186
+ "success": True,
1187
+ "summary": enriched.get("title", ""),
1188
+ "sentiment": enriched.get("sentiment", "neutral"),
1189
+ "sentiment_confidence": enriched.get("sentiment_confidence", 0.5)
1190
+ }
1191
+ except Exception as e:
1192
+ logger.error(f"Error in /api/news/summarize: {e}")
1193
+ return {
1194
+ "success": False,
1195
+ "error": str(e),
1196
+ "summary": item.get("title", ""),
1197
+ "sentiment": "neutral"
1198
+ }
1199
+
1200
+
1201
+ # ===== CHARTS ENDPOINTS =====
1202
+ @app.get("/api/charts/price/{symbol}")
1203
+ async def get_price_chart(symbol: str, timeframe: str = Query(default="7d")):
1204
+ """Get price chart data"""
1205
+ try:
1206
+ # Map timeframe to hours
1207
+ timeframe_map = {"1d": 24, "7d": 168, "30d": 720, "90d": 2160, "1y": 8760}
1208
+ hours = timeframe_map.get(timeframe, 168)
1209
+
1210
+ price_history = await market_collector.get_price_history(symbol, hours=hours)
1211
+
1212
+ chart_data = []
1213
+ for point in price_history:
1214
+ chart_data.append({
1215
+ "timestamp": point.get("timestamp", ""),
1216
+ "price": point.get("price", 0),
1217
+ "date": point.get("timestamp", "")
1218
+ })
1219
+
1220
+ return {
1221
+ "success": True,
1222
+ "symbol": symbol.upper(),
1223
+ "timeframe": timeframe,
1224
+ "data": chart_data,
1225
+ "count": len(chart_data)
1226
+ }
1227
+ except Exception as e:
1228
+ logger.error(f"Error in /api/charts/price/{symbol}: {e}")
1229
+ raise HTTPException(status_code=503, detail=str(e))
1230
+
1231
+
1232
+ @app.post("/api/charts/analyze")
1233
+ async def analyze_chart(payload: Dict[str, Any] = Body(...)):
1234
+ """Analyze chart data"""
1235
+ try:
1236
+ symbol = payload.get("symbol")
1237
+ timeframe = payload.get("timeframe", "7d")
1238
+ indicators = payload.get("indicators", [])
1239
+
1240
+ # Get price data
1241
+ price_history = await market_collector.get_price_history(symbol, hours=168)
1242
+
1243
+ # Analyze with AI
1244
+ from ai_models import analyze_chart_points
1245
+ analysis = analyze_chart_points(price_history, indicators)
1246
+
1247
+ return {
1248
+ "success": True,
1249
+ "symbol": symbol,
1250
+ "timeframe": timeframe,
1251
+ "analysis": analysis
1252
+ }
1253
+ except Exception as e:
1254
+ logger.error(f"Error in /api/charts/analyze: {e}")
1255
+ return {"success": False, "error": str(e)}
1256
+
1257
+
1258
+ # ===== SENTIMENT ENDPOINTS =====
1259
+ @app.post("/api/sentiment/analyze")
1260
+ async def analyze_sentiment(payload: Dict[str, Any] = Body(...)):
1261
+ """Analyze sentiment of text"""
1262
+ try:
1263
+ text = payload.get("text", "")
1264
+
1265
+ from ai_models import ensemble_crypto_sentiment
1266
+ result = ensemble_crypto_sentiment(text)
1267
+
1268
+ return {
1269
+ "success": True,
1270
+ "sentiment": result["label"],
1271
+ "confidence": result["confidence"],
1272
+ "details": result
1273
+ }
1274
+ except Exception as e:
1275
+ logger.error(f"Error in /api/sentiment/analyze: {e}")
1276
+ return {"success": False, "error": str(e)}
1277
+
1278
+
1279
+ # ===== QUERY ENDPOINT =====
1280
+ @app.post("/api/query")
1281
+ async def process_query(payload: Dict[str, Any] = Body(...)):
1282
+ """Process natural language query"""
1283
+ try:
1284
+ query = payload.get("query", "").lower()
1285
+
1286
+ # Simple query processing
1287
+ if "price" in query or "btc" in query or "bitcoin" in query:
1288
+ coins = await market_collector.get_top_coins(limit=10)
1289
+ btc = next((c for c in coins if c.get("symbol", "").upper() == "BTC"), None)
1290
+
1291
+ if btc:
1292
+ price = btc.get("price") or btc.get("current_price", 0)
1293
+ return {
1294
+ "success": True,
1295
+ "type": "price",
1296
+ "message": f"Bitcoin (BTC) is currently trading at ${price:,.2f}",
1297
+ "data": btc
1298
+ }
1299
+
1300
+ return {
1301
+ "success": True,
1302
+ "type": "general",
1303
+ "message": "Query processed",
1304
+ "data": None
1305
+ }
1306
+ except Exception as e:
1307
+ logger.error(f"Error in /api/query: {e}")
1308
+ return {"success": False, "error": str(e), "message": "Query failed"}
1309
+
1310
+
1311
+ # ===== DATASETS & MODELS =====
1312
+ @app.get("/api/datasets/list")
1313
+ async def list_datasets():
1314
+ """List available datasets"""
1315
+ try:
1316
+ from backend.services.hf_registry import REGISTRY
1317
+ datasets = REGISTRY.list(kind="datasets")
1318
+
1319
+ formatted = []
1320
+ for d in datasets:
1321
+ formatted.append({
1322
+ "name": d.get("id"),
1323
+ "category": d.get("category", "other"),
1324
+ "records": "N/A",
1325
+ "updated_at": "",
1326
+ "tags": d.get("tags", []),
1327
+ "source": d.get("source", "hub")
1328
+ })
1329
+
1330
+ return {
1331
+ "success": True,
1332
+ "datasets": formatted,
1333
+ "count": len(formatted)
1334
+ }
1335
+ except Exception as e:
1336
+ logger.error(f"Error in /api/datasets/list: {e}")
1337
+ return {"success": True, "datasets": [], "count": 0}
1338
+
1339
+
1340
+ @app.get("/api/datasets/sample")
1341
+ async def get_dataset_sample(name: str = Query(...), limit: int = Query(default=20)):
1342
+ """Get sample from dataset"""
1343
+ try:
1344
+ # Attempt to load dataset
1345
+ try:
1346
+ from datasets import load_dataset
1347
+ dataset = load_dataset(name, split="train", streaming=True)
1348
+
1349
+ sample = []
1350
+ for i, row in enumerate(dataset):
1351
+ if i >= limit:
1352
+ break
1353
+ sample.append({k: str(v) for k, v in row.items()})
1354
+
1355
+ return {
1356
+ "success": True,
1357
+ "name": name,
1358
+ "sample": sample,
1359
+ "count": len(sample)
1360
+ }
1361
+ except:
1362
+ return {
1363
+ "success": False,
1364
+ "name": name,
1365
+ "sample": [],
1366
+ "count": 0,
1367
+ "message": "Dataset loading requires authentication or is not available"
1368
+ }
1369
+ except Exception as e:
1370
+ logger.error(f"Error in /api/datasets/sample: {e}")
1371
+ return {"success": False, "error": str(e)}
1372
+
1373
+
1374
+ @app.get("/api/models/list")
1375
+ async def list_models():
1376
+ """List available models"""
1377
+ try:
1378
+ from ai_models import get_model_info
1379
+ info = get_model_info()
1380
+
1381
+ models = []
1382
+ catalog = info.get("model_catalog", {})
1383
+
1384
+ for category, model_list in catalog.items():
1385
+ for model_id in model_list:
1386
+ models.append({
1387
+ "name": model_id,
1388
+ "task": "sentiment" if "sentiment" in category else "decision" if category == "decision" else "analysis",
1389
+ "status": "available",
1390
+ "category": category,
1391
+ "notes": f"{category.replace('_', ' ').title()} model"
1392
+ })
1393
+
1394
+ return {
1395
+ "success": True,
1396
+ "models": models,
1397
+ "count": len(models)
1398
+ }
1399
+ except Exception as e:
1400
+ logger.error(f"Error in /api/models/list: {e}")
1401
+ return {"success": True, "models": [], "count": 0}
1402
+
1403
+
1404
+ @app.post("/api/models/test")
1405
+ async def test_model(payload: Dict[str, Any] = Body(...)):
1406
+ """Test a specific model"""
1407
+ try:
1408
+ model_id = payload.get("model", "")
1409
+ text = payload.get("text", "")
1410
+
1411
+ from ai_models import ensemble_crypto_sentiment
1412
+ result = ensemble_crypto_sentiment(text)
1413
+
1414
+ return {
1415
+ "success": True,
1416
+ "model": model_id,
1417
+ "result": result
1418
+ }
1419
+ except Exception as e:
1420
+ logger.error(f"Error in /api/models/test: {e}")
1421
+ return {"success": False, "error": str(e)}
1422
+
1423
+
1424
+ # ===== WEBSOCKET =====
1425
+ @app.websocket("/ws")
1426
+ async def websocket_endpoint(websocket: WebSocket):
1427
+ """WebSocket endpoint for real-time updates"""
1428
+ await ws_manager.connect(websocket)
1429
+
1430
+ try:
1431
+ while True:
1432
+ # Send market updates every 10 seconds
1433
+ try:
1434
+ # Get latest data
1435
+ top_coins = await market_collector.get_top_coins(limit=5)
1436
+ news_items = await news_collector.get_latest_news(limit=3)
1437
+
1438
+ # Compute global sentiment from news
1439
+ from ai_models import ensemble_crypto_sentiment
1440
+ news_texts = " ".join([n.get("title", "") for n in news_items])
1441
+ global_sentiment = ensemble_crypto_sentiment(news_texts) if news_texts else {"label": "neutral", "confidence": 0.5}
1442
+
1443
+ payload = {
1444
+ "market_data": top_coins,
1445
+ "stats": {
1446
+ "total_market_cap": sum([c.get("market_cap", 0) for c in top_coins]),
1447
+ "sentiment": global_sentiment
1448
+ },
1449
+ "news": news_items,
1450
+ "sentiment": global_sentiment,
1451
+ "timestamp": datetime.now().isoformat()
1452
+ }
1453
+
1454
+ await websocket.send_json({
1455
+ "type": "update",
1456
+ "payload": payload
1457
+ })
1458
+ except Exception as e:
1459
+ logger.error(f"Error in WebSocket update: {e}")
1460
+
1461
+ await asyncio.sleep(10)
1462
+ except WebSocketDisconnect:
1463
+ ws_manager.disconnect(websocket)
1464
+ except Exception as e:
1465
+ logger.error(f"WebSocket error: {e}")
1466
+ ws_manager.disconnect(websocket)
requirements.txt CHANGED
@@ -7,8 +7,8 @@
7
  # - Optional Transformers-based AI features
8
 
9
  # Core API Server Requirements
10
- fastapi
11
- uvicorn
12
  pydantic==2.5.3
13
  sqlalchemy==2.0.25
14
  httpx==0.26.0
@@ -22,8 +22,8 @@ aiohttp>=3.8.0
22
  pandas>=2.1.0
23
 
24
  # Gradio Dashboard & UI (Required for app.py)
25
- gradio
26
- plotly
27
  psutil==5.9.6
28
 
29
  # AI/ML Libraries (optional but recommended for AI features)
@@ -35,7 +35,8 @@ sentencepiece>=0.1.99
35
  feedparser>=6.0.10
36
 
37
  # HTML Parsing (Optional)
38
- beautifulsoup4
39
 
40
  # HuggingFace Hub (For model validation)
41
  huggingface-hub>=0.19.0
 
 
7
  # - Optional Transformers-based AI features
8
 
9
  # Core API Server Requirements
10
+ fastapi==0.109.0
11
+ uvicorn[standard]==0.27.0
12
  pydantic==2.5.3
13
  sqlalchemy==2.0.25
14
  httpx==0.26.0
 
22
  pandas>=2.1.0
23
 
24
  # Gradio Dashboard & UI (Required for app.py)
25
+ gradio==4.12.0
26
+ plotly==5.18.0
27
  psutil==5.9.6
28
 
29
  # AI/ML Libraries (optional but recommended for AI features)
 
35
  feedparser>=6.0.10
36
 
37
  # HTML Parsing (Optional)
38
+ beautifulsoup4>=4.12.0
39
 
40
  # HuggingFace Hub (For model validation)
41
  huggingface-hub>=0.19.0
42
+ datasets>=2.16.0