README.md

---
title: AGI Multi-Model API
emoji: 😻
colorFrom: purple
colorTo: green
sdk: docker
pinned: false
license: apache-2.0
short_description: Multi-model LLM API with caching and web search
---

# AGI Multi-Model API

A high-performance FastAPI server for running multiple LLM models with intelligent in-memory caching and web search capabilities. Switch between models instantly and augment responses with real-time web data.

## ✨ Features

- **🔄 Dynamic Model Switching**: Hot-swap between 5 different LLM models
- **⚡ Intelligent Caching**: LRU cache keeps up to 2 models in memory for instant switching
- **🌐 Web-Augmented Chat**: Real-time web search integration via DuckDuckGo
- **📚 OpenAI-Compatible API**: Drop-in replacement for OpenAI chat completions
- **📖 Auto-Generated Documentation**: Interactive API docs with Swagger UI and ReDoc
- **🚀 Optimized Performance**: Continuous batching and multi-port architecture

## 🤖 Available Models

| Model | Use Case | Size |
|-------|----------|------|
| **deepseek-chat** (default) | General purpose conversation | 7B |
| **mistral-7b** | Financial analysis & summarization | 7B |
| **openhermes-7b** | Advanced instruction following | 7B |
| **deepseek-coder** | Specialized coding assistance | 6.7B |
| **llama-7b** | Lightweight & fast responses | 7B |

## 🚀 Quick Start

### Prerequisites

- Python 3.10+
- `llama-server` (llama.cpp)
- 8GB+ RAM (16GB+ recommended for caching multiple models)

### Installation

```bash
# Clone the repository
git clone <your-repo-url>
cd AGI

# Install dependencies
pip install -r requirements.txt
# or
uv pip install -r pyproject.toml

# Start the server
uvicorn app:app --host 0.0.0.0 --port 8000
```

### Docker Deployment

```bash
# Build the image
docker build -t agi-api .

# Run the container
docker run -p 8000:8000 agi-api
```

## 📖 API Documentation

Once the server is running, access the interactive documentation:

- **Swagger UI**: http://localhost:8000/docs
- **ReDoc**: http://localhost:8000/redoc
- **OpenAPI JSON**: http://localhost:8000/openapi.json

## 🔧 Usage Examples

### Basic Chat Completion

```python
import requests

response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    json={
        "messages": [
            {"role": "user", "content": "What is the capital of France?"}
        ],
        "max_tokens": 100,
        "temperature": 0.7
    }
)
print(response.json())
```

### Web-Augmented Chat

```python
response = requests.post(
    "http://localhost:8000/v1/web-chat/completions",
    json={
        "messages": [
            {"role": "user", "content": "What are the latest AI developments?"}
        ],
        "max_tokens": 512,
        "max_search_results": 5
    }
)
result = response.json()
print(result["choices"][0]["message"]["content"])
print(f"Sources: {result['web_search']['sources']}")
```

### Switch Models

```python
# Switch to coding model
response = requests.post(
    "http://localhost:8000/switch-model",
    json={"model_name": "deepseek-coder"}
)
print(response.json())
# Output: {"message": "Switched to model: deepseek-coder (from cache)", "model": "deepseek-coder"}
```

### Check Cache Status

```python
response = requests.get("http://localhost:8000/cache/info")
cache_info = response.json()
print(f"Cached models: {cache_info['current_size']}/{cache_info['max_size']}")
for model in cache_info['cached_models']:
    print(f"  - {model['name']} on port {model['port']}")
```

## 🏗️ Architecture

### Model Caching System

The API uses an intelligent LRU (Least Recently Used) cache to manage models in memory:

```
┌─────────────────────────────────────────────────┐
│  Request: Switch to Model A                    │
├─────────────────────────────────────────────────┤
│  1. Check if A is current → Skip               │
│  2. Check cache for A                          │
│     ├─ Cache Hit  → Instant switch (< 1s)      │
│     └─ Cache Miss → Load model (~2-3 min)      │
│  3. If cache full → Evict LRU model            │
│  4. Add A to cache                             │
└─────────────────────────────────────────────────┘
```

**Benefits:**
- First load: ~2-3 minutes (model download + initialization)
- Subsequent switches: < 1 second (from cache)
- Automatic memory management with LRU eviction
- Each model runs on a separate port (8080, 8081, etc.)

### Multi-Port Architecture

```
┌──────────────────────┐
│   FastAPI Server     │
│   (Port 8000)        │
└──────────┬───────────┘
           │
    ┌──────┴────────────────────┐
    │                           │
┌───▼────────────┐  ┌──────────▼────┐
│ llama-server   │  │ llama-server  │
│ Model A:8080   │  │ Model B:8081  │
└────────────────┘  └───────────────┘
```

## ⚙️ Configuration

### Environment Variables

```bash
# Maximum cached models (default: 2)
MAX_CACHED_MODELS=2

# Base port for llama-server instances (default: 8080)
BASE_PORT=8080

# Default model on startup
DEFAULT_MODEL=deepseek-chat
```

### Model Configuration

Edit `AVAILABLE_MODELS` in `app.py` to add custom models:

```python
AVAILABLE_MODELS = {
    "my-model": "username/model-name-GGUF:model-file.Q4_K_M.gguf"
}
```

## 📊 API Endpoints

### Status & Models

- `GET /` - API status and current model
- `GET /models` - List available models
- `GET /cache/info` - Cache statistics and cached models

### Model Management

- `POST /switch-model` - Switch active model (with caching)

### Chat Completions

- `POST /v1/chat/completions` - Standard chat completions (OpenAI-compatible)
- `POST /v1/web-chat/completions` - Web-augmented chat with search

### Documentation

- `GET /docs` - Swagger UI interactive documentation
- `GET /redoc` - ReDoc alternative documentation
- `GET /openapi.json` - OpenAPI 3.0 specification export

## 🧪 Testing

```bash
# Test basic chat
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 50
  }'

# Check cache status
curl http://localhost:8000/cache/info

# Switch models
curl -X POST http://localhost:8000/switch-model \
  -H "Content-Type: application/json" \
  -d '{"model_name": "deepseek-coder"}'
```

## 🔍 Web Search Integration

The web-augmented chat endpoint automatically:
1. Extracts the user's query from the last message
2. Performs a DuckDuckGo web search
3. Injects search results into the LLM context
4. Returns response with source citations

**Use cases:**
- Current events and news
- Recent developments beyond training data
- Fact-checking with live web data
- Research with source attribution

## 📈 Performance Tips

1. **Cache Size**: Increase `MAX_CACHED_MODELS` if you have sufficient RAM (each model ~4-5GB)
2. **CPU Threads**: Adjust `-t` parameter in `start_llama_server()` based on your CPU cores
3. **Batch Size**: Modify `-b` parameter for throughput vs. latency tradeoff
4. **GPU Acceleration**: Set `-ngl` > 0 if you have a GPU (requires llama.cpp with GPU support)

## 🛠️ Development

### Project Structure

```
AGI/
├── app.py                    # Main FastAPI application
├── client_multi_model.py     # Example client
├── Dockerfile                # Docker configuration
├── pyproject.toml           # Python dependencies
└── README.md                # This file
```

### Adding New Models

1. Find a GGUF model on HuggingFace
2. Add to `AVAILABLE_MODELS` dict
3. Restart the server
4. Switch to your new model via API

## 📝 License

Apache 2.0 - See LICENSE file for details

## 🤝 Contributing

Contributions welcome! Please feel free to submit a Pull Request.

## 🐛 Troubleshooting

### Model fails to load
- Ensure `llama-server` is in your PATH
- Check available disk space for model downloads
- Verify internet connectivity for first-time model downloads

### Out of memory errors
- Reduce `MAX_CACHED_MODELS` to 1
- Use smaller quantized models (Q4_K_M instead of Q8)
- Increase system swap space

### Port conflicts
- Change `BASE_PORT` if 8080+ are in use
- Check for other llama-server instances: `ps aux | grep llama`

## 📚 Additional Resources

- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [llama.cpp GitHub](https://github.com/ggerganov/llama.cpp)
- [Hugging Face Models](https://huggingface.co/models?library=gguf)
- [OpenAI API Reference](https://platform.openai.com/docs/api-reference)

---

Built with ❤️ using FastAPI and llama.cpp