wip

.cursor/rules/clean-code.mdc

@@ -0,0 +1,60 @@
+---
+description:
+globs:
+alwaysApply: true
+---
+---
+description: Guidelines for writing clean, maintainable, and human-readable code. Apply these rules when writing or reviewing code to ensure consistency and quality.
+globs:
+---
+# Clean Code Guidelines
+
+## Constants Over Magic Numbers
+- Replace hard-coded values with named constants
+- Use descriptive constant names that explain the value's purpose
+- Keep constants at the top of the file or in a dedicated constants file
+
+## Meaningful Names
+- Variables, functions, and classes should reveal their purpose
+- Names should explain why something exists and how it's used
+- Avoid abbreviations unless they're universally understood
+
+## Smart Comments
+- Don't comment on what the code does - make the code self-documenting
+- Use comments to explain why something is done a certain way
+- Document APIs, complex algorithms, and non-obvious side effects
+
+## Single Responsibility
+- Each function should do exactly one thing
+- Functions should be small and focused
+- If a function needs a comment to explain what it does, it should be split
+
+## DRY (Don't Repeat Yourself)
+- Extract repeated code into reusable functions
+- Share common logic through proper abstraction
+- Maintain single sources of truth
+
+## Clean Structure
+- Keep related code together
+- Organize code in a logical hierarchy
+- Use consistent file and folder naming conventions
+
+## Encapsulation
+- Hide implementation details
+- Expose clear interfaces
+- Move nested conditionals into well-named functions
+
+## Code Quality Maintenance
+- Refactor continuously
+- Fix technical debt early
+- Leave code cleaner than you found it
+
+## Testing
+- Write tests before fixing bugs
+- Keep tests readable and maintainable
+- Test edge cases and error conditions
+
+## Version Control
+- Write clear commit messages
+- Make small, focused commits
+- Use meaningful branch names

.cursor/rules/cloudflare-worker-typescript.mdc

@@ -0,0 +1,63 @@
+---
+description:
+globs:
+alwaysApply: true
+---
+
+---
+description: TypeScript coding standards and best practices for modern web development
+globs: **/*.ts, **/*.tsx, **/*.d.ts
+---
+
+# TypeScript Best Practices
+
+## Type System
+- Prefer interfaces over types for object definitions
+- Use type for unions, intersections, and mapped types
+- Avoid using `any`, prefer `unknown` for unknown types
+- Use strict TypeScript configuration
+- Leverage TypeScript's built-in utility types
+- Use generics for reusable type patterns
+
+## Naming Conventions
+- Use PascalCase for type names and interfaces
+- Use camelCase for variables and functions
+- Use UPPER_CASE for constants
+- Use descriptive names with auxiliary verbs (e.g., isLoading, hasError)
+- Prefix interfaces for React props with 'Props' (e.g., ButtonProps)
+
+## Code Organization
+- Keep type definitions close to where they're used
+- Export types and interfaces from dedicated type files when shared
+- Use barrel exports (index.ts) for organizing exports
+- Place shared types in a `types` directory
+- Co-locate component props with their components
+
+## Functions
+- Use explicit return types for public functions
+- Use arrow functions for callbacks and methods
+- Implement proper error handling with custom error types
+- Use function overloads for complex type scenarios
+- Prefer async/await over Promises
+
+## Best Practices
+- Enable strict mode in tsconfig.json
+- Use readonly for immutable properties
+- Leverage discriminated unions for type safety
+- Use type guards for runtime type checking
+- Implement proper null checking
+- Avoid type assertions unless necessary
+
+## Error Handling
+- Create custom error types for domain-specific errors
+- Use Result types for operations that can fail
+- Implement proper error boundaries
+- Use try-catch blocks with typed catch clauses
+- Handle Promise rejections properly
+
+## Patterns
+- Use the Builder pattern for complex object creation
+- Implement the Repository pattern for data access
+- Use the Factory pattern for object creation
+- Leverage dependency injection
+- Use the Module pattern for encapsulation

.cursor/rules/nextjs.mdc

@@ -0,0 +1,57 @@
+---
+description:
+globs:
+alwaysApply: true
+---
+---
+description: Next.js with TypeScript and Tailwind UI best practices
+globs: **/*.tsx, **/*.ts, src/**/*.ts, src/**/*.tsx
+---
+
+# Next.js Best Practices
+
+## Project Structure
+- Use the App Router directory structure
+- Place components in `app` directory for route-specific components
+- Place shared components in `components` directory
+- Place utilities and helpers in `lib` directory
+- Use lowercase with dashes for directories (e.g., `components/auth-wizard`)
+
+## Components
+- Use Server Components by default
+- Mark client components explicitly with 'use client'
+- Wrap client components in Suspense with fallback
+- Use dynamic loading for non-critical components
+- Implement proper error boundaries
+- Place static content and interfaces at file end
+
+## Performance
+- Optimize images: Use WebP format, size data, lazy loading
+- Minimize use of 'useEffect' and 'setState'
+- Favor Server Components (RSC) where possible
+- Use dynamic loading for non-critical components
+- Implement proper caching strategies
+
+## Data Fetching
+- Use Server Components for data fetching when possible
+- Implement proper error handling for data fetching
+- Use appropriate caching strategies
+- Handle loading and error states appropriately
+
+## Routing
+- Use the App Router conventions
+- Implement proper loading and error states for routes
+- Use dynamic routes appropriately
+- Handle parallel routes when needed
+
+## Forms and Validation
+- Use Zod for form validation
+- Implement proper server-side validation
+- Handle form errors appropriately
+- Show loading states during form submission
+
+## State Management
+- Minimize client-side state
+- Use React Context sparingly
+- Prefer server state when possible
+- Implement proper loading states

.cursor/rules/tailwind.mdc

@@ -0,0 +1,83 @@
+---
+description:
+globs:
+alwaysApply: false
+---
+---
+description: Tailwind CSS and UI component best practices for modern web applications
+globs: **/*.css, **/*.tsx, **/*.jsx, tailwind.config.js, tailwind.config.ts
+---
+
+# Tailwind CSS Best Practices
+
+## Project Setup
+- Use proper Tailwind configuration
+- Configure theme extension properly
+- Set up proper purge configuration
+- Use proper plugin integration
+- Configure custom spacing and breakpoints
+- Set up proper color palette
+
+## Component Styling
+- Use utility classes over custom CSS
+- Group related utilities with @apply when needed
+- Use proper responsive design utilities
+- Implement dark mode properly
+- Use proper state variants
+- Keep component styles consistent
+
+## Layout
+- Use Flexbox and Grid utilities effectively
+- Implement proper spacing system
+- Use container queries when needed
+- Implement proper responsive breakpoints
+- Use proper padding and margin utilities
+- Implement proper alignment utilities
+
+## Typography
+- Use proper font size utilities
+- Implement proper line height
+- Use proper font weight utilities
+- Configure custom fonts properly
+- Use proper text alignment
+- Implement proper text decoration
+
+## Colors
+- Use semantic color naming
+- Implement proper color contrast
+- Use opacity utilities effectively
+- Configure custom colors properly
+- Use proper gradient utilities
+- Implement proper hover states
+
+## Components
+- Use shadcn/ui components when available
+- Extend components properly
+- Keep component variants consistent
+- Implement proper animations
+- Use proper transition utilities
+- Keep accessibility in mind
+
+## Responsive Design
+- Use mobile-first approach
+- Implement proper breakpoints
+- Use container queries effectively
+- Handle different screen sizes properly
+- Implement proper responsive typography
+- Use proper responsive spacing
+
+## Performance
+- Use proper purge configuration
+- Minimize custom CSS
+- Use proper caching strategies
+- Implement proper code splitting
+- Optimize for production
+- Monitor bundle size
+
+## Best Practices
+- Follow naming conventions
+- Keep styles organized
+- Use proper documentation
+- Implement proper testing
+- Follow accessibility guidelines
+- Use proper version control

.cursor/rules/tech-stack.mdc

@@ -0,0 +1,22 @@
+---
+description:
+globs:
+alwaysApply: true
+---
+
+# Your rule content
+
+- use python version 3.12 only
+- Use FastAPI exclusively
+- always add tests using pytest
+- always add imports at the top of the file, never in a function
+- if a code is defined somewhere reuse it or refactor it
+- files can't exceed 300 lined of code at that point refactor
+- don't document obvious lined of code
+- use pydantic for types and objects
+- if a database is needed use SQLModel
+- if a LLM call is needed use LiteLLM
+- if an email is needed use resend
+- use httpx for requests
+- for logging use loguru
+-

README.md

@@ -11,11 +11,117 @@ short_description: Duplicate this leaderboard to initialize your own!
 sdk_version: 5.19.0
 ---
 
+# AI Evaluation Judge Arena
+
+An interactive platform for comparing and ranking AI evaluation models (judges) based on human preferences.
+
+## Overview
+
+This application allows users to:
+
+1. View AI-generated outputs based on input prompts
+2. Compare evaluations from two different AI judges
+3. Select the better evaluation
+4. Build a leaderboard of judges ranked by ELO score
+
+## Features
+
+- **Blind Comparison**: Judge identities are hidden until after selection
+- **ELO Rating System**: Calculates judge rankings based on user preferences
+- **Leaderboard**: Track performance of different evaluation models
+- **Sample Examples**: Includes pre-loaded examples for immediate testing
+
+## Setup
+
+### Prerequisites
+
+- Python 3.6+
+- Required packages: gradio, pandas, numpy
+
+### Installation
+
+1. Clone this repository:
+
+```
+git clone https://github.com/yourusername/eval-arena.git
+cd eval-arena
+```
+
+2. Install dependencies:
+
+```
+pip install -r requirements.txt
+```
+
+3. Run the application:
+
+```
+python app.py
+```
+
+4. Open your browser and navigate to the URL displayed in the terminal (typically http://127.0.0.1:7860)
+
+## Usage
+
+1. **Get Random Example**: Click to load a random input/output pair
+2. **Get Judge Evaluations**: View two anonymous evaluations of the output
+3. **Select Better Evaluation**: Choose which evaluation you prefer
+4. **See Results**: Learn which judges you compared and update the leaderboard
+5. **Leaderboard Tab**: View current rankings of all judges
+
+## Extending the Application
+
+### Adding New Examples
+
+Add new examples in JSON format to the `data/examples` directory:
+
+```json
+{
+  "id": "example_id",
+  "input": "Your input prompt",
+  "output": "AI-generated output to evaluate"
+}
+```
+
+### Adding New Judges
+
+Add new judges in JSON format to the `data/judges` directory:
+
+```json
+{
+  "id": "judge_id",
+  "name": "Judge Name",
+  "description": "Description of judge's evaluation approach"
+}
+```
+
+### Integrating Real Models
+
+For production use, modify the `get_random_judges_evaluations` function to call actual AI evaluation models instead of using the simulated evaluations.
+
+## License
+
+MIT
+
+## Citation
+
+If you use this platform in your research, please cite:
+
+```
+@software{ai_eval_arena,
+  author = {Your Name},
+  title = {AI Evaluation Judge Arena},
+  year = {2023},
+  url = {https://github.com/yourusername/eval-arena}
+}
+```
+
 # Start the configuration
 
 Most of the variables to change for a default leaderboard are in `src/env.py` (replace the path for your leaderboard) and `src/about.py` (for tasks).
 
 Results files should have the following format and be stored as json files:
+
 ```json
 {
     "config": {
@@ -40,7 +146,8 @@ If you encounter problem on the space, don't hesitate to restart it to remove th
 
 # Code logic for more complex edits
 
-You'll find 
+You'll find
+
 - the main table' columns names and properties in `src/display/utils.py`
 - the logic to read all results and request files, then convert them in dataframe lines, in `src/leaderboard/read_evals.py`, and `src/populate.py`
-- the logic to allow or filter submissions in `src/submission/submit.py` and `src/submission/check_validity.py`
+- the logic to allow or filter submissions in `src/submission/submit.py` and `src/submission/check_validity.py`

app.py

@@ -1,204 +1,375 @@
+import json
+import random
+from pathlib import Path
+
 import gradio as gr
-from gradio_leaderboard import Leaderboard, ColumnFilter, SelectColumns
 import pandas as pd
-from apscheduler.schedulers.background import BackgroundScheduler
-from huggingface_hub import snapshot_download
-
-from src.about import (
-    CITATION_BUTTON_LABEL,
-    CITATION_BUTTON_TEXT,
-    EVALUATION_QUEUE_TEXT,
-    INTRODUCTION_TEXT,
-    LLM_BENCHMARKS_TEXT,
-    TITLE,
-)
-from src.display.css_html_js import custom_css
-from src.display.utils import (
-    BENCHMARK_COLS,
-    COLS,
-    EVAL_COLS,
-    EVAL_TYPES,
-    AutoEvalColumn,
-    ModelType,
-    fields,
-    WeightType,
-    Precision
-)
-from src.envs import API, EVAL_REQUESTS_PATH, EVAL_RESULTS_PATH, QUEUE_REPO, REPO_ID, RESULTS_REPO, TOKEN
-from src.populate import get_evaluation_queue_df, get_leaderboard_df
-from src.submission.submit import add_new_eval
-
-
-def restart_space():
-    API.restart_space(repo_id=REPO_ID)
-
-### Space initialisation
-try:
-    print(EVAL_REQUESTS_PATH)
-    snapshot_download(
-        repo_id=QUEUE_REPO, local_dir=EVAL_REQUESTS_PATH, repo_type="dataset", tqdm_class=None, etag_timeout=30, token=TOKEN
-    )
-except Exception:
-    restart_space()
-try:
-    print(EVAL_RESULTS_PATH)
-    snapshot_download(
-        repo_id=RESULTS_REPO, local_dir=EVAL_RESULTS_PATH, repo_type="dataset", tqdm_class=None, etag_timeout=30, token=TOKEN
+
+# Constants
+DATA_DIR = Path("data")
+JUDGES_DIR = DATA_DIR / "judges"
+EXAMPLES_DIR = DATA_DIR / "examples"
+LEADERBOARD_PATH = DATA_DIR / "leaderboard.csv"
+HISTORY_PATH = DATA_DIR / "history.csv"
+
+# Initialize data directories
+DATA_DIR.mkdir(exist_ok=True)
+JUDGES_DIR.mkdir(exist_ok=True)
+EXAMPLES_DIR.mkdir(exist_ok=True)
+
+# ELO calculation parameters
+K_FACTOR = 32  # Standard chess K-factor
+
+# Initialize leaderboard if it doesn't exist
+if not LEADERBOARD_PATH.exists():
+    leaderboard_df = pd.DataFrame(
+        {"judge_id": [], "judge_name": [], "elo_score": [], "wins": [], "losses": [], "total_evaluations": []}
     )
-except Exception:
-    restart_space()
-
-
-LEADERBOARD_DF = get_leaderboard_df(EVAL_RESULTS_PATH, EVAL_REQUESTS_PATH, COLS, BENCHMARK_COLS)
-
-(
-    finished_eval_queue_df,
-    running_eval_queue_df,
-    pending_eval_queue_df,
-) = get_evaluation_queue_df(EVAL_REQUESTS_PATH, EVAL_COLS)
-
-def init_leaderboard(dataframe):
-    if dataframe is None or dataframe.empty:
-        raise ValueError("Leaderboard DataFrame is empty or None.")
-    return Leaderboard(
-        value=dataframe,
-        datatype=[c.type for c in fields(AutoEvalColumn)],
-        select_columns=SelectColumns(
-            default_selection=[c.name for c in fields(AutoEvalColumn) if c.displayed_by_default],
-            cant_deselect=[c.name for c in fields(AutoEvalColumn) if c.never_hidden],
-            label="Select Columns to Display:",
-        ),
-        search_columns=[AutoEvalColumn.model.name, AutoEvalColumn.license.name],
-        hide_columns=[c.name for c in fields(AutoEvalColumn) if c.hidden],
-        filter_columns=[
-            ColumnFilter(AutoEvalColumn.model_type.name, type="checkboxgroup", label="Model types"),
-            ColumnFilter(AutoEvalColumn.precision.name, type="checkboxgroup", label="Precision"),
-            ColumnFilter(
-                AutoEvalColumn.params.name,
-                type="slider",
-                min=0.01,
-                max=150,
-                label="Select the number of parameters (B)",
-            ),
-            ColumnFilter(
-                AutoEvalColumn.still_on_hub.name, type="boolean", label="Deleted/incomplete", default=True
-            ),
-        ],
-        bool_checkboxgroup_label="Hide models",
-        interactive=False,
+    leaderboard_df.to_csv(LEADERBOARD_PATH, index=False)
+else:
+    leaderboard_df = pd.read_csv(LEADERBOARD_PATH)
+
+# Initialize history if it doesn't exist
+if not HISTORY_PATH.exists():
+    history_df = pd.DataFrame(
+        {
+            "timestamp": [],
+            "input": [],
+            "output": [],
+            "judge1_id": [],
+            "judge1_name": [],
+            "judge1_evaluation": [],
+            "judge2_id": [],
+            "judge2_name": [],
+            "judge2_evaluation": [],
+            "winner_id": [],
+            "user_ip": [],
+        }
     )
+    history_df.to_csv(HISTORY_PATH, index=False)
+else:
+    history_df = pd.read_csv(HISTORY_PATH)
 
+# Sample data - For demonstration purposes
+# In production, these would be loaded from a proper dataset
+if not list(EXAMPLES_DIR.glob("*.json")) and not list(JUDGES_DIR.glob("*.json")):
+    # Create sample examples
+    sample_examples = [
+        {
+            "id": "example1",
+            "input": "Write a poem about the ocean.",
+            "output": "The waves crash and foam,\nSalt spray fills the air like mist,\nOcean breathes deeply.",
+        },
+        {
+            "id": "example2",
+            "input": "Explain how photosynthesis works.",
+            "output": "Photosynthesis is the process where plants convert sunlight, water, "
+            "and carbon dioxide into glucose and oxygen. The chlorophyll in plant "
+            "cells captures light energy, which is then used to convert CO2 and "
+            "water into glucose, releasing oxygen as a byproduct.",
+        },
+        {
+            "id": "example3",
+            "input": "Solve this math problem: If x + y = 10 and x - y = 4, what are x and y?",
+            "output": "To solve this system of equations:\nx + y = 10\nx - y = 4\n\n"
+            "Add these equations:\n2x = 14\nx = 7\n\nSubstitute back:\n"
+            "7 + y = 10\ny = 3\n\nTherefore, x = 7 and y = 3.",
+        },
+    ]
 
-demo = gr.Blocks(css=custom_css)
-with demo:
-    gr.HTML(TITLE)
-    gr.Markdown(INTRODUCTION_TEXT, elem_classes="markdown-text")
+    for example in sample_examples:
+        with open(EXAMPLES_DIR / f"{example['id']}.json", "w") as f:
+            json.dump(example, f)
 
-    with gr.Tabs(elem_classes="tab-buttons") as tabs:
-        with gr.TabItem("🏅 LLM Benchmark", elem_id="llm-benchmark-tab-table", id=0):
-            leaderboard = init_leaderboard(LEADERBOARD_DF)
+    # Create sample judges
+    sample_judges = [
+        {
+            "id": "judge1",
+            "name": "EvalGPT",
+            "description": "A comprehensive evaluation model focused on accuracy and completeness",
+        },
+        {
+            "id": "judge2",
+            "name": "CritiqueBot",
+            "description": "An evaluation model specializing in identifying factual errors",
+        },
+        {
+            "id": "judge3",
+            "name": "GradeAssist",
+            "description": "A holistic evaluation model that balances substance and style",
+        },
+        {
+            "id": "judge4",
+            "name": "PrecisionJudge",
+            "description": "A technical evaluator that emphasizes precision and correctness",
+        },
+    ]
 
-        with gr.TabItem("📝 About", elem_id="llm-benchmark-tab-table", id=2):
-            gr.Markdown(LLM_BENCHMARKS_TEXT, elem_classes="markdown-text")
+    for judge in sample_judges:
+        with open(JUDGES_DIR / f"{judge['id']}.json", "w") as f:
+            json.dump(judge, f)
 
-        with gr.TabItem("🚀 Submit here! ", elem_id="llm-benchmark-tab-table", id=3):
-            with gr.Column():
-                with gr.Row():
-                    gr.Markdown(EVALUATION_QUEUE_TEXT, elem_classes="markdown-text")
-
-                with gr.Column():
-                    with gr.Accordion(
-                        f"✅ Finished Evaluations ({len(finished_eval_queue_df)})",
-                        open=False,
-                    ):
-                        with gr.Row():
-                            finished_eval_table = gr.components.Dataframe(
-                                value=finished_eval_queue_df,
-                                headers=EVAL_COLS,
-                                datatype=EVAL_TYPES,
-                                row_count=5,
-                            )
-                    with gr.Accordion(
-                        f"🔄 Running Evaluation Queue ({len(running_eval_queue_df)})",
-                        open=False,
-                    ):
-                        with gr.Row():
-                            running_eval_table = gr.components.Dataframe(
-                                value=running_eval_queue_df,
-                                headers=EVAL_COLS,
-                                datatype=EVAL_TYPES,
-                                row_count=5,
-                            )
-
-                    with gr.Accordion(
-                        f"⏳ Pending Evaluation Queue ({len(pending_eval_queue_df)})",
-                        open=False,
-                    ):
-                        with gr.Row():
-                            pending_eval_table = gr.components.Dataframe(
-                                value=pending_eval_queue_df,
-                                headers=EVAL_COLS,
-                                datatype=EVAL_TYPES,
-                                row_count=5,
-                            )
-            with gr.Row():
-                gr.Markdown("# ✉️✨ Submit your model here!", elem_classes="markdown-text")
-
-            with gr.Row():
-                with gr.Column():
-                    model_name_textbox = gr.Textbox(label="Model name")
-                    revision_name_textbox = gr.Textbox(label="Revision commit", placeholder="main")
-                    model_type = gr.Dropdown(
-                        choices=[t.to_str(" : ") for t in ModelType if t != ModelType.Unknown],
-                        label="Model type",
-                        multiselect=False,
-                        value=None,
-                        interactive=True,
-                    )
-
-                with gr.Column():
-                    precision = gr.Dropdown(
-                        choices=[i.value.name for i in Precision if i != Precision.Unknown],
-                        label="Precision",
-                        multiselect=False,
-                        value="float16",
-                        interactive=True,
-                    )
-                    weight_type = gr.Dropdown(
-                        choices=[i.value.name for i in WeightType],
-                        label="Weights type",
-                        multiselect=False,
-                        value="Original",
-                        interactive=True,
-                    )
-                    base_model_name_textbox = gr.Textbox(label="Base model (for delta or adapter weights)")
-
-            submit_button = gr.Button("Submit Eval")
-            submission_result = gr.Markdown()
-            submit_button.click(
-                add_new_eval,
+    # Initialize leaderboard with sample judges
+    for judge in sample_judges:
+        if judge["id"] not in leaderboard_df["judge_id"].values:
+            leaderboard_df = pd.concat(
                 [
-                    model_name_textbox,
-                    base_model_name_textbox,
-                    revision_name_textbox,
-                    precision,
-                    weight_type,
-                    model_type,
+                    leaderboard_df,
+                    pd.DataFrame(
+                        {
+                            "judge_id": [judge["id"]],
+                            "judge_name": [judge["name"]],
+                            "elo_score": [1500],  # Starting ELO
+                            "wins": [0],
+                            "losses": [0],
+                            "total_evaluations": [0],
+                        }
+                    ),
                 ],
-                submission_result,
+                ignore_index=True,
             )
 
-    with gr.Row():
-        with gr.Accordion("📙 Citation", open=False):
-            citation_button = gr.Textbox(
-                value=CITATION_BUTTON_TEXT,
-                label=CITATION_BUTTON_LABEL,
-                lines=20,
-                elem_id="citation-button",
-                show_copy_button=True,
+    leaderboard_df.to_csv(LEADERBOARD_PATH, index=False)
+
+
+# Function to get a random example
+def get_random_example():
+    example_files = list(EXAMPLES_DIR.glob("*.json"))
+    if not example_files:
+        return {"input": "No examples available", "output": ""}
+
+    example_file = random.choice(example_files)
+    with open(example_file, "r") as f:
+        example = json.load(f)
+
+    return example
+
+
+# Function to get random judges' evaluations
+def get_random_judges_evaluations(example_input, example_output):
+    judge_files = list(JUDGES_DIR.glob("*.json"))
+    if len(judge_files) < 2:
+        return None, None
+
+    # Choose two different judges
+    selected_judge_files = random.sample(judge_files, 2)
+
+    judges = []
+    for judge_file in selected_judge_files:
+        with open(judge_file, "r") as f:
+            judge = json.load(f)
+        judges.append(judge)
+
+    # In a real application, we'd call the judge models here
+    # For demonstration, we'll create sample evaluations
+    evaluations = []
+    for judge in judges:
+        # Simulate different evaluation styles
+        if "factual" in judge["description"].lower():
+            evaluation = (
+                f"Evaluation by {judge['name']} (ID: {judge['id']}):\n\n"
+                f"Factual Accuracy: 8/10\nCompleteness: 7/10\n"
+                f"Conciseness: 9/10\n\nThe response addresses the question "
+                f"but could include more specific details."
+            )
+        elif "holistic" in judge["description"].lower():
+            evaluation = (
+                f"Evaluation by {judge['name']} (ID: {judge['id']}):\n\n"
+                f"Content Quality: 7/10\nStructure: 8/10\nClarity: 9/10\n\n"
+                f"The response is clear and well-structured, though it could "
+                f"be more comprehensive."
             )
+        elif "technical" in judge["description"].lower():
+            evaluation = (
+                f"Evaluation by {judge['name']} (ID: {judge['id']}):\n\n"
+                f"Technical Accuracy: 9/10\nCompleteness: 7/10\n"
+                f"Logical Flow: 8/10\n\nThe technical aspects are accurate, "
+                f"but some key details are missing."
+            )
+        else:
+            evaluation = (
+                f"Evaluation by {judge['name']} (ID: {judge['id']}):\n\n"
+                f"Quality: 8/10\nRelevance: 9/10\nPrecision: 7/10\n\n"
+                f"The response is relevant and of good quality, but could "
+                f"be more precise."
+            )
+
+        # Remove the judge ID from the displayed evaluation for blindness
+        display_evaluation = evaluation.replace(f" (ID: {judge['id']})", "")
+
+        evaluations.append({"judge": judge, "evaluation": evaluation, "display_evaluation": display_evaluation})
+
+    return evaluations[0], evaluations[1]
+
+
+# Calculate new ELO scores
+def calculate_elo(winner_rating, loser_rating):
+    expected_winner = 1 / (1 + 10 ** ((loser_rating - winner_rating) / 400))
+    expected_loser = 1 / (1 + 10 ** ((winner_rating - loser_rating) / 400))
+
+    new_winner_rating = winner_rating + K_FACTOR * (1 - expected_winner)
+    new_loser_rating = loser_rating + K_FACTOR * (0 - expected_loser)
+
+    return new_winner_rating, new_loser_rating
+
+
+# Update leaderboard after a comparison
+def update_leaderboard(winner_id, loser_id):
+    global leaderboard_df
+
+    # Get current ratings
+    winner_row = leaderboard_df[leaderboard_df["judge_id"] == winner_id].iloc[0]
+    loser_row = leaderboard_df[leaderboard_df["judge_id"] == loser_id].iloc[0]
+
+    winner_rating = winner_row["elo_score"]
+    loser_rating = loser_row["elo_score"]
+
+    # Calculate new ratings
+    new_winner_rating, new_loser_rating = calculate_elo(winner_rating, loser_rating)
+
+    # Update dataframe
+    leaderboard_df.loc[leaderboard_df["judge_id"] == winner_id, "elo_score"] = new_winner_rating
+    leaderboard_df.loc[leaderboard_df["judge_id"] == loser_id, "elo_score"] = new_loser_rating
+
+    # Update win/loss counts
+    leaderboard_df.loc[leaderboard_df["judge_id"] == winner_id, "wins"] += 1
+    leaderboard_df.loc[leaderboard_df["judge_id"] == loser_id, "losses"] += 1
+
+    # Update total evaluations
+    leaderboard_df.loc[leaderboard_df["judge_id"] == winner_id, "total_evaluations"] += 1
+    leaderboard_df.loc[leaderboard_df["judge_id"] == loser_id, "total_evaluations"] += 1
+
+    # Sort by ELO score and save
+    leaderboard_df = leaderboard_df.sort_values(by="elo_score", ascending=False).reset_index(drop=True)
+    leaderboard_df.to_csv(LEADERBOARD_PATH, index=False)
+
+    return leaderboard_df
+
+
+# Gradio interface functions
+def refresh_example():
+    example = get_random_example()
+    return example["input"], example["output"]
+
+
+def submit_example(input_text, output_text):
+    # Global state to store evaluations
+    global eval1, eval2
+
+    eval1, eval2 = get_random_judges_evaluations(input_text, output_text)
+
+    if not eval1 or not eval2:
+        return ("Error: Not enough judges available", "Error: Not enough judges available", None, None)
+
+    return (eval1["display_evaluation"], eval2["display_evaluation"], gr.update(visible=True), gr.update(visible=True))
+
+
+def select_winner(choice):
+    if not eval1 or not eval2:
+        return "Error: No evaluations available"
+
+    if choice == "Evaluation 1":
+        winner_eval = eval1
+        loser_eval = eval2
+    else:
+        winner_eval = eval2
+        loser_eval = eval1
+
+    # Update leaderboard
+    updated_leaderboard = update_leaderboard(winner_eval["judge"]["id"], loser_eval["judge"]["id"])
+
+    # Construct result message
+    result_message = f"You selected: {choice}\n\n"
+    result_message += f"Evaluation 1 was by: {eval1['judge']['name']} "
+    result_message += f"({eval1['judge']['description']})\n"
+    result_message += f"Evaluation 2 was by: {eval2['judge']['name']} "
+    result_message += f"({eval2['judge']['description']})\n\n"
+
+    winner_elo = updated_leaderboard[updated_leaderboard["judge_id"] == winner_eval["judge"]["id"]][
+        "elo_score"
+    ].values[0]
+
+    result_message += f"Winner: {winner_eval['judge']['name']} (New ELO: {winner_elo:.2f})\n"
+
+    return result_message
+
+
+# Create Gradio interface
+with gr.Blocks(title="AI Evaluation Judge Arena") as demo:
+    gr.Markdown("# AI Evaluation Judge Arena")
+    gr.Markdown(
+        "Choose which AI judge provides better evaluation of the output. "
+        "The judges' identities are hidden until you make your choice."
+    )
+
+    with gr.Tab("Evaluate Judges"):
+        with gr.Row():
+            with gr.Column(scale=1):
+                refresh_button = gr.Button("Get Random Example")
+
+            with gr.Column(scale=2):
+                input_text = gr.Textbox(label="Input", lines=4)
+                output_text = gr.Textbox(label="Output", lines=6)
+                submit_button = gr.Button("Get Judge Evaluations")
+
+        with gr.Row():
+            with gr.Column():
+                evaluation1 = gr.Textbox(label="Evaluation 1", lines=10)
+                select_eval1 = gr.Button("Select Evaluation 1", visible=False)
+
+            with gr.Column():
+                evaluation2 = gr.Textbox(label="Evaluation 2", lines=10)
+                select_eval2 = gr.Button("Select Evaluation 2", visible=False)
+
+        result_text = gr.Textbox(label="Result", lines=6)
+
+    with gr.Tab("Leaderboard"):
+        leaderboard_dataframe = gr.DataFrame(
+            value=leaderboard_df,
+            headers=["Judge Name", "ELO Score", "Wins", "Losses", "Total Evaluations"],
+            datatype=["str", "number", "number", "number", "number"],
+            col_count=(5, "fixed"),
+            interactive=False,
+        )
+        refresh_leaderboard = gr.Button("Refresh Leaderboard")
+
+    with gr.Tab("About"):
+        gr.Markdown(
+            """
+        ## About AI Evaluation Judge Arena
+
+        This platform allows users to compare and rate different AI evaluation models (judges).
+
+        ### How it works:
+        1. You are presented with an input prompt and AI-generated output
+        2. Two AI judges provide evaluations of the output
+        3. You select which evaluation you think is better
+        4. The judges' identities are revealed, and their ELO ratings are updated
+
+        ### ELO Rating System
+        The platform uses the ELO rating system (like in chess) to rank the judges. When you choose a winner:
+        - The winning judge gains ELO points
+        - The losing judge loses ELO points
+        - The amount of points transferred depends on the difference in current ratings
+
+        ### Purpose
+        This platform helps determine which AI evaluation methods are most aligned with human preferences.
+        """
+        )
+
+    # Set up event handlers
+    refresh_button.click(refresh_example, [], [input_text, output_text])
+    submit_button.click(
+        submit_example, [input_text, output_text], [evaluation1, evaluation2, select_eval1, select_eval2]
+    )
+    select_eval1.click(lambda: select_winner("Evaluation 1"), [], result_text)
+    select_eval2.click(lambda: select_winner("Evaluation 2"), [], result_text)
+    refresh_leaderboard.click(lambda: leaderboard_df, [], leaderboard_dataframe)
+
+# Initialize global variables for evaluation state
+eval1 = None
+eval2 = None
 
-scheduler = BackgroundScheduler()
-scheduler.add_job(restart_space, "interval", seconds=1800)
-scheduler.start()
-demo.queue(default_concurrency_limit=40).launch()
+# Launch the app
+if __name__ == "__main__":
+    demo.launch()

data/examples/example1.json

@@ -0,0 +1 @@
+{"id": "example1", "input": "Write a poem about the ocean.", "output": "The waves crash and foam,\nSalt spray fills the air like mist,\nOcean breathes deeply."}

data/examples/example2.json

@@ -0,0 +1 @@
+{"id": "example2", "input": "Explain how photosynthesis works.", "output": "Photosynthesis is the process where plants convert sunlight, water, and carbon dioxide into glucose and oxygen. The chlorophyll in plant cells captures light energy, which is then used to convert CO2 and water into glucose, releasing oxygen as a byproduct."}

data/examples/example3.json

@@ -0,0 +1 @@
+{"id": "example3", "input": "Solve this math problem: If x + y = 10 and x - y = 4, what are x and y?", "output": "To solve this system of equations:\nx + y = 10\nx - y = 4\n\nAdd these equations:\n2x = 14\nx = 7\n\nSubstitute back:\n7 + y = 10\ny = 3\n\nTherefore, x = 7 and y = 3."}

data/history.csv

@@ -0,0 +1 @@
+timestamp,input,output,judge1_id,judge1_name,judge1_evaluation,judge2_id,judge2_name,judge2_evaluation,winner_id,user_ip

data/judges/judge1.json

@@ -0,0 +1 @@
+{"id": "judge1", "name": "EvalGPT", "description": "A comprehensive evaluation model focused on accuracy and completeness"}

data/judges/judge2.json

@@ -0,0 +1 @@
+{"id": "judge2", "name": "CritiqueBot", "description": "An evaluation model specializing in identifying factual errors"}

data/judges/judge3.json

@@ -0,0 +1 @@
+{"id": "judge3", "name": "GradeAssist", "description": "A holistic evaluation model that balances substance and style"}

data/judges/judge4.json

@@ -0,0 +1 @@
+{"id": "judge4", "name": "PrecisionJudge", "description": "A technical evaluator that emphasizes precision and correctness"}

data/leaderboard.csv

@@ -0,0 +1,5 @@
+judge_id,judge_name,elo_score,wins,losses,total_evaluations
+judge2,CritiqueBot,1516.0,1.0,0.0,1.0
+judge4,PrecisionJudge,1500.736306793522,1.0,1.0,2.0
+judge3,GradeAssist,1500.0,0.0,0.0,0.0
+judge1,EvalGPT,1483.263693206478,0.0,1.0,1.0

requirements.txt

@@ -1,16 +1,3 @@
-APScheduler
-black
-datasets
 gradio
-gradio[oauth]
-gradio_leaderboard==0.0.13
-gradio_client
-huggingface-hub>=0.18.0
-matplotlib
 numpy
 pandas
-python-dateutil
-tqdm
-transformers
-tokenizers>=0.15.0
-sentencepiece