auto-arena
端到端对比多模型、自动化生成评测标准，适合模型选择与研究对比。
会调用外部模型端点并需 API 密钥，可能产生费用或泄露输入数据。
自动生成测试用例、收集模型响应并用判定模型生成排行胜率的对比框架技能。
自动为一个任务生成测试用例、并对多个模型/代理并发采集响应，然后用评判模型两两对比打分，最终给出胜率排名和可视化报告。适合做模型基准测试、对比评估或选型时使用，不需要事先准备测试集。亮点是端点可增量加入、支持断点续跑和随时替换 judge 模型，流程全自动化。
▸ 展开 SKILL.md 英文原文
Automatically evaluate and compare multiple AI models or agents without pre-existing test data. Generates test queries from a task description, collects responses from all target endpoints, auto-generates evaluation rubrics, runs pairwise comparisons via a judge model, and produces win-rate rankings with reports and charts. Supports checkpoint resume, incremental endpoint addition, and judge model hot-swap. Use when the user asks to compare, benchmark, or rank multiple models or agents on a custom task, or run an arena-style evaluation.
帮我安装这个 skill：https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/auto-arena/SKILL.mdcurl -fsSL "https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/auto-arena/SKILL.md"# Auto Arena Skill End-to-end automated model comparison using the OpenJudge `AutoArenaPipeline`: 1. **Generate queries** — LLM creates diverse test queries from task description 2. **Collect responses** — query all target endpoints concurrently 3. **Generate rubrics** — LLM produces evaluation criteria from task + sample queries 4. **Pairwise evaluation** — judge model compares every model pair (with position-bias swap) 5. **Analyze & rank** — compute win rates, win matrix, and rankings 6. **Report & charts** — Markdown report + win-rate bar chart + optional matrix heatmap ## Prerequisites ```bash # Install OpenJudge pip install py-openjudge # Extra dependency for auto_arena (chart gen