openjudge

仓库创建 2025年7月8日最近提交 18 天前SkillHot 收录 20 天前

▸ 精选理由

完整的评测组件与策略，适合模型对比与打分

这个 Skill 做什么

用 OpenJudge 构建 LLM 输出的自动化评估与评分流水线

用 OpenJudge 搭一条自动化的 LLM 评估流水线：如何选评分器、批量跑评估、合并分数并做统计分析。适合想比对模型、量化输出质量或自动化评审生成器效果的场景。亮点是支持多种 grader 类型和评估策略，方便做大规模、可复现的比较实验。

▸ 展开 SKILL.md 英文原文

Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an automated evaluation system.

研究检索评测框架自动评分批量评估通用

748

Stars

Forks

仓库内 Skill

+40

7 日增星

安装 / 使用

给你的 Agent 一句话（通用）

帮我安装这个 skill：https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/openjudge/SKILL.md

或 curl 直取 SKILL.md

curl -fsSL "https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/openjudge/SKILL.md"

SKILL.MD 节选查看完整文件 ↗

# OpenJudge Skill

Build evaluation pipelines for LLM applications using the `openjudge` library.

## When to Use This Skill

- User wants to evaluate LLM output quality (correctness, relevance, hallucination, etc.)
- User wants to compare two or more models and rank them
- User wants to design a scoring rubric and automate evaluation
- User wants to analyze evaluation results statistically
- User wants to build a reward model or quality filter

## Sub-documents — Read When Relevant

| Topic | File | Read when… |
|-------|------|------------|
| Grader selection & configuration | `graders.md` | User needs to pick or configure an evaluator |
| Batch evaluation pipeline | `pipeline.md` | User n

via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有