metric-design

仓库创建 2025年7月8日最近提交 18 天前SkillHot 收录 8 小时前

▸ 精选理由

输出可执行的 GradingRunner 代码，便于将多种评估器整合到管道中。

▸ 风险提示

需要安装并依赖 py-openjudge 等外部 SDK

这个 Skill 做什么

选择、配置并组合评测打分器，生成可用于自动评估的度量与代码。

帮你把评测思路变成可执行的自动化评估流水线：挑合适的 graders、写 judge prompt、把多个度量合成复合分并输出可跑的 OpenJudge 代码。用在你有数据或评估原则但不知道怎么量化、组合评分或自动化评测时。特点是偏 SDK/代码产出，会给出能运行的 GradingRunner 风格管道，而不是只讲理论。

▸ 展开 SKILL.md 英文原文

Use when the user has evaluation principles or a dataset but needs help choosing the right graders, designing evaluation metrics, creating LLM-as-judge prompts, combining multiple metrics into a composite score, or building an automated evaluation pipeline. Also use when the user mentions grader selection, metric design, judge prompt engineering, rubric design, evaluation pipeline code, or "how to evaluate [X] automatically." Outputs executable OpenJudge pipeline code.

研究检索指标设计打分器配置评估管道通用

748

Stars

Forks

仓库内 Skill

+40

7 日增星

安装 / 使用

给你的 Agent 一句话（通用）

帮我安装这个 skill：https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/02-metric-design/SKILL.md

或 curl 直取 SKILL.md

curl -fsSL "https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/02-metric-design/SKILL.md"

SKILL.MD 节选查看完整文件 ↗

# Metric Design

Select, configure, and combine evaluation graders into a working pipeline. You choose
the right tool for each evaluation dimension — from zero-cost code checks to LLM judges
— and produce executable `GradingRunner` code that runs on OpenJudge.

> **Requires OpenJudge** (`pip install py-openjudge`). This skill is intentionally
> SDK-centric — grader selection, `GradingRunner`, and aggregators are OpenJudge APIs. The
> design/decision logic still applies if you use another harness; only the code does not.

## When to Activate

- User has eval dimensions/principles but doesn't know which grader type to use
- User wants to write an LLM-as-judge prompt for a specific failure mode

via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有