eval-design

仓库创建 2025年7月8日最近提交 18 天前SkillHot 收录 8 小时前

▸ 精选理由

能产出可直接用于自动评测（如 OpenJudge）的评测集，节省人工设计成本。

这个 Skill 做什么

为 LLM/agent 设计高质量评测数据、测试用例与分层样本并输出可用数据集。

为 LLM 和 agent 设计高质量的评测数据集，包含分层样本、对抗用例和难度分层，输出可直接使用的评测集。什么时候用：有模型日志、交互轨迹或需求，需要构建覆盖面好、难度可控的测试时。特别之处是能把业务维度抽成评测维度并导出兼容 OpenJudge/GradingRunner 的格式，方便自动化打分与复现。

▸ 展开 SKILL.md 英文原文

Use when the user needs to design evaluation datasets, create test cases, stratify samples, generate adversarial examples, extract eval dimensions from traces/specs, or build a labeled evaluation set. Also use when the user mentions test data design, eval coverage, difficulty stratification, synthetic data generation for eval, or "how to create good evaluation data." Outputs datasets in OpenJudge-compatible format.

研究检索评测数据测试用例分层采样通用

748

Stars

Forks

仓库内 Skill

+40

7 日增星

安装 / 使用

给你的 Agent 一句话（通用）

帮我安装这个 skill：https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/01-eval-design/SKILL.md

或 curl 直取 SKILL.md

curl -fsSL "https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/01-eval-design/SKILL.md"

SKILL.MD 节选查看完整文件 ↗

# Eval Design

Design high-quality evaluation datasets that measure what actually matters for your
application. You extract evaluation dimensions from business context, structure them
into stratified test cases, and output datasets ready for OpenJudge `GradingRunner`.

## When to Activate

- User has agent traces / production logs and wants to build an eval set from them
- User has evaluation principles but needs properly stratified test data
- User wants to generate adversarial examples that stress-test their system
- User needs coverage analysis — are they testing all the right things?
- User wants a labeling guide for human annotators

## Checklist

You MUST create a task for each item an

via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有