align-human

仓库创建 2025年7月8日最近提交 18 天前SkillHot 收录 8 小时前

▸ 精选理由

有严格门控条件，有助于安全且可验证地降低人工审核比例。

这个 Skill 做什么

评估自动评判器与人工标注的一致性，检测系统性偏差并规划人力替代路径。

帮你把自动 judge/grader 的判定和人工标注逐条比对，量化一致率并算出 TPR/TNR、Cohen's kappa 等指标，定位系统性偏差和边界样本问题。还能评估自动评估能否替代人工审核，给出逐步减少人工参与的人力替代路线和必须通过的检查。适合要验证评判器可信度、做校准或检查偏差时使用。

▸ 展开 SKILL.md 英文原文

Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.

研究检索人机对齐一致性分析偏差检测通用

748

Stars

Forks

仓库内 Skill

+40

7 日增星

安装 / 使用

给你的 Agent 一句话（通用）

帮我安装这个 skill：https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/03-align-human/SKILL.md

或 curl 直取 SKILL.md

curl -fsSL "https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/03-align-human/SKILL.md"

SKILL.MD 节选查看完整文件 ↗

<HARD-GATE>
NO calibrated:true WITHOUT TPR >= 0.8 AND TNR >= 0.8 AND boundary stratum TPR >= 0.6 AND TNR >= 0.6 AND n_dev >= 10 per class AND test-set drop < 10%.
NO human_reduction_phase >= 2 WITHOUT kappa >= 0.6 AND boundary kappa >= 0.6.
NO alignment conclusion WITHOUT all 5 bias checks completed.
</HARD-GATE>

# Align Human

Measure whether your automatic judge agrees with human judgment, detect where and
why they disagree, and build a roadmap to reduce human review over time.

## When to Activate

- You have a working judge/grader and 50+ human-labeled examples
- You want to know if the judge is trustworthy enough to replace human review
- You've noticed the judge's decisions being over

via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有