prompt-regression

仓库创建 2025年7月8日最近提交 18 天前SkillHot 收录 8 小时前

▸ 精选理由

强制置信区间与去偏方法，适合需要严谨验证提示改动效果的工程化场景。

这个 Skill 做什么

用严格统计方法对比两版提示，报告置信区间并校正位置偏倚。

把两版提示做严谨的对比，告诉你改动到底有没有统计学显著的提升，会给每个评测维度的胜率和置信区间。用在你改了 system prompt、RAG 模板或 agent 指令，想验证改动有无效果或是否出现回归时。最大的特点是做位置偏倚校正（swap-aggregate）并报告 bootstrap 95% CI，避免顺序或样本太少导致误判。

▸ 展开 SKILL.md 英文原文

Use when the user has changed a prompt (system prompt, RAG template, agent instruction, etc.) and wants to know whether the candidate is better or worse than the baseline. Also use when the user mentions prompt A/B testing, prompt comparison, prompt optimization validation, "did my prompt change help," or prompt regression testing. Outputs per-dimension win rates with statistical significance using OpenJudge PairwiseAnalyzer.

研究检索提示对比A/B 测试统计检验通用

748

Stars

Forks

仓库内 Skill

+40

7 日增星

安装 / 使用

给你的 Agent 一句话（通用）

帮我安装这个 skill：https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/06-prompt-regression/SKILL.md

或 curl 直取 SKILL.md

curl -fsSL "https://raw.githubusercontent.com/agentscope-ai/OpenJudge/main/skills/eval_pipeline/06-prompt-regression/SKILL.md"

SKILL.MD 节选查看完整文件 ↗

<HARD-GATE>
NO conclusion about which prompt is better WITHOUT bootstrap 95% CI reported.
NO candidate declared "better" WITHOUT position-debiased (swap-aggregate) comparison.
NO comparison with fewer than 10 samples per axis — CI is too wide to be meaningful.
</HARD-GATE>

# Prompt Regression

Compare two prompts head-to-head and determine, with statistical rigor, whether
the candidate is better, worse, or tied on each evaluation dimension.

## When to Activate

- You changed the system prompt and want to verify it's actually better
- You're iterating on RAG answer templates
- You're optimizing agent step-by-step instructions
- You want data to support a prompt change decision

## Checklist

via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有