ai-system-testing
专为 AI 功能独有的非确定性与注入攻击等故障模式设计的测试工具箱。
可能产生大量模型调用成本并涉及敏感数据处理
针对 AI/LLM 功能的系统化测试框架，涵盖提示回归、幻觉检测与安全红队策略。
帮你给产品里的 AI/LLM 功能做一套系统化的测试，检验提示回归、响应质量、tool-call 行为、幻觉（hallucination）和 RAG 归因等问题。上线新模型、改 prompt、怀疑模型静默降级或要做红队/安全扫描时会用。特点是考虑非确定性输出、支持评估框架和红队注入场景，还带有对不可信内容的检测能力，方便把 AI 特有的失败模式找出来并回归测试。
▸ 展开 SKILL.md 英文原文
Test AI/LLM features that ship in your product. Covers prompt regression testing, response quality evaluation, tool-call validation, hallucination and RAG grounding checks, nondeterministic-output strategies, red-team/safety scans, eval frameworks, and agent-as-target injection (indirect injection via tool output / RAG / scan reports, self-propagating payloads, data exfiltration via an agent) plus a bundled detector for untrusted content. Use when: "test our LLM feature," "prompt regression test," "eval framework," "hallucination test," "RAG grounding," "nondeterministic output," "AI feature testing," "red-team our chatbot," "indirect prompt injection," "agent reading untrusted tool output," "production AI quality." Not for: using AI to generate your own test code — use ai-test-generation. Not for: classifying CI failures with AI — use ai-bug-triage. Not for: EU AI Act / GDPR conformity of an AI feature — use compliance-testing. Not for: canary/flag rollout of an AI feature — use testi
帮我安装这个 skill：https://raw.githubusercontent.com/petrkindlmann/qa-skills/main/skills/ai-system-testing/SKILL.mdcurl -fsSL "https://raw.githubusercontent.com/petrkindlmann/qa-skills/main/skills/ai-system-testing/SKILL.md"<objective> AI features fail differently from deterministic software. The same input produces different outputs, correctness is subjective, and failure modes include hallucination, prompt injection, and silent quality decay. A chatbot that confidently cites a fabricated URL passes every `toBeDefined()` check. This skill covers how to test AI features rigorously despite nondeterminism: versioned prompts, eval suites scored against golden datasets, statistical and property-based assertions, tool-call validation, grounding checks, and red-team safety scans. </objective> --- ## Quick Route | Situation | Go to | |-----------|-------| | Prompt changed, need to catch quality regressions | Prompt