awq-quantization
仓库创建 2025年11月3日最近提交 1 个月前SkillHot 收录 20 天前
▸ 精选理由
比传统量化更保精度，适合在受限 GPU 上部署大模型以显著降成本与加速推理。
▸ 风险提示
量化可能导致精度下降或与特定模型/任务不兼容
这个 Skill 做什么
AWQ 提供激活感知的 4-bit 权重量化，在显存受限时提升推理速度并尽量保留精度。
把大型模型压到 4-bit，能大幅省显存并把推理速度提升到约 2.5–3×，同时尽量保留精度。适合在 GPU 显存受限时部署 7B–70B 模型、想要比 GPTQ 更快或更准的场景，或用于 instruction-tuned / multimodal 模型。特别之处是按激活模式（activation-aware）保留关键权重，实际效果好且是 MLSys 2024 的获奖工作。
▸ 展开 SKILL.md 英文原文
Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss. Use when deploying large models (7B-70B) on limited GPU memory, when you need faster inference than GPTQ with better accuracy preservation, or for instruction-tuned and multimodal models. MLSys 2024 Best Paper Award winner.
1.1w
Stars
817
Forks
40
仓库内 Skill
+704
7 日增星
安装 / 使用
给你的 Agent 一句话（通用）
帮我安装这个 skill：https://raw.githubusercontent.com/Orchestra-Research/AI-Research-SKILLs/main/10-optimization/awq/SKILL.md或 curl 直取 SKILL.md
curl -fsSL "https://raw.githubusercontent.com/Orchestra-Research/AI-Research-SKILLs/main/10-optimization/awq/SKILL.md"SKILL.MD 节选查看完整文件 ↗
# AWQ (Activation-aware Weight Quantization) 4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss. ## When to use AWQ **Use AWQ when:** - Need 4-bit quantization with <5% accuracy loss - Deploying instruction-tuned or chat models (AWQ generalizes better) - Want ~2.5-3x inference speedup over FP16 - Using vLLM for production serving - Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support **Use GPTQ instead when:** - Need maximum ecosystem compatibility (more tools support GPTQ) - Working with ExLlamaV2 backend specifically - Have older GPUs without Marlin support **Use bitsandbytes instead when:** - N
via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有