Orchestra-Research/AI-Research-SKILLs✦ 精选0°

sentencepiece

仓库创建 2025年11月3日最近提交 1 个月前SkillHot 收录 20 天前

▸ 精选理由

适用于多语种与 CJK 场景，保证确定性与轻量部署。

这个 Skill 做什么

语言无关的分词器，直接在原始 Unicode 文本上训练 BPE 或 Unigram 词表。

一个语言无关的分词器，在原始 Unicode 文本上直接用 BPE 或 Unigram 训练词表，不依赖语言特定的预处理。适合训练多语言模型、处理 CJK（中日韩）文本或需要确定性、可复现分词的场景。特点是速度快、内存占用低、训练时不需预分词，产出的词表可复现，被 T5、ALBERT、XLNet、mBART 等采用。

▸ 展开 SKILL.md 英文原文

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-tokenization. Use when you need multilingual support, CJK languages, or reproducible tokenization.

数据与抓取无语言假设可训练词表多语种通用

1.1w

Stars

817

Forks

仓库内 Skill

+704

7 日增星

安装 / 使用

给你的 Agent 一句话（通用）

帮我安装这个 skill：https://raw.githubusercontent.com/Orchestra-Research/AI-Research-SKILLs/main/02-tokenization/sentencepiece/SKILL.md

或 curl 直取 SKILL.md

curl -fsSL "https://raw.githubusercontent.com/Orchestra-Research/AI-Research-SKILLs/main/02-tokenization/sentencepiece/SKILL.md"

SKILL.MD 节选查看完整文件 ↗

# SentencePiece - Language-Independent Tokenization

Unsupervised tokenizer that works on raw text without language-specific preprocessing.

## When to use SentencePiece

**Use SentencePiece when:**
- Building multilingual models (no language-specific rules)
- Working with CJK languages (Chinese, Japanese, Korean)
- Need reproducible tokenization (deterministic vocabulary)
- Want to train on raw text (no pre-tokenization needed)
- Require lightweight deployment (6MB memory, 50k sentences/sec)

**Performance**:
- **Speed**: 50,000 sentences/sec
- **Memory**: ~6MB for loaded model
- **Languages**: All (language-independent)

**Use alternatives instead**:
- **HuggingFace Tokenizers**: Faster tr

via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有