Orchestra-Research/AI-Research-SKILLs✦ 精选0°

huggingface-tokenizers

仓库创建 2025年11月3日最近提交 1 个月前SkillHot 收录 20 天前

▸ 精选理由

性能与可定制性兼备，适合从研究到生产的分词需求与代理文本处理。

这个 Skill 做什么

Rust 实现的高性能分词器，支持 BPE/WordPiece/Unigram 与自训练词表。

基于 Rust 的超快分词器，支持 BPE、WordPiece 和 Unigram 算法，可以训练自定义词表并跟踪 token 到原文的对齐。适合大规模文本预处理或需要自训练 tokenizer 的场景，速度和工程特性都面向生产级别。能无缝接入 transformers，处理 padding/truncation、对齐等细节，便于上线部署。

▸ 展开 SKILL.md 英文原文

Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrates seamlessly with transformers. Use when you need high-performance tokenization or custom tokenizer training.

数据与抓取高速分词BPE/Unigram生产级通用

1.1w

Stars

817

Forks

仓库内 Skill

+704

7 日增星

安装 / 使用

给你的 Agent 一句话（通用）

帮我安装这个 skill：https://raw.githubusercontent.com/Orchestra-Research/AI-Research-SKILLs/main/02-tokenization/huggingface-tokenizers/SKILL.md

或 curl 直取 SKILL.md

curl -fsSL "https://raw.githubusercontent.com/Orchestra-Research/AI-Research-SKILLs/main/02-tokenization/huggingface-tokenizers/SKILL.md"

SKILL.MD 节选查看完整文件 ↗

# HuggingFace Tokenizers - Fast Tokenization for NLP

Fast, production-ready tokenizers with Rust performance and Python ease-of-use.

## When to use HuggingFace Tokenizers

**Use HuggingFace Tokenizers when:**
- Need extremely fast tokenization (<20s per GB of text)
- Training custom tokenizers from scratch
- Want alignment tracking (token → original text position)
- Building production NLP pipelines
- Need to tokenize large corpora efficiently

**Performance**:
- **Speed**: <20 seconds to tokenize 1GB on CPU
- **Implementation**: Rust core with Python/Node.js bindings
- **Efficiency**: 10-100× faster than pure Python implementations

**Use alternatives instead**:
- **SentencePiece**: Langu

via SKILL·HOT · 数据来自 GitHub 公开信息 · 原文版权归作者所有