training-llms-megatron

使用 Megatron-Core 在多GPU集群上高效训练 2B–462B 参数的语言模型。

用 Megatron-Core 在多 GPU 集群上训练 2B–462B 参数的语言模型，支持 tensor/pipeline/sequence/expert 等多种并行策略以最大化 GPU 利用率。适合需要训练 >1B 参数模型并追求最高 MFU（如在 H100 上）和生产级效率的场景。特点是面向大规模训练优化、生产就绪，能把复杂并行配置落地到稳定的训练流程。

# Megatron-Core - Large-Scale LLM Training

## Quick start

Megatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.

**Installation**:
```bash
# Docker (recommended)
docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:25.04-py3

# Or pip
pip install megatron-core
```

**Simple distributed training**:
```bash
# Train with 2 GPUs using data parallelism
torchrun --nproc_per_node=2 examples/run_simple_mcore_train_loop.py

# Or LLaMA-3 8B training
./examples/llama/train_llama3_8b_fp8.sh
```

## Common workflows

### Workflow 1: Train LLaMA-style model with 3D parallelism

Copy this checklist:

```
LLaMA Tra