Skip to content

About

Controlled ablation of 24 small GPT-style models (60M params, 1B tokens): MHA/MQA/GQA/MLA, RoPE, differential attention, MoE, and Muon vs AdamW.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

codemindv2 — SLM Attention / Position / Optimizer Ablation Study

A controlled ablation study of 19 ~63M-parameter GPT-style language models, each varying exactly one architectural axis (attention mechanism, positional encoding, optimizer, or FFN type) from a shared baseline. Trained on 1B tokens of fineweb-edu, evaluated on perplexity, zero-shot benchmarks (LAMBADA, HellaSwag, PIQA, SciQ, ARC-Easy, WinoGrande, BLiMP), inference throughput, and FLOPs/VRAM.

Paper (with full methodology, ablation pairs, and analysis): paper/main.pdf. Trained checkpoints: see Hugging Face below.

Training and validation loss curves for all 19 models

Zero-shot capability vs. validation perplexity, showing PPL is not a reliable proxy for capability Perplexity vs. KV-cached decode speed Pareto trade-off

What's varied

Axis Values covered
Attention MHA, MQA, GQA, MLA, Differential Attention, KDA (linear attention)
Position learned, RoPE, NoPE
Optimizer AdamW, Muon
FFN dense, MoE, shared-expert MoE

See configs/ for the full per-model config table (one small override yaml per model, all inheriting from configs/base.yaml) and paper/main.pdf for the comparison pairs (e.g. M01→M06 isolates the RoPE effect on MHA; M11→M19 isolates the KDA vs. softmax-attention effect with RoPE+Muon held fixed).

Setup

uv sync                                   # installs deps (CUDA 12.8 / cu128 torch wheels)
cp .env.example .env                      # fill in HUGGING_FACE_TOKEN
uv run python -m dataset.prepare          # -> data/fineweb-edu/ (400M train + 5M val tokens)

Requires an RTX 5060 or other Blackwell (sm_120) GPU for the pinned cu128 PyTorch wheels; adjust pyproject.toml's [tool.uv.sources] for other GPUs.

Training

uv run python scripts/train.py --config configs/m01_baseline.yaml
bash scripts/run_all.sh                   # full matrix, phase-ordered

Resume with --resume results/<model_id>/step_N.pt (resumes from the last completed step; scripts/train.py prunes checkpoints to the 2 most recent step_*.pt and prints the exact resume command on interrupt or crash). Use --smoke for a sandboxed test run that never touches final.pt.

Evaluation

uv run python scripts/evaluate.py --checkpoint results/m01/final.pt \
    --config configs/m01_baseline.yaml --mode all      # perplexity + throughput + generation

uv run python scripts/benchmark.py --checkpoint results/m01/final.pt --device cpu  # zero-shot suite

Figures

uv run python analysis/paper_figures.py --results-dir results/ --output-dir paper/figures/

Repository layout

dataset/     data pipeline (fineweb-edu -> tokenized .bin shards)
models/      model code; models/shared/ has model-agnostic building blocks,
             models/m01-m19/ each add one attention.py + build_model(config)
configs/     configs/base.yaml + one small override yaml per model
training/    config loader, training loop, Muon optimizer
scripts/     train.py, evaluate.py, benchmark.py, run_all.sh, export/upload to HF
analysis/    figure and table generation for the paper
paper/       LaTeX source, compiled PDF, figures
results/     per-model eval_results.json / benchmark_results.json / mlflow_run_id.txt
             (checkpoints themselves are .gitignore'd — see Hugging Face below)

Hugging Face

Trained checkpoints (weights only, optimizer state stripped) are published at https://huggingface.co/B4K2xx/Codemind-study, one subfolder per model ID with model.safetensors + config.json + eval/benchmark results.

License

MIT

About

Controlled ablation of 24 small GPT-style models (60M params, 1B tokens): MHA/MQA/GQA/MLA, RoPE, differential attention, MoE, and Muon vs AdamW.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages