A controlled ablation study of 19 ~63M-parameter GPT-style language models, each varying exactly one architectural axis (attention mechanism, positional encoding, optimizer, or FFN type) from a shared baseline. Trained on 1B tokens of fineweb-edu, evaluated on perplexity, zero-shot benchmarks (LAMBADA, HellaSwag, PIQA, SciQ, ARC-Easy, WinoGrande, BLiMP), inference throughput, and FLOPs/VRAM.
Paper (with full methodology, ablation pairs, and analysis): paper/main.pdf.
Trained checkpoints: see Hugging Face below.
| Axis | Values covered |
|---|---|
| Attention | MHA, MQA, GQA, MLA, Differential Attention, KDA (linear attention) |
| Position | learned, RoPE, NoPE |
| Optimizer | AdamW, Muon |
| FFN | dense, MoE, shared-expert MoE |
See configs/ for the full per-model config table (one small override yaml per
model, all inheriting from configs/base.yaml) and paper/main.pdf for the
comparison pairs (e.g. M01→M06 isolates the RoPE effect on MHA; M11→M19
isolates the KDA vs. softmax-attention effect with RoPE+Muon held fixed).
uv sync # installs deps (CUDA 12.8 / cu128 torch wheels)
cp .env.example .env # fill in HUGGING_FACE_TOKEN
uv run python -m dataset.prepare # -> data/fineweb-edu/ (400M train + 5M val tokens)Requires an RTX 5060 or other Blackwell (sm_120) GPU for the pinned cu128
PyTorch wheels; adjust pyproject.toml's [tool.uv.sources] for other GPUs.
uv run python scripts/train.py --config configs/m01_baseline.yaml
bash scripts/run_all.sh # full matrix, phase-orderedResume with --resume results/<model_id>/step_N.pt (resumes from the last
completed step; scripts/train.py prunes checkpoints to the 2 most recent
step_*.pt and prints the exact resume command on interrupt or crash). Use
--smoke for a sandboxed test run that never touches final.pt.
uv run python scripts/evaluate.py --checkpoint results/m01/final.pt \
--config configs/m01_baseline.yaml --mode all # perplexity + throughput + generation
uv run python scripts/benchmark.py --checkpoint results/m01/final.pt --device cpu # zero-shot suiteuv run python analysis/paper_figures.py --results-dir results/ --output-dir paper/figures/dataset/ data pipeline (fineweb-edu -> tokenized .bin shards)
models/ model code; models/shared/ has model-agnostic building blocks,
models/m01-m19/ each add one attention.py + build_model(config)
configs/ configs/base.yaml + one small override yaml per model
training/ config loader, training loop, Muon optimizer
scripts/ train.py, evaluate.py, benchmark.py, run_all.sh, export/upload to HF
analysis/ figure and table generation for the paper
paper/ LaTeX source, compiled PDF, figures
results/ per-model eval_results.json / benchmark_results.json / mlflow_run_id.txt
(checkpoints themselves are .gitignore'd — see Hugging Face below)
Trained checkpoints (weights only, optimizer state stripped) are published at
https://huggingface.co/B4K2xx/Codemind-study, one subfolder per model ID with
model.safetensors + config.json + eval/benchmark results.
MIT


