# Small Fine-tuned Models are All You Need

**Author:** Stefan Webb  
**Published:** October 16, 2025  
**Source:** https://oumi.ai/blog/small-fine-tuned-models-are-all-you

---

## Core Thesis

Small fine-tuned foundation models can outperform large general-purpose ones (like GPT-4/GPT-5) on specialized tasks, with higher task-specific performance, faster inference, and lower cost. However, getting the details right requires technical expertise and intelligently designed infrastructure.

---

## Key Evidence

### Mid-2024 Empirical Study (Zhao et al., 2024)

- Researchers selected **31 tasks** across a wide range of domains.
- Fine-tuned **10 small base models** (<8B parameters: Llama, Mistral, Zephyr, Phi, Gemma) using **LoRA** on each task.
- **Main finding:** 6 of the 10 small models outperformed **GPT-4 on average** after fine-tuning. All 10 outperformed GPT-3.5-Turbo.
- Fine-tuning was done with rank-8 LoRA, 4-bit precision, 2,500 steps, batch size 16 — all on a single consumer-grade GPU (<24GB memory).

### Where Small Models Excel

- **Traditional NLP tasks (GLUE benchmark):** Largest improvement from fine-tuning and smallest gap to GPT-4.
- **Coding and math reasoning:** Initially lagged behind GPT-4 — but the base models used were pre-February 2024 and not pretrained for coding/reasoning.

### Where We Are Now (Late 2025)

- Newer base models like **Qwen3-4B-Instruct** have closed the gap on coding and reasoning performance.
- GPT-5 is stronger, but the open-source ecosystem has advanced dramatically.
- The author posits that repeating the study with late-2025 models would show small fine-tuned models closing the gap on coding and reasoning.

---

## Why Small Models Aren't Yet Ubiquitous

1. **Long development cycles** — earlier attempts at productionizing small fine-tuned models required months of custom development.
2. **Misconception about data** — belief that big data is required, but actually 1,000 carefully curated samples can suffice for successful fine-tuning.
3. **Catastrophic forgetting** — LoRA and parameter-efficient methods largely avoid this issue; RL-based methods prevent loss of generalization.

---

## Key Takeaways

- Small fine-tuned models <8B can match or exceed GPT-4 on domain-specific tasks.
- **LoRA** enables fine-tuning on a single consumer GPU.
- **1,000 high-quality samples** can be sufficient for strong fine-tuning results.
- The claim is nuanced — task type, base model choice, and fine-tuning methodology matter greatly.