Small Language Models

For two years the story was “bigger is better.” The 2024– 2026 counter-narrative: 1–8B parameter models, carefully trained on carefully chosen data, can match models five to fifty times their size on specific tasks.

§ 00 · WHY SMALL MODELS?The push for compactness

From 2020 to 2023, the dominant trajectory was scale. GPT-3 at 175B, PaLM at 540B, GPT-4 estimated at over a trillion (with sparsity). Bigger models won benchmarks; the rule looked simple — more parameters, more data, more compute, better model.

The rule held in absolute terms. What changed: the cost of running these models in production. A 70B model costs roughly 70× a 1B model in inference compute. For real applications — at scale, with real latency budgets — that math is brutal. Whole product categories stopped being economical.

Small Language ModelsSmall Language Models. Language models in the 1–10B parameter range, deliberately optimized for deployability. The category emerged in 2023–2024 as researchers demonstrated that data quality and training recipe could let small models match much larger ones on focused tasks. are the response: build the smallest model that does the job. The bet — vindicated repeatedly in 2024 — is that a lotof useful tasks don’t actually need 100B parameters of model capacity.

§ 01 · WHAT MAKES AN SLM VIABLERecipe, not just size

Three things separate a useful SLM from a dumb-and-small toy:

• Better data.The Phi line from Microsoft argued that “textbook-quality” data — carefully filtered, synthetic where useful, focused on educational content — produces more capable small models than raw web crawl. The same number of tokens does more work when the tokens are denser.
• Better distillation.Train the small model to imitate a large one’s outputs. The large model effectively curates and labels the training data for the small one. Used throughout Gemma and Llama-3.2 small variants.
• Better architecture choices. Grouped-query attention, sliding-window attention, and improved tokenizers all let small models punch above their weight on inference cost without losing quality.

§ 02 · PHI, GEMMA, LLAMA-3.2 SMALL, AND FRIENDSThe public SLM lineup

• Phi-3 / Phi-4.Microsoft’s line. Curated/synthetic data, strong on reasoning for its size. 3.8B and 14B versions widely used.
• Gemma 2 / Gemma 3.Google’s open small models. Strong general capabilities and a permissive license.
• Llama-3.2 1B/3B.Meta’s small variants, distilled from Llama-3.1 70B. Edge-device-friendly.
• Qwen 2.5 / Qwen 3 small.Alibaba’s line. The Qwen 3 family includes especially strong small models (see the Qwen 3 lesson).
• SmolLM, TinyLlama, OLMo small. Research-oriented; fully open training recipes.

Sizes commonly seen: 1B, 1.5B, 3B, 7B, 8B. The 7–8B class has become the sweet-spot “laptop-runnable” tier — Q4-quantized, a 7B model fits in 4–5 GB VRAM and runs at usable speeds on consumer GPUs.

§ 03 · WHERE SLMS WIN, WHERE THEY FOLDHonest task fit

Tasks SLMs handle well:

• Classification. Sentiment, intent, safety, topic.
• Extraction. Structured output from text. Phone numbers, addresses, entities.
• Routing / orchestration. Decide which path to take in a workflow.
• Summarization. Especially of bounded inputs.
• Embedding generation. Often produced by SLM-class encoders.
• Reasoning on narrow domains after fine-tuning. A 7B fine-tune on a domain often beats a 70B base model on that domain.

Where SLMs fold:

• Broad world knowledge.“What year was X founded?” — small models forget more, hallucinate more.
• Long, open-ended reasoning. Multi-hop math, novel problem solving, creative writing at high quality.
• Long contexts in practice. Even when an SLM advertises 128k context, it tends to use it worse than a frontier model would.
• Adversarial inputs. Jailbreaks, edge cases, ambiguous instructions. Smaller models are more brittle.

§ 04 · THE PICTURE FOR 2026Both, not one

The likely steady state isn’t “SLMs replace LLMs” — it’s a tiered architecture:

• SLM as default.Most of the traffic in a real application is routine. Routing, classification, simple Q&A all go to a small fast model.
• Frontier model for escalation. Harder requests — long reasoning, novel domains, unbounded queries — escalate to a frontier model.
• Hybrid serving stacks. Tools like Llama-stack, OpenRouter, vLLM with adapter routing make the SLM/LLM split something you can reason about per-request.

§ 05 · TAKING THIS FORWARDWhere to look next

Two related lessons in this series: SFT vs RL (the training-recipe choices that affect SLM quality) and Qwen 3 (a specific SLM-friendly model family that exemplifies the modern recipe).

§ · GOING DEEPERWhy small models suddenly got good

The Phi line from Microsoft (Gunasekar et al. 2023, “Textbooks Are All You Need”) made the argument that careful curation of training data — heavily filtered, synthetic when appropriate, focused on educational content — produces small models that punch far above their parameter count. Phi-3 (Abdin et al. 2024) extended this to 3.8B-parameter models matching GPT-3.5 on many benchmarks. The lesson: data quality is a multiplier on parameter count, and at small scales it matters more than raw model size.

Two threads run downstream. On-device deployment: Apple Intelligence (Mehta et al. 2024) and Llama 3.2 1B/3B target phone-scale inference. Distillation(Hsieh et al. 2023): use a strong teacher to generate chain-of-thought training data for a small student, and the student matches the teacher on the target task at a fraction of cost. The economics of 2026 increasingly route routine queries through SLMs and reserve frontier models for what genuinely needs them.