Small Language Models for AI Agents: A Response

Small language models for AI agents are compact models, under about 10 billion parameters in NVIDIA's working definition, that take over the narrow, repeated steps an agent runs: choosing a tool, extracting fields, classifying a request, formatting a reply. NVIDIA Research argues in "Small Language Models are the Future of Agentic AI" that these models are capable enough for most of those steps, better suited to them and cheaper to run. We agree with most of the paper, and our own measurements support its core claim on a defined task. Our qualifications are about engineering: how small a model each step needs, which optimisations hold the quality bar, and the volume at which running your own model beats a frontier API.

This post sets out the paper's argument as written, where our evidence agrees, where we would add detail, which open model families suit agent work as of October 2026, and how small language models fit enterprise agents.

What are small language models for AI agents?

A small language model (SLM) for an AI agent is a model small enough to run on an ordinary device or a single modest server, trained or tuned for the narrow jobs an agent hands to a language model. The NVIDIA paper defines an SLM by where it runs: "a LM that can fit onto a common consumer electronic device and perform inference with latency sufficiently low to be practical when serving the agentic requests of one user." In parameters, the authors say that "as of 2025, we would be comfortable with considering most models below 10bn parameters in size to be SLMs."

An agent sends many requests to a language model for each task it completes. Some API calls need open-ended reasoning or conversation. Many are narrow: pick a tool and fill its arguments, pull fields from a document, classify a request, summarise one record, return JSON in a fixed schema. Those narrow API calls are where a small model fits.

"Small language model" is the paper's term and the one people search. In our own work we describe the result by what it does: a task-specific model, trained for one job and right-sized to the smallest model that holds that job's quality bar.

What does NVIDIA's paper argue?

The paper is a position paper by Peter Belcak and seven colleagues at NVIDIA Research and Georgia Tech (arXiv 2506.02153, first posted 2 June 2025, version 3 posted 22 September 2026). Its central claim, in its own words, is that SLMs are "sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI."

It rests that claim on three views:

In full, the paper says SLMs are "principally sufficiently powerful to handle language modeling errands of agentic applications", "inherently more operationally suitable for use in agentic systems than LLMs", and "necessarily more economical for the vast majority of LM uses in agentic systems". For capability it cites published results for Phi, Nemotron-H, SmolLM2, Hymba, DeepSeek-R1-Distill, RETRO and xLAM-2. For economics it puts serving a 7B model at 10 to 30 times cheaper in latency, energy and FLOPs than a 70B to 175B model, and fine-tuning a small model at a few GPU-hours.

Four points carry most of the agent-specific argument:

• Agents use a narrow slice of the model. The paper calls an agent "a heavily instructed and externally choreographed gateway to a language model", and argues that a small model "appropriately fine-tuned for the selected prompts would suffice."
• Agents need one output format, every time. Tool calls and parsed outputs must match a strict schema. A model trained on a single format suits that job better than a generalist that occasionally answers in a different one.
• Agents can mix models. Each request from the agent's code can go to a different model, so the paper favours heterogeneous systems "where SLMs are used by default and LLMs are invoked selectively and sparingly."
• Agent traffic is training data. Each logged model request "is itself a natural source of data for future improvement", which makes training an expert small model "a natural step in agent deployment".

The paper closes with a six-step conversion algorithm from LLMs to SLMs: log the agent's model requests securely, curate the data and strip personal and sensitive information, cluster the requests into recurring tasks, select candidate small models, fine-tune them on each task, and retrain periodically on new data. Its appendix estimates the share of LLM queries in three open-source agents that "could be reliably handled by appropriately specialized SLMs": about 60% for MetaGPT, 40% for Open Operator and 70% for Cradle.

The paper also answers its critics. It accepts the evidence that a large model beats a small one of the same generation on general language understanding, and replies that a small model "can be easily fine-tuned" for a given task to the reliability needed. On economics it is more cautious. It calls the counter-view, that centralised LLM serving stays cheaper through economy of scale, "a valid view, with the exact economical considerations being highly case-specific", and says "the jury is still out".

Where does our evidence agree?

On a defined task, our measurements support the paper's capability and economics views: a model trained for the job held the quality bar at a fraction of a general model's size, and the cost case stands on two dimensions.

Trained models outscored hosted proprietary models on one benchmark. Our QMSum study (arXiv 2609.25028) scored query-focused meeting summarisation under one implementation on the full 281-query test split. Trained models of 406M and 1.2B parameters scored 36.33 and 35.41 ROUGE-1. Five hosted proprietary models, run zero-shot under one concise prompt, scored 27.89 to 32.43. The limits matter: the hosted models wrote longer answers, which contributes to the gap on a reference-overlap metric; no human or factuality evaluation was run; and the conclusions are limited to QMSum and automatic metrics. The strongest published QMSum systems report higher scores in their own papers, up to 38.82 ROUGE-1.

Training mattered more than size. Within one 1.2B base model, fine-tuning on the same retrieved transcript spans the model reads at inference added 5.29 ROUGE-1 (95% interval +4.02 to +6.56). A 406M model trained the same way scored 36.33 against the 1.2B system's 35.41, a difference QMSum cannot statistically separate, with less than half the peak inference memory. On one task, that is the paper's reply to its critics: a small model trained for the task reaches the reliability the task needs.

The spend sits on tasks agents repeat. In OpenRouter's public data for the 30 days to 12 October 2026, classification, extraction and transformation took 20.8% of classified model spend, and closed frontier models took 75% to 87% of the top ten models' spend on those tasks. These are the narrow, checkable jobs the paper says agents hand to a language model. The data describes marketplace activity: it shows where to look, and each buyer still measures its own saving.

Cost falls on two dimensions. The paper's economics rest on price per token, since a smaller model costs less to serve. A trained model also sends fewer tokens per task. The paper describes agents held together by "tediously written prompts and meticulously orchestrated context management"; once the instructions, protocol and examples repeated in those prompts are trained into the model, they stop being sent, and only the per-request data remains. A trained model can also answer directly, without long reasoning traces. The size of this second saving depends on how much of each prompt is repeated boilerplate and whether your current model bills reasoning tokens. Prompt caching already discounts repeated input on the frontier APIs, so against a well-cached setup the input saving is smaller. Our open-weight vs frontier comparison works through both dimensions at list prices.

Where would we qualify the paper?

The paper argues a direction. A team putting a small model into a production agent still has three engineering and economics questions to answer, and each is settled by measurement on its own task.

How small the model should be. The paper's definition sets a ceiling: anything that fits a consumer device. Inside that ceiling the right size varies by step. In our QMSum study a 406M model scored within the benchmark's resolution of a 1.2B one, while the paper's own case studies leave planning, debugging and error recovery with larger models. Size is an output of evaluation: train candidates at two or three sizes on your examples and keep the smallest one that holds your quality bar.

Which optimisations hold the bar. The paper names fine-tuning, inference-time compute, tool use and routing. Our results add that the training data has to match the agent's real inputs. A released 406M summariser lost 6.30 ROUGE-1 when its input changed from long transcripts to the retrieved spans our pipeline feeds it, and fine-tuning it on the new inputs recovered the loss. An agent's inputs shift whenever its retrieval, tools or prompts change, so a specialist is retrained on the inputs it now sees. Retrieval, quantisation and routing each move cost and quality, and each is checked against the same evaluation set.

Where the cost crossover sits. The paper leaves this open, and it is the right question. A small model costs less per token to serve, but a dedicated model also carries training, hosting, load balancing and the people to run it, which the paper accepts as "highly case-specific". At low volume a frontier API often wins on total cost. The frontier labs also sell budget tiers that can undercut a hosted open model per token, and against those the case rests on fewer tokens per task and on quality. The crossover is a volume, and you can estimate it from your current token counts and prices before building anything, using the cost-per-task calculator.

What to measure. The paper notes that small-model development leans on the same generalist benchmarks as LLM work, and names that as a barrier to adoption. A leaderboard score says little about one agent step. Build an evaluation set from the agent's own requests with expected outputs, and score every candidate, small or large, on it. Our guide to domain-specific LLM evaluation covers how, with the QMSum study's protocol as the worked example.

What are the best small language models for AI agents?

The best small language model for an agent step is the smallest one that holds your quality bar after training on your own examples, under a licence that allows your use. As starting candidates, these open families publish sizes under 10B and state agent features on their own model cards, read 30 October 2026:

New versions arrive often (Granite 4.2's card gives a release date of 25 August 2026), so check the date on any list, this one included. Licences differ too. Most of these families use Apache 2.0, while NVIDIA's model carries its own open model licence, so read the terms before commercial use. For licensing and sizing from an enterprise angle, see our enterprise small model comparison.

Three checks matter more than the model card:

• Format. Does the model produce your exact tool-call or JSON format after training on your examples?
• Reasoning. Does it hold your quality bar with reasoning switched off? Several of these models think before answering by default, and reasoning tokens add to the cost of every task.
• Licence. Does the licence allow your use, including serving the model inside a commercial product?

Are small language models ready for enterprise agents?

Small language models for enterprise agents are ready for narrow, repeated, checkable steps, once each one is trained and evaluated on the enterprise's own examples. They bring three practical gains: a lower cost per task, latency low enough for agents that chain many steps, and a choice of where the model runs, since a small model fits in the enterprise's own cloud or on-premise as well as on a managed service. The paper's conversion steps carry enterprise duties too: encrypted logging with role-based access, and removing personal and sensitive data before any of it becomes training data.

Enterprise agents keep a place for large models. Open-ended conversation, planning across many steps and unusual requests can stay on a frontier model while trained specialists take the volume, which is the heterogeneous design the paper recommends. For running small models on-premise, see small language models for enterprise. For routing between small and large models, see SLM-first architecture.

How do you move an AI agent from an LLM to small language models?

Start with one repeated step, log it, and switch it over once a trained model holds the bar on your evaluation:

1. Log the agent's model requests. Record inputs, outputs and tool calls for the steps that run behind the scenes, with access controls in place and personal data removed.
2. Find the repeated steps. Group the requests into recurring tasks such as tool selection, extraction, classification or summarising one document type.
3. Pick the highest-volume checkable task first. Volume sets the saving, and a checkable output sets a clear quality bar.
4. Build the evaluation set before training. Collect expected outputs for real requests and score the current model and every candidate the same way.
5. Train two or three sizes and keep the smallest that holds the bar.
6. Route that step to the specialist. Keep the large model for the rest, and retrain as the step's inputs change.

This follows the paper's algorithm, with the evaluation set moved to the front. For when one task earns its own model, see foundation models vs task-specific models.

Ertas, an applied AI lab, runs the adaptive model endpoint for this: a model trained on your agents' work, served through an OpenAI-compatible API, with an evaluation gate on every release. New versions train on your agents' traffic in the background, are scored against your evaluations, and go live only with your sign-off. If a new version falls short, the current model stays in service. It runs Ertas-managed, in your own cloud or on-premise, and your data stays private. The endpoint is currently invitation only, and we are developing a more streamlined version for wider access. For how the loop works, see continual learning for AI agents.

Frequently Asked Questions

What counts as a small language model?

NVIDIA's paper defines one by where it runs: a model that fits on a common consumer device and responds fast enough to serve one user's agent. In parameters, its authors put the line at about 10 billion as of 2025. Vendors use the label loosely, so check what "small" means before comparing two claims.

Can small language models do tool calling?

Yes. Qwen3.5, Gemma 4, Ministral 3, Granite 4.2 and SmolLM3 all state tool or function calling support on their model cards as of October 2026, and the NVIDIA paper cites Salesforce's xLAM-2-8B as surpassing GPT-4o and Claude 3.5 on tool calling. Test the exact format your agent uses, and train on your own tool schemas if the base model drifts from it.

Will small language models replace LLMs in AI agents?

For many steps, according to the paper, which estimates that 40% to 70% of LLM queries in three open-source agents could move to specialised small models. Large models keep the open-ended work: conversation, planning and recovering from unexpected states. The paper's recommended design mixes the two.

What is an example of a small language model?

Qwen3.5-4B, Gemma 4 E4B, Ministral 3 8B, Granite 4.2 3B and SmolLM3-3B are small language models with open weights. A task-specific example is the 406M meeting summariser from our QMSum study, trained for one job and published with the paper.

Are small language models cheaper for AI agents?

Usually per token: the paper puts serving a 7B model at 10 to 30 times cheaper in latency, energy and compute than a 70B to 175B model. Per task, a trained model also sends fewer tokens, because the instructions repeated in every prompt are trained into it. Total cost depends on volume, since a dedicated model carries training and hosting costs. The cost-per-task calculator finds the crossover for your numbers.

The Short Version

NVIDIA's paper argues that most of what an agent asks of a language model is narrow and repeated, and that small models trained for those steps are capable enough, better suited and cheaper. On a defined task our results agree: a trained 406M model scored within the benchmark's resolution of a model three times its size, and both trained models outscored five hosted proprietary models on ROUGE-1, within the limits above. The open questions are engineering and economics: the right size for each step, the optimisations that hold the bar, and the volume at which your own model beats an API. Your own evaluation set answers each one.

For the wider comparison, see open-weight vs frontier models.

---

Sources, read 30 October 2026: Belcak et al., "Small Language Models are the Future of Agentic AI" (arXiv 2506.02153v3, 22 September 2026) and its NVIDIA Research page; model cards for Qwen3.5-4B, Gemma 4 E4B, Ministral 3 8B, Granite 4.2 8B, Nemotron 3 Nano 4B and SmolLM3-3B. Our research: the QMSum study and the OpenRouter structured-task report.