AI startup gross margins run well below software margins because inference costs grow with every task a customer runs. ICONIQ's State of AI 2026 report puts the average AI product's gross margin at 45% in 2025, projected to reach 53% in 2026 and 59% in 2027, against the 60% to 80% or more that 1752vc's guide cites from a16z for traditional software. Most of that gap is the model bill, and the bill has two parts: the price of each token and the number of tokens each task sends.
This note covers why margins compress as agentic usage grows, how to measure both cost dimensions, a worked example from our cost-per-task calculator, what a trained model changes, and the cases where it fails to pay off. It draws on our own token measurements and two design-partner workloads, anonymised.
Why do AI gross margins compress as agentic usage grows?
In software, the thousandth customer costs almost nothing to serve. In an AI product, every task the customer runs sends tokens to a model, and you pay for each one. As 1752vc puts it: "In AI, the thousandth customer sends you a bill."
Agentic usage makes that bill grow faster than revenue, for three reasons.
- Revenue is fixed per seat while cost scales per task. ICONIQ reports that subscriptions are still the most common pricing model as of its Q2 2026 survey. A customer who loves the product runs more tasks for the same monthly fee.
- Each task sends more tokens. An agent resends its history on every step, so a twelve-step run costs far more than twice a six-step run. Our guide to cost per task works through that arithmetic.
- The harness adds tokens before the work starts. We measured Claude Code sending 36,055 tokens in its default setup, in an empty folder, before the task itself (measured 6 October 2026). Tool definitions, instructions and context like these travel with every request. Caching discounts most of them on price, and they still fill the context window and add latency on every step.
1752vc's worked example shows the result. A contract-review agent sold at $200 per seat a month, at Anthropic's September 2026 list prices, spends $108 of each average seat's revenue on inference and runs at a 36% gross margin. A power user running 1,500 tasks a month costs $560 to serve against the same $200, a margin of minus 180%.
ICONIQ's survey also shows where the pressure is going. Two thirds of respondents report better per-query unit economics, helped by inference cost management and model routing, and "as products scale, talent's share of cost falls while inference rises." Consumption-based pricing rose from 35% to 42% in six months, as companies tie revenue to the usage that drives their cost.
What are the two dimensions of inference cost?
Cost per task is the input tokens times the input price, plus the output tokens times the output price. Reasoning models bill their hidden thinking as output tokens, so those count too. Every lever on the bill moves one of two numbers.
| Dimension | What sets it | Typical levers |
|---|---|---|
| Price per token | The model and provider you call | Routing, batch pricing, a cheaper model |
| Tokens per task | What each request carries | Shorter prompts, caching, training |
Most cost work goes into the first dimension. The second can be larger, because a production prompt carries the same instructions, protocol, examples and guardrails on every request. Those repeated tokens cost the same per token as the data that changes.
Two design-partner workloads show how large the repeated share gets:
- A consumer chat app sent about 37,000 input tokens per message. The prompt carried a persona and guardrail brief that repeats on every message. The build we modelled for it trains that brief into the model and sends a structured prompt of about 1,000 tokens, 37 times fewer input tokens per message.
- A design partner that runs simulated test sessions sends the same fixed protocol with every session. We modelled tokens per session falling from roughly 13,000 to 37,000 down to about 8,000 once the protocol is trained in, depending on the protocol's length. That is between about a third and three quarters fewer tokens.
The chat app's 37,000 tokens per message is the partner's own figure; the 1,000-token prompt and the per-session figures are modelled from each partner's prompt structure. The real cut depends on how much of each prompt repeats, which is why we measure it from production traffic before putting a number on it.
Prompt caching already discounts repeated input. As of October 2026, OpenAI and Anthropic bill cached input at a tenth to a twentieth of the standard input rate on the models in our calculator. Caching leaves output and reasoning tokens at full price, and the cached tokens still fill the context window and add latency.
A worked example from the cost-per-task calculator
Here are the calculator's default inputs, at OpenAI's list prices for GPT-6.1 Sol as of 28 October 2026: $2.00 per million input tokens, $0.10 cached and $10.00 per million output tokens, from OpenAI's pricing page.
| Input | Value |
|---|---|
| Tasks per month | 100,000 |
| Repeated instructions per prompt | 4,000 tokens |
| Per-request data | 500 tokens |
| Answer | 300 tokens |
| Hidden reasoning | 500 tokens |
The trained model keeps 500 tokens of instructions and produces no hidden reasoning. Its price per token is set to the frontier rate, so the whole difference comes from tokens per task.
| Setup | Tokens per task | Cost per task | Monthly cost |
|---|---|---|---|
| Frontier model | 5,300 | $0.0170 | $1,700 |
| Frontier, instructions cached | 5,300 | $0.0094 | $940 |
| Trained model, same token price | 1,300 | $0.0050 | $500 |
| Trained model, half the token price | 1,300 | $0.0025 | $250 |
At the same price per token, the trained model costs about 71% less than the uncached frontier run and about 47% less than the cached one. A lower price per token compounds on top: at half the frontier rate the task costs $0.0025. The last row is illustrative; the calculator lets you enter your own trained-model rate.
Now put the task inside a product. Suppose it is sold in a $40 monthly seat, and count inference only, before hosting, retrieval and support.
| Setup | Average seat, 1,000 tasks | Power user, 4,000 tasks |
|---|---|---|
| Frontier model | 58% margin | Minus 70% |
| Frontier, instructions cached | 77% margin | 6% margin |
| Trained model, same token price | 88% margin | 50% margin |
The average seat looks healthy on every row. The power user is where the margin goes, and that customer grows as agentic usage grows. Run your own numbers in the cost-per-task calculator, which splits any saving into its price effect and its volume effect.
What does a trained model change?
A model trained for one task changes both dimensions at once.
Tokens per task fall. The instructions, protocol and examples repeated in every prompt are trained into the model, so they stop being sent. A model trained on the task also answers directly, without hidden reasoning tokens or padding. The per-request data and the answer itself stay.
Price per token can fall. A model trained for one job carries only the capability that job needs, so it costs less to serve. In our QMSum study, a model trained for one meeting-summarization task had an estimated serving cost per query 99% below Claude Opus 5's and 96% below GPT-5.6 Luna's, and answered 23 times faster than Claude Opus 5. Those figures use September 2026 prices and exclude training cost.
Cost per task becomes more predictable. A trained model changes only when you approve a new version, so a provider's model release leaves the unit cost of the traffic it serves unchanged. Volume still drives the bill. Margin planning gets a stable cost per task to plan against.
The work worth starting with is the repeated, checkable kind. In OpenRouter's public data for the 30 days to 12 October 2026, classification, extraction and transformation took 20.8% of classified model spend, at 1.50 to 1.84 times the marketplace's average price per token, with 75% to 87% of the top ten models' spend going to closed frontier models. The outputs are labels and records you can check against a schema, which is where a trained model is easiest to evaluate.
This is the work Ertas does with the adaptive model endpoint: one API endpoint for one of your tasks. Frontier models answer at the start, a model trained on your requests and your team's feedback takes over once it passes your evals, and hard cases stay on frontier models. It runs Ertas-managed, in your own cloud or on-premise. The endpoint is currently invitation only, and we are developing a more streamlined version for wider access. For the break-even arithmetic, see when a custom model beats a frontier API on cost.
When does a trained model fail to pay off?
Training is a fixed cost, and the saving scales with volume and with the repeated share of each prompt. Four situations leave too little to save.
| Situation | Why the saving is small |
|---|---|
| A few thousand tasks a month | The build and evaluation outweigh the bill |
| Prompts mostly per-request data | Little repeated context to train away |
| Caching already on, no reasoning tokens | The input saving has mostly been taken |
| The task changes every week | Each change needs a new training round |
The second case is the easiest to miss. Take a task that sends 8,000 tokens of the customer's own document and 200 tokens of instructions, with a 300-token answer and no hidden reasoning, at the GPT-6.1 Sol rates above. Training away all 200 instruction tokens saves about 2% per task. The case for a trained model then rests on price per token alone, plus latency and consistency.
Start with a measurement. Take a week of production traffic for your highest-volume task, count the repeated tokens, the per-request tokens, the output and the reasoning, and put them in the calculator. If the repeated share is large and the volume is high, a trained model moves your gross margin. If either is small, routing and caching will do more for less.
Frequently Asked Questions
What is a good gross margin for an AI startup?
ICONIQ's State of AI 2026 report puts the average AI product at 45% in 2025, projected to reach 53% in 2026 and 59% in 2027, below the 60% to 80% or more typical of traditional software. Investors tend to weigh the trend and a credible path up as heavily as today's figure.
Why are AI gross margins lower than SaaS margins?
Every task an AI product runs costs model inference, so the cost of serving a customer grows with their usage. In SaaS one more user barely moves the hosting bill. Agentic products widen the gap, because each task makes several API calls and resends its context on every step.
How much does LLM fine-tuning cost?
As of 28 October 2026, OpenAI bills supervised fine-tuning at $5 per million training tokens for GPT-4.1 mini and $25 for GPT-4.1 and GPT-4o, and closes new training jobs on 6 January 2027. Training open models yourself in Ertas Studio is free to start, with plans from $10 a month as of October 2026. A managed build is a fixed fee scoped after a free audit.
Which LLM is best for fine-tuning?
The smallest open-weight model that holds your quality bar on a held-out evaluation set, with a licence that allows commercial use. A model trained for one task can be much smaller than a general model and still match it on that task, and its size sets most of the running cost.
Does prompt caching fix AI inference margins?
It helps on input. As of October 2026, cached input costs a tenth to a twentieth of the standard rate at OpenAI and Anthropic. Output and reasoning tokens stay at full price, and cached tokens still fill the context window, so caching narrows the tokens-per-task problem without removing it.
The Short Version
AI gross margins averaged 45% in 2025, with 53% projected for 2026, because inference scales with usage, and agentic products send more tokens per task every year. The bill has two dimensions: price per token and tokens per task. A model trained for a high-volume task cuts both and makes cost per task more predictable, provided the volume is high and most of each prompt repeats. For the closed labs' retreat from fine-tuning and what it means for this choice, read is fine-tuning dead?. If you fine-tuned on OpenAI, our migration guide covers the January 2027 deadlines.
Sources, read 28 October 2026: ICONIQ, State of AI 2026; 1752vc, AI Gross Margins: Inference Costs and Pricing for Startups; OpenAI API pricing. Our measurements: the QMSum study (arXiv 2609.25028), the OpenRouter structured-task report, and harness token counts measured 6 October 2026.