The On-Device AI Economics Cheat Sheet
API costs are 3-5x what most builders estimate. Here are the real numbers — and why fine-tuned on-device models cost $0 per inference.
The Real Cost of API-Based AI
Most builders calculate API cost as: tokens in + tokens out. The actual formula is far more expensive.
The True Cost Formula
actual_cost = (prompt + completion + system_prompt + rag_context + retries) x calls/month
System prompts, RAG context, retries, and conversation history multiply your base token cost by 3-5x. A call you estimated at $0.002 actually costs $0.006-$0.01.
System prompts
Every request includes your system prompt tokens. At 500 tokens per call, that's 30-60% added cost most people don't account for.
RAG context injection
Retrieved documents add 1,000-4,000 tokens per request. Your actual cost per call is 2-5x the naive estimate.
Retries and fallbacks
Rate limits, timeouts, and quality retries mean 10-30% of your API budget is spent on failed or repeated requests.
Conversation history
Multi-turn conversations compound token usage exponentially. A 10-turn chat costs 5-10x the first message.
API vs. On-Device at Scale
Once you fine-tune a model, every inference runs on the user's hardware for free.
| Active Users | API Cost / mo | On-Device Cost / mo | Annual Savings |
|---|---|---|---|
| 100 | $50 – $120 | $0 | $600 – $1,440 / yr |
| 1,000 | $500 – $1,200 | $0 | $6,000 – $14,400 / yr |
| 10,000 | $5,000 – $12,000 | $0 | $60,000 – $144,000 / yr |
| 100,000 | $50,000 – $120,000 | $0 | $600K – $1.44M / yr |
Estimates based on GPT-4o-mini pricing at ~50 tokens/request with system prompt, RAG context, and retry overhead.
Break-Even: 1-2 Months
Fine-tuning has a one-time cost. After that, every inference is free. Most builders break even within weeks.
One-time fine-tuning cost on Ertas (depending on model size and dataset)
Typical break-even vs API costs at just 100 daily active users
Per-inference cost forever after — runs on the user's own hardware
Beyond Cost: Why On-Device Wins
$0 per inference
After one-time fine-tuning, every prediction is free. No per-token billing, no usage spikes, no budget surprises.
Works offline
No network dependency. Your app works on planes, in basements, in rural areas, in air-gapped environments.
Faster response
Local inference eliminates network round-trip. Sub-100ms for classification tasks vs 200-800ms via API.
Data stays on device
User data never leaves their hardware. GDPR, HIPAA, and user trust handled by architecture, not policy.
Ready to Eliminate Per-Inference Costs?
Fine-tune your first on-device model on Ertas. No ML expertise needed — upload your data, train in the cloud, download a GGUF, and run locally forever.