How to Reduce LLM Token Costs with Small Language Models, Caching and RAG

Many organisations start with AI the same way: a large general-purpose model behind an API, a prompt that works, and a pilot everyone likes. Then usage grows, and so does the bill. Every customer message, document summary and support reply consumes tokens, and the largest models charge the most for each one.

This guide explains how to reduce LLM token cost without simply accepting worse results: where tokens and costs actually come from, when a small language model (SLM) is the right tool, and which techniques — quantisation, caching, routing, retrieval and output limits — make the biggest difference. It ends with an evaluation framework, because a cost reduction is only real if quality holds up.

SLM vs LLM: What Is the Difference?

A large language model is a general-purpose model with a very large number of parameters, able to handle a wide range of tasks. A small language model has far fewer parameters and is usually fine-tuned for a narrower set of tasks. There is no official threshold between the two; what matters is the trade-off.

Typical SLM-friendly tasks include classifying enquiries, extracting fields from forms, routing support tickets, detecting language, short summaries and answering questions over a fixed knowledge base.

Token Economics: Where the Money Goes

API-based models are usually priced per token, with separate rates for input (the prompt) and output (the response). Output tokens are often priced higher than input tokens. A simple estimate for one request is:

Cost per request ≈ (input tokens × input price) + (output tokens × output price)

In practice, total cost also includes retries, failed requests, conversation history that is resent on every turn, embedding and retrieval calls, and — for self-hosted models — servers, GPUs and engineering time.

Hidden token drivers

• Long system prompts repeated on every call.
• Whole documents pasted into the prompt when only a few paragraphs are relevant.
• Growing chat history resent in full on each turn.
• Verbose outputs when a short structured answer would do.
• Indian-language text: many tokenisers split Devanagari and other Indic scripts into more tokens than equivalent English text, so the same message can cost noticeably more. Measure this for your own content.

Technique 1: Use Smaller, Task-Specific Models

The biggest saving usually comes from not using a large model where a small one will do. A fine-tuned SLM for a narrow task can match a large model on that task while using far less compute. The cost is up-front work: collecting examples, fine-tuning and evaluating. It pays off when the task is frequent and well defined.

Technique 2: Model Routing

Routing sends each request to the cheapest model that can handle it:

1. A small model, or simple rules, classifies the request.
2. Routine requests are answered by the small model.
3. Hard, ambiguous or high-stakes requests go to a larger model — or to a person.

A related pattern is a cascade: try the small model first and escalate only if its answer fails a confidence or validation check. Routing needs monitoring, because a router that escalates too rarely saves money by quietly lowering quality.

Technique 3: Retrieval-Augmented Generation (RAG)

Instead of putting a whole policy manual or product catalogue into the prompt, RAG retrieves the few passages relevant to the question and sends only those. This reduces input tokens and usually improves accuracy, because the model works from the right source text. Good RAG depends on clean documents, sensible chunking and checking that the retrieved passages actually contain the answer.

Technique 4: Prompt Compression and Context Reduction

• Shorten instructions; remove repeated examples once the behaviour is stable.
• Summarise older conversation turns instead of resending them in full.
• Send structured data (IDs, fields) rather than long prose where possible.
• Use provider-side prompt caching where it is available, so a long, unchanging prefix is billed at a lower rate on repeat calls.

Technique 5: Semantic Caching

Many requests are near-duplicates: "What are your clinic timings?" and "When are you open?" A semantic cache stores previous answers with an embedding of the question and returns the stored answer when a new question is similar enough. Two cautions: answers can go stale, so set expiry rules; and never serve one user's personal information to another from the cache.

Technique 6: Output Token Limits

Because output tokens are often the more expensive kind, asking for concise, structured responses — a JSON object, a three-line summary, a yes/no plus reason — can cut costs significantly. Set a maximum output length and design prompts so the model knows what "short" means.

Technique 7: Quantisation and Batching (Self-Hosted Models)

When you run open-weight models on your own infrastructure, cost is driven by hardware time rather than per-token prices.

• Quantisation stores model weights at lower precision (for example 8-bit or 4-bit), reducing memory and often increasing speed. It can reduce quality, so re-evaluate after quantising.
• Batching processes many requests together; inference servers that support continuous batching keep the hardware busy and raise throughput per server.
• Utilisation matters: a GPU server that sits idle most of the day can cost more per request than an API.

Why Fewer Tokens Does Not Always Mean Lower Cost

Token reduction is a means, not the goal. A cheaper configuration can cost more overall if it:

• produces more errors that staff must fix;
• needs more retries or escalations;
• requires dedicated infrastructure that is under-used;
• is too slow for users, who then abandon the task;
• adds engineering and maintenance work that outweighs the savings.

The right unit is cost per successfully completed task, including human review time.

A Quality-vs-Cost Evaluation Framework

Test every option on the same representative task set and compare five measures:

Set the quality bar first, then choose the cheapest option that clears it. Re-run the evaluation whenever you change models, prompts or data.

A Practical Rollout Plan

1. Measure: log tokens, latency and cost per task for your current setup.
2. Trim: shorten prompts, cap outputs and add retrieval — usually the quickest wins.
3. Cache: add caching for repeated, non-personal questions.
4. Route: send routine requests to a smaller model; escalate the rest.
5. Specialise: fine-tune an SLM for the highest-volume task once you have good examples.
6. Re-evaluate: repeat the framework above and keep monitoring in production.

For practical, industry-specific examples, see AI development and automation for Pune businesses. For Indian-language workloads, see our research notes on building a Sanskrit small language model, and for agent-based systems, multi-agent AI architecture.