Building Efficient Small Language Models for Indian Industries: Architecture, RAG and Multi-Agent AI
Table of contents
- Why Domain-Specific SLMs?
- A Reference Architecture
- Indic and Industry-Specific Models
- AI Token Cost Optimisation
- Model Routing
- Quantisation
- Semantic Caching
- Retrieval-Augmented Generation (RAG)
- Multi-Agent Architecture
- LangGraph
- MCP
- A2A
- Evaluation and Deployment Considerations
- Where This Work Stands
Large language models made AI accessible, but running them for every task can be expensive, slow and difficult to control — especially for Indian organisations that serve many users in many languages on tight budgets. Small language model development offers another route: compact, domain-specific models that do a few jobs well, combined with retrieval, routing and agent orchestration so that each request uses only as much AI as it needs.
This guide brings together the architecture choices behind efficient SLM systems for Indian industries — from model selection and token cost optimisation to RAG, multi-agent design, LangGraph, MCP and A2A — and links to deeper articles on each topic. It reflects the engineering approach of our AI Transformation Lab. Much of this is research and project work: we do not claim that every component described here is deployed in our products.
Why Domain-Specific SLMs?
A domain-specific AI model is trained or fine-tuned for one area — for example, classifying hospital enquiries, extracting fields from invoices, or summarising school circulars in Marathi. Compared with a general large model, a well-tuned small model can be:
- cheaper and faster to run, because it has far fewer parameters;
- easier to host privately, which matters for healthcare, HR and financial data;
- more consistent on its narrow task, with output formats that are easier to validate; and
- easier to evaluate, because the task is well defined.
The trade-off is breadth: a small model fails outside its domain. That is why SLMs work best inside a larger architecture that knows when to escalate.
A Reference Architecture
A practical SLM system for an Indian business typically has these layers:
- Channels: web, mobile, WhatsApp, email or internal tools, where requests arrive.
- Pre-processing: language identification, script normalisation, removal of unnecessary personal data.
- Router: decides whether a request needs a rule, a cached answer, a small model, a large model or a human.
- Retrieval (RAG): finds relevant passages from approved documents and data, with permissions applied.
- Models: one or more task-specific SLMs, plus an optional larger model for hard cases.
- Orchestration: for multi-step tasks, an agent workflow that calls tools and other agents.
- Validation and approval: schema checks, policy checks and human approval for important actions.
- Observability: logs of tokens, latency, cost, errors and outcomes for every step.
Indic and Industry-Specific Models
For India, two kinds of specialisation matter. Language specialisation adapts a model to Hindi, Marathi, Tamil, Bengali, Sanskrit and other languages, which often requires tokeniser work and continued pre-training — see Indic language small language models and Sanskrit small language model research. Industry specialisation adapts a model to the vocabulary, documents and tasks of a sector such as agriculture, education or healthcare.
Parameter-efficient fine-tuning (for example, LoRA adapters) makes it practical to keep one shared base model and swap small adapters per language or task, instead of maintaining many full models.
AI Token Cost Optimisation
Token costs grow with every word sent to and returned from a model. The main levers, explained in detail in how to reduce LLM token costs, are:
- shorter prompts and tighter context — send only what the task needs;
- output limits and structured outputs;
- retrieval of a few relevant passages instead of whole documents;
- caching repeated answers; and
- routing routine work to smaller models.
Measure the real cost per completed task, including retries and human review. Fewer tokens is not a saving if quality drops and staff have to fix the output.
Model Routing
AI model routing sends each request to the most economical option that can handle it. A simple router might use rules (keywords, form fields), a lightweight classifier, or the confidence of a small model's own answer. Common patterns:
- Cascade: try the small model first; if confidence is low or validation fails, escalate to a larger model.
- Task-based routing: classification and extraction go to SLMs; open-ended drafting goes to a larger model.
- Risk-based routing: anything involving money, health or legal matters goes to a human for approval regardless of model confidence.
Quantisation
Quantisation stores model weights at lower precision — for example, 8-bit or 4-bit integers instead of 16-bit floating point — which reduces memory use and can increase throughput on the same hardware. It makes self-hosting small models on modest GPUs or even CPUs more realistic. The cost is a possible loss of accuracy, which varies by model, task and language. Always re-run your evaluation set after quantising, and check lower-resource languages separately, as they can be affected more.
Semantic Caching
Semantic caching stores previous answers and reuses them when a new question has the same meaning, even if the wording differs. It works well for frequent, non-personal questions — opening hours, policy explanations, product information. Use it carefully: set expiry times so answers stay current, never cache answers that contain personal data, and scope caches by language and user permissions.
Retrieval-Augmented Generation (RAG)
Enterprise RAG keeps knowledge in documents and databases rather than in model weights. That lets a small model answer accurately about information it was never trained on, and lets you update knowledge without retraining. Good enterprise RAG requires:
- clean, versioned source documents;
- chunking and indexing for both keyword and semantic search, including for Indian-language text;
- permission-aware retrieval, so users only see what they are allowed to see;
- answers that cite their sources, and say so when the documents do not contain the answer; and
- evaluation of both retrieval (were the right passages found?) and generation (is the answer faithful to them?).
Multi-Agent Architecture
Some business tasks involve several steps: read an enquiry, look up a record, draft a response, check it against policy, and schedule a follow-up. A multi-agent AI architecture splits such work across specialised agents, each with a narrow role and only the tools it needs. The benefits are clearer responsibilities, easier testing and tighter permissions. The risks are added complexity, cost and new failure modes, so use multiple agents only when a single workflow is genuinely not enough. Our detailed guide covers this in depth: multi-agent AI with LangGraph, MCP and A2A.
LangGraph
LangGraph is an open-source framework for building agent workflows as graphs of steps with explicit, shared state. Each node is a step (call a model, call a tool, wait for approval); edges define what happens next, including branches and retries. Because state is explicit, workflows can be paused for human approval, resumed and inspected — useful properties for business processes that must be auditable.
MCP
The Model Context Protocol (MCP) is an open standard for connecting AI applications to tools and data sources through a common interface. Instead of writing custom integration code for every model and every tool, a tool is exposed once as an MCP server and can be used by any compatible client. For enterprise systems, MCP servers should enforce their own authentication and permissions, and treat content returned from tools as untrusted input that could contain prompt-injection attempts.
A2A
Agent-to-Agent (A2A) is an open protocol for agents built by different teams or vendors to discover each other's capabilities and exchange tasks. Where MCP connects an agent to tools, A2A connects agents to other agents. It is most relevant when workflows cross organisational or system boundaries — for example, an agent in a CRM handing a task to an agent in a separate billing system. As with any integration, authentication, authorisation and audit logging come first.
Evaluation and Deployment Considerations
| Area | What to check before going live |
|---|---|
| Quality | Task-specific test sets per language; human review of samples; faithfulness of RAG answers |
| Cost | Cost per completed task including retries, escalations and human time |
| Latency | Median and slow-tail (p95) response times for the real user journey |
| Safety | Prompt-injection tests, refusal of out-of-scope requests, approval for high-impact actions |
| Privacy | Data minimisation, hosting location, retention, compliance with the DPDP Act, 2023 |
| Operations | Monitoring, alerts, fallbacks when a model or tool fails, and a way to roll back |
Start with one high-volume, low-risk task, measure it honestly against the current process, and expand only when the numbers justify it. For a practical view of moving from pilot to production, see AI transformation for Indian businesses.
Where This Work Stands
At ConnectHiveCRM, AI features such as a chatbot, lead scoring and WhatsApp automation are part of our AI-Powered CRM. The SLM, routing, RAG and multi-agent architecture described here is the subject of our applied research and project-based work in the AI Transformation Lab; Indic, agriculture, education and healthcare SLMs are research initiatives or roadmap items, not released models.
Frequently Asked Questions
What is a domain-specific small language model?
A compact model fine-tuned for a narrow set of tasks in one domain or language, such as classifying hospital enquiries or summarising Marathi circulars.
Does quantisation reduce accuracy?
It can. The effect depends on the model, task and language, so re-run your evaluation set after quantising and check lower-resource languages separately.
What is the difference between MCP and A2A?
MCP connects an AI application to tools and data sources through a standard interface; A2A lets agents built by different teams discover each other and exchange tasks.
Are these systems deployed in ConnectHiveCRM products?
Our AI-Powered CRM includes features such as a chatbot, lead scoring and WhatsApp automation. The SLM, routing and multi-agent architecture in this article is research and project-based work.