Building a Sanskrit Small Language Model: Research Challenges and Architecture

Large language models have become remarkably good at English, and steadily better at Hindi and other major Indian languages. Sanskrit is different. It is one of the most precisely described languages ever studied — Pāṇini's grammar set out its rules more than two thousand years ago — yet for modern AI it is a low-resource language: there is relatively little clean, digitised, openly licensed text, and very little of it is paired with translations or annotations.

This article explains what it would take to build a Sanskrit small language model: why the language is hard for today's NLP tools, how to prepare data, how tokenisation and model choice affect results, what a complete system architecture looks like, and how to evaluate honestly. It also sets out the research roadmap we are following in our AI Transformation Lab. To be clear from the start: this is an ongoing research direction. We have not released a Sanskrit model and we do not publish benchmark claims we cannot reproduce.

What Is a Small Language Model?

A small language model (SLM) is a language model with far fewer parameters than the largest general-purpose models, usually trained or fine-tuned for a narrower set of tasks. There is no strict cut-off for "small"; the useful idea is that the model is compact enough to run on modest hardware and focused enough to do a specific job well.

For Sanskrit, an SLM makes sense because the target tasks are specific — translation, summarisation, morphological analysis, question answering over texts — and because a focused model can be evaluated thoroughly by scholars. A small model that does three things reliably is more useful than a large one that does everything approximately.

Why Sanskrit Is Hard for NLP

Sandhi: words that merge

In Sanskrit, sounds at word boundaries combine according to sandhi rules. Two or more words can fuse into a single written string, and the boundary may disappear entirely. A system that does not "undo" sandhi sees a long, unfamiliar token where a reader sees several words.

Compounds (samāsa)

Sanskrit freely builds long compound words. A single compound can carry the meaning of a whole English phrase, and its internal structure determines that meaning. Splitting compounds correctly is a research problem in its own right.

Rich morphology and flexible word order

Nouns inflect for eight cases, three numbers (singular, dual and plural) and three genders; verbs carry tense, mood, person and number. Because grammatical relationships are marked on the words themselves, word order is relatively free. Models trained mostly on English-like word order have to learn very different patterns.

Many scripts and transliterations

Sanskrit is most often written in Devanagari today, but historically it appears in many Indic scripts, and scholarly text frequently uses romanised transliteration such as IAST. The same verse can therefore exist in several encodings, with inconsistent Unicode, punctuation (the daṇḍa । and double daṇḍa ॥) and, in Vedic texts, accent marks.

Scarce parallel and annotated data

Machine translation and summarisation need examples: Sanskrit sentences paired with translations or summaries. Such pairs exist, but in far smaller quantities than for high-resource languages, and many are drawn from a narrow range of classical texts.

Step 1: Data Preparation

Data work is where most of the effort goes, and where most of the risk lies.

• Sourcing and licensing. Useful starting points include digitised text collections and research corpora such as the Digital Corpus of Sanskrit and GRETIL, and translation datasets built from classical texts. Every source has its own licence and terms; each document's origin and licence should be recorded before it is used.
• Deduplication. Popular texts appear many times in slightly different editions. Duplicates inflate apparent dataset size and can leak test material into training data.
• Quality filtering. OCR errors, broken encodings and mixed-language fragments need to be detected and removed or corrected.
• Domain balance. A corpus dominated by one genre — for example, epic poetry — produces a model that struggles with philosophical prose, technical śāstra literature or modern Sanskrit.
• Held-out evaluation sets. Set aside test material before any training begins, and keep it away from every training step.

Step 2: Text Normalisation

Before tokenisation, the text must be consistent:

1. Convert transliterated text to one target representation, or keep script and transliteration as clearly labelled parallel forms.
2. Apply Unicode normalisation (for example, NFC) so visually identical characters are stored identically.
3. Standardise punctuation, numerals and whitespace.
4. Decide how to treat accent marks, editorial brackets and verse numbering, and apply the decision everywhere.

These choices sound mundane, but inconsistent normalisation silently splits the vocabulary and makes evaluation unreliable.

Step 3: Tokenisation

Most modern models use subword tokenisers such as byte-pair encoding (BPE) or SentencePiece. Tokenisers trained mainly on English and other high-resource languages tend to break Devanagari text into many short pieces. That has two costs: the model sees less meaningful units, and every sentence consumes more tokens — which also raises inference cost.

Options we are evaluating include:

• A tokeniser trained on Sanskrit text, so common stems and endings become single tokens.
• Extending an existing tokeniser's vocabulary with Sanskrit subwords, which requires resizing and training the model's embedding layer.
• Sandhi- and compound-aware pre-processing using rule-based or statistical segmenters, so the model sees word boundaries a reader would recognise.

The right choice is an empirical question. We compare options by how many tokens a typical sentence needs and, more importantly, by downstream task quality.

Step 4: Model Selection

For a low-resource language, the starting model matters more than almost anything else. Practical criteria:

Step 5: Training From Scratch vs Fine-Tuning

Training a Sanskrit model from scratch requires a large corpus, significant compute and a great deal of evaluation effort. With the data available today, it is rarely the best first step.

The more practical path is to adapt an existing multilingual model:

• Continued pre-training on cleaned Sanskrit text, so the model absorbs the language's patterns.
• Task fine-tuning on translation, summarisation or question-answering examples.
• Parameter-efficient fine-tuning such as LoRA, which updates a small set of additional weights and keeps compute needs modest.
• Distillation, where a larger model's outputs — checked by humans — help train a smaller one.
• Quantisation after training, to reduce memory and speed up inference, followed by re-evaluation because compression can reduce accuracy.

Retrieval augmentation (RAG) is a useful complement: instead of relying on what the model memorised, the system retrieves relevant passages and shows the sources it used, which makes review much easier.

Step 6: Evaluation

Evaluation is where Sanskrit research most often goes wrong, because fluent output can hide serious errors.

Automatic metrics

Character-level metrics such as chrF tend to suit morphologically rich languages better than word-level BLEU, because a correct stem with a slightly different ending still earns partial credit. Learned metrics can help where they support the language pair. All automatic metrics are a first filter, not a verdict.

Human linguistic review

Sanskrit scholars should review samples for meaning, grammatical accuracy, handling of names and technical terms, and invented content. A simple, consistent error taxonomy — mistranslation, omission, addition, grammar, terminology — makes reviews comparable over time.

Hallucination checks

Models can produce confident text that is not in the source. Checks include comparing named entities and numbers between source and output, flagging outputs that are much longer or shorter than expected, and requiring citations to retrieved passages.

Honest reporting

Results should state the test set, the metric, the baseline and the known weaknesses. A single headline number, without method, should never be trusted — including our own.

A Reference Architecture for a Sanskrit SLM System

A useful Sanskrit model is more than the model itself. The architecture we are researching has seven layers, each of which can be tested on its own:

Separating the layers matters: when quality is poor, you can tell whether the problem is normalisation, tokenisation, the model or retrieval — and fix the right thing. After fine-tuning, quantisation can shrink the model for cheaper inference, but it is re-evaluated against the same test sets before use.

Our Research Roadmap

1. Foundations: curate openly licensed text, define normalisation rules and build evaluation sets with linguists.
2. Baselines: measure existing multilingual models on those sets to see where they succeed and fail.
3. Tokeniser experiments: compare tokenisation strategies by token efficiency and task quality.
4. Task fine-tuning: adapt compact models for narrow translation and summarisation tasks.
5. Efficiency: test quantisation and distillation against the same evaluation sets.
6. Supervised pilots: use outputs only with human review, and share what we learn.

The same principles shape our wider work on translation and summarisation for Sanskrit and Indian languages and on reducing AI costs with small language models.