Teaching Whisper to Listen to Sanskrit: What Worked and What the Curves Say

How I fine-tuned OpenAI Whisper for Sanskrit speech recognition, what the public training runs show, and what I would not claim from the numbers.

2026-11-21
4 min read

Share

Sanskrit is one of the oldest living languages, and one of the hardest to give to a speech model. I spent a good part of my master's degree teaching OpenAI's Whisper to listen to it. This is the honest version of that story: what worked, what I would not claim, and what the training curves actually say.

Why Sanskrit is hard for a machine

Three things make Sanskrit unusually awkward for automatic speech recognition (ASR):

  1. There is very little digital audio. Models learn from hours of aligned speech, and for Sanskrit those hours are scarce.
  2. The sounds are precise. Retroflex consonants and vowel length change meaning, so a model that blurs them produces a different word.
  3. Sandhi. Sounds merge across word boundaries, so where one word ends and the next begins is not always audible. A system trained on languages with clear spaces between words has to unlearn that assumption.

Why start from Whisper

Training a speech model from scratch needs far more data than Sanskrit offers. Whisper has already heard hundreds of thousands of hours of speech in many languages, so it arrives knowing what speech sounds like. Fine-tuning asks a smaller question: given that, how do I teach it this language?

Fine-tuned model=f(Whisper,Sanskrit audio)\text{Fine-tuned model} = f(\text{Whisper}, \text{Sanskrit audio})

In the paper with Punyeshwarananda (IEEE C3IT 2024) I fine-tuned Whisper Small on about 78 hours of Sanskrit audio with Hugging Face Transformers. It reached a 15.42% word error rate (WER), meaning roughly one word in seven was wrong on the evaluation set.

What the public experiments show

Alongside the paper model I published a few training runs on Hugging Face. Each was evaluated on its own validation split, so the numbers are useful for comparing how training behaves, not for comparing against the paper's 15.42%.

  • whisper-small-sanskrit (Whisper Small, linear schedule): final WER 27.94%, best 27.30% at step 8,500.
  • whisper-small-sanskrit-lr-scheduler-2 (Whisper Small, cosine schedule): final WER 29.26%, best 27.61% at step 9,500.
  • whisper-medium-sanskrit-try-2 (Whisper Medium, reduce-on-plateau schedule): final WER 24.88%, best 24.45% at step 4,500.

Three things stand out.

Bigger helped. The medium model started lower and finished lower than either small run, even though it used a much smaller learning rate.

The best checkpoint is not the last one. The medium run reached its best WER at step 4,500 of 10,000. After that its validation loss kept rising while WER stayed roughly flat. That is what early overfitting looks like, and it is the reason I look at the curve and not just the final number.

WER is noisy. The cosine run bounced between 27.6% and 31.5% near the end. A single checkpoint is a sample, not a verdict, so I would not read much into a one-point difference between schedules.

What I would not claim

  • The public runs are not the model in the paper, so their WER should not be compared with 15.42%.
  • WER measures words, and for Sanskrit a "wrong word" can be a single sandhi boundary placed differently. A human reader often sees those errors as smaller than the number suggests, but I have not measured that, so I am not claiming it.
  • One speaker set and one domain is not "Sanskrit speech" in general.

What I would do next

Collect more varied audio, evaluate on a held-out set recorded by different speakers, and report character error rate next to WER, since it forgives sandhi-boundary slips and shows whether the model is hearing the sounds correctly.

Try it

You can record or upload speech in the live demo. It runs one of the earlier public models, so it shows the approach rather than the paper model's accuracy. The training curves and the full project story are in the case study.

Whisper
Sanskrit
Speech recognition
Fine-tuning
Hugging Face
WER
Transfer learning

You may also like