Automatic Speech Recognition for Sanskrit with Transfer Learning
Teaching a general speech model to understand one of the world's oldest languages: how it was fine-tuned, how the training curves looked, and how you can try it yourself.
0.00%
Word error rate
reported in the paper
~0 h
Sanskrit audio
used to fine-tune
0
Early models on the Hub
public experiments, not the final model
0
Public demos
running those early models
0
IEEE C3IT
peer-reviewed paper
The problem
One of the world’s oldest languages, and almost no data for machines to learn from
Sanskrit has an enormous body of books and manuscripts, but very little of it exists as digital audio, and audio is exactly what a speech-recognition model needs. The language also fights back: retroflex consonants and precise vowel lengths make sounds easy to confuse, and sandhi rules fuse neighbouring words together so the boundaries a model usually relies on are blurred.
The idea was to avoid starting from nothing. A large multilingual model like OpenAI’s Whisper already knows a great deal about speech; transfer learning teaches it Sanskrit with a comparatively small amount of data.
The method
From a general speech model to a Sanskrit one
- 01
Start from Whisper
Take OpenAI’s pretrained Whisper checkpoints, small and medium, as the starting point instead of training from scratch.
openai/whisper-smallopenai/whisper-medium - 02
Fine-tune on Sanskrit speech
Fine-tune with Hugging Face Transformers on roughly 78 hours of Sanskrit audio, tuning the learning rate, batch size and schedule across several runs.
Hugging Face TransformersPyTorch~78 h of audio - 03
Measure with word error rate
Score every checkpoint with word error rate (WER), the share of words the system gets wrong, so runs can be compared on one number.
WERValidation split - 04
Share what can be shared
Publish the paper, release early experiments on the Hugging Face Hub and host public demos, so anyone can speak Sanskrit into a browser and read a transcript. The final model stays private.
IEEE paperHugging Face HubGradio Spaces
The experiments
Three fine-tuning runs, checkpoint by checkpoint
These are the training logs from three of my earlier experiments, the ones I made public. The model behind the paper’s result is kept private, so this is the path towards it, not the destination. Toggle runs on and off and move across the chart to see how word error rate fell as training went on.
Hover or drag across the chart to read each checkpoint. Lower is better.
| Run | Starting model | Learning rate | Batch | Schedule | Final WER | Best WER |
|---|---|---|---|---|---|---|
| Small · linear schedule | whisper-small | 2.5e-5 | 8 | linear, 500 warm-up steps | 27.94% | 27.30% @ 8,500 |
| Small · cosine schedule | whisper-small | 2.55e-5 | 8 | cosine, 500 warm-up steps | 29.26% | 27.61% @ 9,500 |
| Medium · plateau schedule | whisper-medium | 6.25e-6 | 16 | reduce-on-plateau, 500 warm-up steps | 24.88% | 24.45% @ 4,500 |
Try it
Use the model in a few lines
The public demo runs one of those early small models through the standard Transformers pipeline, and so can you. It is a good way to hear the approach working, though not the final model’s accuracy.
from transformers import pipeline
asr = pipeline(
"automatic-speech-recognition",
model="Bidwill/whisper-small-sanskrit",
chunk_length_s=30, # split long audio into 30-second windows
)
print(asr("sanskrit_recording.wav")["text"])Go deeper
The paper and the models
You may also like


