Automatic Speech Recognition for Sanskrit with Transfer Learning

Teaching a general speech model to understand one of the world's oldest languages: how it was fine-tuned, how the training curves looked, and how you can try it yourself.

0.00%

Word error rate

reported in the paper

~0 h

Sanskrit audio

used to fine-tune

0

Early models on the Hub

public experiments, not the final model

0

Public demos

running those early models

0

IEEE C3IT

peer-reviewed paper

The problem

One of the world’s oldest languages, and almost no data for machines to learn from

Sanskrit has an enormous body of books and manuscripts, but very little of it exists as digital audio, and audio is exactly what a speech-recognition model needs. The language also fights back: retroflex consonants and precise vowel lengths make sounds easy to confuse, and sandhi rules fuse neighbouring words together so the boundaries a model usually relies on are blurred.

The idea was to avoid starting from nothing. A large multilingual model like OpenAI’s Whisper already knows a great deal about speech; transfer learning teaches it Sanskrit with a comparatively small amount of data.

The method

From a general speech model to a Sanskrit one

  1. 01

    Start from Whisper

    Take OpenAI’s pretrained Whisper checkpoints, small and medium, as the starting point instead of training from scratch.

    openai/whisper-smallopenai/whisper-medium
  2. 02

    Fine-tune on Sanskrit speech

    Fine-tune with Hugging Face Transformers on roughly 78 hours of Sanskrit audio, tuning the learning rate, batch size and schedule across several runs.

    Hugging Face TransformersPyTorch~78 h of audio
  3. 03

    Measure with word error rate

    Score every checkpoint with word error rate (WER), the share of words the system gets wrong, so runs can be compared on one number.

    WERValidation split
  4. 04

    Share what can be shared

    Publish the paper, release early experiments on the Hugging Face Hub and host public demos, so anyone can speak Sanskrit into a browser and read a transcript. The final model stays private.

    IEEE paperHugging Face HubGradio Spaces

The experiments

Three fine-tuning runs, checkpoint by checkpoint

These are the training logs from three of my earlier experiments, the ones I made public. The model behind the paper’s result is kept private, so this is the path towards it, not the destination. Toggle runs on and off and move across the chart to see how word error rate fell as training went on.

20%25%30%35%40%45%50%02k4k6k8k10ktraining steps

Hover or drag across the chart to read each checkpoint. Lower is better.

RunStarting modelLearning rateBatchScheduleFinal WERBest WER
Small · linear schedulewhisper-small2.5e-58linear, 500 warm-up steps27.94%27.30% @ 8,500
Small · cosine schedulewhisper-small2.55e-58cosine, 500 warm-up steps29.26%27.61% @ 9,500
Medium · plateau schedulewhisper-medium6.25e-616reduce-on-plateau, 500 warm-up steps24.88%24.45% @ 4,500

Try it

Use the model in a few lines

The public demo runs one of those early small models through the standard Transformers pipeline, and so can you. It is a good way to hear the approach working, though not the final model’s accuracy.

pythontransformers
from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="Bidwill/whisper-small-sanskrit",
    chunk_length_s=30,   # split long audio into 30-second windows
)

print(asr("sanskrit_recording.wav")["text"])

Go deeper

The paper and the models

You may also like