01 — Dataset
Kaggle “Audio dataset with 10 Indian languages”; three classes were carved out for the competition entry — English, Gujarati, and Hindi.
kaggle datasets download · 3 classes · Train / Test split 80/20
Case Study · Machine Learning
IEEE Multimedia Signal Processing CupSpoken Language Recognition from Mel-Spectrograms
A first-place entry that classifies the spoken Indian language directly from audio by converting speech into mel-spectrogram images and training an InceptionV3 CNN from scratch — 99.67% validation and 100% held-out test accuracy across English, Gujarati, and Hindi.
PythonTensorFlow / KerasInceptionV3librosaMel-SpectrogramsSignal Processing1st Place — IEEE MMSP Cup
- 1st
- IEEE MMSP Cup
- 99.67%
- Validation accuracy
- 100%
- Held-out test accuracy
- 9,043
- Spectrogram images
Pipeline
From raw microphone audio to a language label
Five stages turn variable-length speech recordings into fixed-size spectrogram images and a single softmax decision.
02 — Augmentation
Every clip is re-encoded to WAV and can be perturbed with additive Gaussian noise, expanding the effective training set and hardening the model against recording conditions.
add_noise(gain=0.005) · soundfile.write
03 — Spectrogram imaging
Audio is resampled to 8 kHz, mel-spectrogrammed with 128 mel bands and a per-clip hop length, converted to dB, min-max normalised, and written as a 128×500 grayscale PNG.
librosa.feature.melspectrogram · power_to_db · imageio.imwrite
04 — CNN classification
InceptionV3 is trained from scratch on the single-channel spectrogram images with RMSprop, exponential LR decay, and early stopping over 20 epochs.
InceptionV3(input_shape=(128,500,1)) · RMSprop(0.045) · categorical_crossentropy
05 — Live inference app
A microphone recorder captures speech in the browser, the clip is spectrogrammed with the identical front-end, and the saved model returns the predicted language.
MediaRecorder → recording.wav → spectrogram → model.predict
Mel-spectrogram input (128 × 500)
Configuration
Signal, model, and dataset parameters
Every value below is taken directly from the competition notebook.
Signal front-end
| Parameter | Value | Notes |
|---|---|---|
| Sample rate | 8 kHz | Resampled on load with librosa |
| Mel bands | 128 | Spectrogram image height |
| Image width | 500 frames | Hop length derived per clip length |
| Amplitude scaling | power_to_db + min-max | Normalised to [0, 1], stored as uint8 PNG |
| Colour mode | Grayscale (1 channel) | Single-channel spectrogram input |
Model & training
| Parameter | Value | Notes |
|---|---|---|
| Architecture | InceptionV3 (weights=None) | Trained from scratch, 3-way softmax |
| Optimiser | RMSprop | Initial learning rate 0.045 |
| LR schedule | Exponential decay | 0.045 → 9.25e-05 by epoch 20 |
| Batch size / epochs | 128 / 20 | 56 steps per epoch, EarlyStopping enabled |
| Loss | Categorical cross-entropy | Final train loss 0.0024 |
Data volume
| Parameter | Value | Notes |
|---|---|---|
| Training images | 7,236 | flow_from_directory, 3 classes |
| Validation images | 1,807 | 20% validation split |
| Test images | 240 | Held-out evaluation set |
| Classes | English · Gujarati · Hindi | Balanced per-language folders |
Results
Twenty epochs of measured training history
Curves reconstructed from the notebook's per-epoch logs — accuracy, loss, and the learning-rate decay that unlocked convergence.
Training vs validation accuracy
- Train accuracy
- Validation accuracy
Final train accuracy
99.94%
Epoch 20, loss 0.0024
Best validation accuracy
99.67%
1,807 held-back spectrograms
Held-out test accuracy
100.0%
240 images, loss 4.09e-04
Inference
Live language prediction from a microphone clip
The demo application reuses the training front-end verbatim, so a fresh recording is embedded exactly like the dataset.
Runtime chain
- 01Browser MediaRecorder captures the utterance and writes recording.wav.
- 02audio_to_image_file() resamples to 8 kHz and renders a 128×500 mel-spectrogram PNG.
- 03The PNG is normalised to [0,1] and resized to (128, 500, 1).
- 04model_Language_best.h5 predicts a 3-way softmax; argmax maps to the language label.
Sample prediction
preds = model.predict(image) # [[3.0403108e-08 9.9999285e-01 7.1120421e-06]] # English Gujarati Hindi You were speaking : Gujarati
The softmax is effectively saturated — 0.99999 on the correct class — which is typical of the trained model on clean 10-second clips.
Takeaways
What the competition entry proved
Signal-processing choices, not model size, decided the outcome.
Reframing an audio problem as an image problem — mel-spectrograms fed to a vision CNN — let a proven ImageNet-class architecture do language discrimination without hand-crafted MFCC features.
Validation accuracy stayed near chance (≈33%) for the first eight epochs while training accuracy climbed past 97%: the aggressive 0.045 initial learning rate produced exploding validation loss until exponential decay stabilised the batch-norm statistics.
Once the learning rate dropped below ~0.007, validation accuracy jumped from 62% to 97% in a single epoch and settled at 99.67% — evidence the schedule, not model capacity, was the bottleneck.
Additive-noise augmentation plus fixed-length spectrogram framing kept the model robust on the 240-image held-out set, which it classified at 100% with a loss of 4.09e-04.
The same preprocessing function is reused at inference time, so a microphone recording travels through an identical transform to the training images — the detail that keeps live predictions trustworthy.
Deployed end-to-end as a user-facing demo: record → spectrogram → predict, returning a confident single-label result (e.g. 0.99999 on a Gujarati sample).