The best tiny voice in the world
Natural, expressive speech, empathetic where it is needed, from a model small enough to run on the cheapest phones, with no cloud and no connection. Excellent quality on very low infrastructure, reached by new methods, and built in the open: this page is the training as it happens.
a two-minute walk through the page, for anyone new to how models learn
The model
Versions
How Kalarav learns to speak
Kalarav is Neelverse Lab's own voice, small enough to speak in real time on an entry-level 2 GB device. It learns from a big "teacher" voice (Supertone's Supertonic 3) by distillation: a small student watches the teacher's work until it can do the same job in one quick step instead of sixteen slow ones. Everything on this page is live from the PC that trains it. Hover or tap anything to see what it is.
How a voice is made
The violet box is the part being trained now. The teacher's thinker (dashed) is only used while teaching; on the phone, Kalarav's own thinker replaces it.
For 28,428 sentences in six languages (51 hours), the teacher turned random noise into finished speech in 8 careful steps. Each result was saved.
The student gets the same noise and text and must produce the same speech in one step. First it learnt the answers directly (phase A).
Copying answers makes speech dull, an average. So two helper models judge how the teacher's speech sounds in general against how the student's does, and push the student towards the teacher's (phase B, now).
When the scores are good, it is tested blind against the teacher and round six. Only then is it released.
One training step, as it happens
The loop that teaches Kalarav, about 40 times a minute. The dot walks through one step; the numbers are the latest step's.
Is everything OK?
Now
Inside the thinker
All 7,168 neurons of the 17M thinker that is learning right now: 28 layers of 256, sound coming in on the left, going out on the right.
- Each dot is a neuron; each disc of dots is a layer.
- Bright dots fired strongly on a test sentence.
- Lines are the strongest links into the next layer.
- The wave running left to right is speech passing through.
- Layers pulse when training is changing them.
The thinker's parts
Every part of the 17M thinker, sized by how many numbers it holds and coloured by how much the last training step changed it (bright = changing). Hover a part.
Where it is learning
Every 50 training steps, how far each of the thinker's 44 parts moved. Bright = learning; dark = settled. Hover a cell for its numbers.
The losses, live
How hard each stage is being taught
Is it close to the teacher?
Scores on 287 sentences the student never trained on, every 2,000 steps, for every round so far. Dashed lines mark restarts and what changed.
Liveliness (teacher = 1)
Gap from the teacher's 8-step speech (lower is closer)
What it hears and says
One held-out sentence, looked at inside the thinker every 500 steps.
Which letters each moment of sound listens to (four stages, front to back; rows: moments, columns: letters)
Activity of every layer over time
The push from distribution matching
The student's latent
The teacher's latent
Difference
Which of the 50 style tokens each stage uses
Time machine
Slide back through the saved looks inside to watch the letters' diagonal form, the layers wake up and the speech path open out as training goes on.
Speech as a path
The test sentence drawn as two paths through space, one point every 70 ms. Gold: the teacher. Violet: the student. The bright dots walk along both at the speed of speech.
- Each point is 70 ms of speech (its 144 numbers, seen from the 3 directions where they vary most).
- Close paths = similar speech.
- A smaller, smoother violet path = flatter, averaged speech: what training is fixing.
Hear and see
The same sentence from the teacher, round six and the 17M student, as sound and as a picture of the sound (a spectrogram: time to the right, pitch upwards, brightness is loudness). Press play and watch the line follow.
The voices' style space
Each voice is 12,800 numbers. Seen from the three directions where the ten voices differ most, women and men separate, and Kalarav's F1 and M1 sit among them. Learnt tones will appear as arrows from F1 and M1.
- Violet: women's voices; teal: men's.
- Close together = sound alike.
- Gold arrows (coming): a tone such as happy, as a small move of the same voice.
The technical side
Sizes, work per second of speech, and speed on an entry-level 2015-class Android device (4 × Cortex-A53, 2 GB).
Work per second of speech (multiply-adds, log scale)
What it learns from
The computing it took
The maths, in plain words
Each idea: what it means in one line, an everyday comparison, the formula (hover any symbol), and today's numbers. Then the step-by-step recipes the computer follows.
The recipes (algorithms)
Glossary
Research map
Every road tried since 29 October. Each box is one attempt; a line means it grew out of the box on its left. Hover a box to trace its road back to the start, click it for its story, or replay how it all unfolded.