Case study · AI/ML · Data & governance
Speech AI that works on children's real-world audio
Open-source models that tell who is speaking, and when, in day-long recordings worn by young children: state of the art across 40+ languages, approaching human-annotator agreement.
- My role
- Built the multi-country data consortium and annotation pipeline behind the models; co-led VTC; co-supervised BabyHuBERT
- When
- 2019–2026
- Read
- 3 min read
The problem
Researchers and NGOs now record children's everyday language environment with small wearable devices, producing thousands of hours of audio. Nobody can listen to it all, and speech models trained on clean adult speech break down on it: roughly 80% of a day-long recording is silence or noise, voices are far from the microphone, and families speak dozens of languages, many with little or no speech technology. The only widely used automatic tool was a closed, English-centred commercial system.
Constraints
- Recordings of children at home are highly sensitive: data cannot simply be pooled or published
- Human annotations are scarce and expensive, especially for under-represented languages
- Must work across very different languages, cultures and acoustic environments
- End users are researchers and field teams, mostly without machine-learning expertise
My approach
First, we built and released the Voice Type Classifier (VTC, 2020), an open-source model that labels speech as coming from the key child, other children, female adults or male adults. It outperformed the commercial alternative. Next, to move past the ceiling of supervised training on small annotated sets, I co-supervised BabyHuBERT: a self-supervised speech model pretrained on about 13,000 hours of child-centred recordings from 40+ languages, after voice-activity detection cut the share of non-speech from ~80% to ~8%. We fine-tuned it for voice type classification and evaluated it on six corpora, using child-disjoint train/validation/test splits so that no child's voice leaks from training into testing.
Results
- 65.1%, vs. 69.8% agreement between human annotatorsVoice type classification, average F1
- ~13,000 hours, 40+ languagesPretraining data
- +14.0 and +18.3 F1 points over HuBERT (Vanuatu, Solomon Islands)Gains on under-represented languages
BabyHuBERT consistently outperformed both an adult-speech model (HuBERT) and a model pretrained on English child recordings (W2V2-LL4300) on all six evaluation corpora (F1 55.0–76.1%). An independent follow-up found that BabyHuBERT is also the best starting point for recognising the speech sounds young children produce. Code and models are shared with the research community, under the governance described in a separate case study.
Trade-offs
Pretrain on noisy, in-domain, multilingual audio instead of adapting a model built for clean adult speech
Years of experiments showed that new architectures and tuning brought only marginal gains on this data. The bottleneck was the mismatch between training audio and real-world recordings.
- Fine-tune an off-the-shelf adult-speech model (HuBERT, wav2vec 2.0)
- Reuse a model pretrained only on English child recordings
Split data by child, not by audio segment
Random segment splits put the same child's voice in both training and test sets, which inflates scores and hides how the model behaves on new families.
- Random segment-level splits (higher reported numbers, weaker generalisation)
Report results per corpus and per language, not just an average
The largest gains, and the largest risks, are in under-represented languages. An average would have hidden both.
What I learned
- In messy real-world audio, choosing the right data mattered more than tweaking the architecture.
- Averages hide who a model fails: evaluating per language and per population is part of building it fairly.
Skills used
- Self-supervised speech models
- Speaker diarization
- Evaluation design
- Dataset curation
- Open-source release
- Supervising ML engineers