Overview

An end-to-end speaker diarization and speech recognition experiment on the AMI Meeting Corpus: who spoke when (DER) and what was said (WER) in real, messy, multi-party conversation.

1. AudioMeeting recording
2. VADVoice activity detection (Silero)
3. EmbeddingsSpeaker voice vectors (ECAPA‑TDNN)
4. ClusteringSpectral, number of speakers estimated
5. ASRSpeech recognition (Whisper)
6. TranscriptLabelled by speaker

Status: work in progress. Phase 1, a careful, reproducible baseline and a learning experiment, is done and measured. Phase 2 (a production-style pipeline with state-of-the-art models) and Phase 3 (an API and a Streamlit interface) are planned.

28.4%mean diarization error rate on 12 held-out test meetings

10 / 12test meetings where the number of speakers was estimated correctly

≈26%word error rate on a dev meeting with Whisper medium (≈30% with base)

19.5%of the 28.4% error is missed speech, decided before clustering

Data and experimental design

The data is the AMI Edinburgh scenario meetings (ES2002–ES2016): 15 series of 4 meetings each, so 60 recordings and 60 speakers (each speaker appears in exactly one series), about 28 hours of speech, recorded with individual headset microphones. I pre-extracted the data from Hugging Face into .wav clips plus a transcript.csv, because streaming it repeatedly caused timeouts.

The split is fixed up front, by series, so no speaker is shared between sets:

Split	Series	Used for
Dev	ES2011	Exploration and every design decision (embedding, clustering algorithm, k estimation)
Train	11 other series	Fitting an optional embedding adaptation head
Test	ES2004, ES2006, ES2007 (12 meetings)	Evaluated once with the frozen pipeline

Results

Held-out test set, 12 meetings. DER (the share of speech time that is missed, falsely detected or given to the wrong speaker) uses a ±0.25 s collar and scores overlapped speech, which is the standard DER. My DER implementation matches pyannote.metrics to within 0.0001 on synthetic hypotheses over the real AMI reference. ARI measures how closely the clusters match the true speakers (1 is a perfect match).

System	k	Mean DER	Missed	False alarm	Confusion	ARI
Pretrained ECAPA + spectral	estimated	28.4%	19.5%	0.0%	8.8%	0.80
Pretrained ECAPA + spectral	true k	27.6%	19.5%	0.0%	8.1%	0.80
+ adaptation head	estimated	29.5%	19.5%	0.0%	10.0%	0.80
+ adaptation head	true k	27.5%	19.5%	0.0%	8.0%	0.82

Dev-meeting WER is about 30% with Whisper base and about 26% with medium.

What this says

• Missed speech dominates (19.5% of the 28.4%), and it is the same for every system. It is decided before clustering: speech the VAD drops, plus the second speaker in overlapped speech (9–25% of speech frames per test meeting), which a pipeline that gives each moment a single label cannot represent.
• The embedding is not the bottleneck. A trained adaptation head did not beat the pretrained ECAPA-TDNN on unseen speakers (held-out EER 4.1% pretrained against 3.8% at its best epoch and 4.2% after 20 epochs), and it did not help in clustering either.
• Estimating k is the fragile step. The number of speakers is right in 10 of 12 test meetings. When it is wrong, DER jumps by 10 points or more.

Caveats

• Oracle-segmented audio. Meetings are rebuilt from ground-truth utterance clips, so the gaps are digital silence. False alarm is 0% by construction, and the VAD scores are optimistic.
• The audio is clean, summed headset audio, not far-field.
• It is a small subset of the corpus, with a single run per configuration. Whisper’s sampling makes WER vary by about half a point between runs.
• WER on the test meetings is off by default (RUN_TEST_ASR in Stage 8), so the WER figures above are dev-meeting numbers.

Code and next steps

The full pipeline is one notebook, meeting-transcription-colab_version.ipynb, written for Colab and runnable locally with the same code. A second notebook, download-dataset.ipynb, writes the data layout. Its stages are:

1. Meeting reconstruction
2. Voice activity detection
3. Speaker embeddings (ECAPA against x-vector)
4. Clustering: algorithm selection on all dev meetings, and k estimation
5. Whisper (tiny and base)
6. The speaker-labelled transcript
7. DER and WER
8. Embedding adaptation with speaker-held-out cross-validation, and Whisper medium
9. Held-out test evaluation

Next. Re-measure on real continuous audio, then move to an overlap-aware diarizer such as pyannote, a larger Whisper with word-level alignment, and wrap it in a FastAPI backend with a Streamlit front end. The code and the roadmap are on GitHub.