Igbo is a tonal language: the same letters can mean different things depending on pitch, and careful writing marks tone with grave and acute accents. Igbo also uses vowels with a dot below (ị, ọ, ụ), which are letters in their own right. This page takes 21 recordings of one speaker, run through the speech recognizer facebook/omniASR-CTC-1B, and shows what happens to those marks. Data: measured from the omniASR-igbo-blindspots dataset. Scores and charts are computed live from its rows.
Read this first. One speaker (Afikpo Igbo), 21 short clips, one model. This is a proof of concept that finds specific blind spots. It does not say how the model does on Igbo speech in general, and the category numbers below rest on 5 or 6 clips each.
What was said (reference transcript)The same words with tone marks removedWhat the model wrote
tone mark (grave, acute)dot-below letter (ị ọ ụ)both on one letter
Position by position (alignment estimate, see tab 4)
Four ways to score the same marks
Whiskers are 95% bootstrap intervals, 10,000 resamples of the clips in each group, computed in your browser. With 5 or 6 clips per category they are wide, which is the honest picture.
Why one category gives two different numbers
Which kind of mark is it?
Every letter that carries a mark, split by kind. Counts are letters with a mark, taken from the reference and from the model output, not matched position by position.
tone mark onlydot-below and tonedot-below only
What tone-blind transcription collapses
Words from the 21 reference transcripts that are spelled differently only by tone marks. Remove the tone marks and they become one spelling.
Beyond counting: are the marks in the right place?
alignment estimate
The dataset's scores compare how many marks there are. That cannot tell a correct mark from a wrong one: if the reference has an acute accent and the model writes a dot-below letter in the same spot, the count still matches. Here the reference and the model output are lined up letter by letter (ignoring marks), and each expected mark is checked at its aligned position. This is my extension, not the dataset's metric, and the alignment can mis-pair letters when the output is badly garbled.
Mark confusion matrix
Rows: what the reference has on a letter. Columns: what the model has on the aligned letter. The diagonal is correct. The "none" column is a dropped mark.
Why resample clips, not marks, and why more iterations do not help
Drop rate, bootstrap 95% interval. A bootstrap redraws the data with replacement many times and looks at how much the statistic moves. The unit you redraw decides the answer.
Does character error rate see tone loss?
One dot per clip that has expected marks. Horizontal: the dataset's character error rate. Vertical: share of expected marks dropped.
What this can and cannot show
What it supports
On these clips, the model's output lacks most of the tone marks in the reference.
The loss is concentrated in the clips built to test tone, and it shows up even when the output has marks (they are often the wrong ones).
Character error rate alone does not flag it.
What it does not show
That other speakers, dialects or recording conditions behave the same: one speaker, 21 clips.
That other models behave this way: one model.
Why it happens. Marks added to the monotone clip suggest the output is not tightly tied to the audio there, but a mechanism needs controlled experiments such as resynthesized pitch.
That the reference is the only valid transcription: one annotator.
That counts are alignment-exact: tab 4 is an estimate.
Anything about control clips beyond their role as controls: Yoruba, French and Pidgin rows sit inside category totals.