How YouTube Auto Captions Work and Why Accuracy Matters

When a creator uploads a video, YouTube can run its speech recognition over the audio and produce a caption track on its own. According to YouTube's automatic captions help page, these captions are published automatically when they are available, so nobody has to type them or approve them first.

That is different from captions a creator uploads or edits by hand. A creator can write a caption file, correct the automatic one in YouTube Studio, or leave the machine version as it is. Viewers usually cannot tell which kind they are looking at, but the difference shows up later in this post: one YouTube setting that masks words applies to automatic captions only, not to captions that were manually edited.

Either way, a caption track is more than a block of text. Each line comes with a start time and an end time, so the player knows when to show it. That timing is what lets a caption-based tool act at the right moment in the video. When the words in a line are wrong, or the timestamps drift away from the speech, everything built on top of the track is affected.

YouTube is open about the limits. The same help page says automatic captions might misrepresent the spoken content, and it points to things like accents, background noise, and people talking over each other. Anyone who has watched a fast-talking gaming video with captions on has seen this happen.

The company has also described progress. In a February 16, 2017 post, YouTube said more than one billion videos had automatic captions and that English caption accuracy had improved by 50 percent. That figure describes a relative improvement at one point in time. It does not say how accurate captions are today, and it does not cover other languages.

You will find many accuracy percentages on other websites. They disagree with each other, they were measured on different kinds of video, and we could not find a current primary source for any of them. So this post gives no number. A single percentage would also hide what matters for a family: a cooking tutorial with one clear speaker and a noisy multiplayer stream can produce very different captions.

A more useful question is whether the captions on the video in front of you are good enough. You can answer that yourself in under a minute. Turn captions on, play a stretch where people are talking, and read along. If the lines track the speech closely, the captions are probably usable. If names come out garbled, sentences are missing, or the text runs well behind the voice, treat anything that depends on those captions with more caution.

Automatic captions are not guaranteed. YouTube's help page lists several reasons a video may not have them:

• •The video is still processing, so captions are not ready yet.
• •The spoken language is not supported for automatic captions.
• •The video is too long.
• •The audio quality is poor or the speech is hard to recognize.
• •There is a long stretch of silence at the start of the video.
• •Several people talk over each other, or speakers switch between languages.

The quickest way to check is the player itself. YouTube says you can turn captions on only for videos that have them, and the CC icon shows up only when captions are available. If the icon is missing, a tool that depends on captions has nothing to read.

Sometimes a caption line contains a blank placeholder that looks like two square brackets around an underscore. That comes from a YouTube Studio setting called "Don't show potentially inappropriate words." Per YouTube, the setting is on by default for automatic captions and replaces words it considers potentially inappropriate with "[ __ ]".

Two details are easy to miss. The setting does not change the audio, so the word is still spoken in the video. And it does not touch captions that a creator edited by hand. The result is that the captions and the audio can disagree: the screen shows a blank while the speaker says the word out loud.

It also means two videos with the same language in them can look very different on screen. One creator leaves the automatic captions alone and viewers see placeholders. Another edits the captions by hand, and the setting no longer applies to that text. As a viewer, you usually have no way of knowing which one you are about to get.

YouTube publishes a long list of supported languages on its automatic captions page, running alphabetically from Afrikaans to Zulu. The list changes over time, so it is better to check Google's page than to rely on a copy.

A language being on the list does not mean every video in it gets good captions. Quality still depends on the audio, the accent, and how much people talk over each other. Our page on language support and caption availability covers what that means for filtering in languages other than English.

Mixed-language videos are a special case. YouTube names speakers who mix languages as one of the reasons automatic captions may be missing. A family that watches a bilingual creator, or a gamer who switches languages mid-sentence, should check the captions on those videos before assuming a caption-based tool will cover them.

A filter that works from captions can only act on what the captions say. Caption errors turn into filtering errors in a few predictable ways.

• •A missed word. If speech recognition hears a different, harmless word, there is nothing in the text to flag.
• •A false flag. If it mishears a harmless word as an offensive one, a filter may mute speech that was fine.
• •A timing offset. If the caption timing drifts from the audio, a mute can land a little early or a little late.

How the filter reads the text matters too. A simple word list matches strings and has trouble with spelling variants, slang, and meaning that depends on the sentence. We cover that in why word-list filters fail. Reading the caption text in context helps with those problems, but it cannot recover a word the captions never contained.

• •Look for the CC icon before you rely on any caption-based tool. No icon usually means no captions.
• •Turn captions on for a minute and skim them. If they are full of obvious mistakes, filtering will be less reliable on that video too.
• •Expect live streams, noisy gameplay, and videos with crosstalk to be the weaker cases.
• •Remember that "[ __ ]" in the captions means the word is masked on screen and still present in the audio.
• •If you are trying Bleepster on the Free plan, a video counts toward the 10 free videos after one minute of watching, and rewatching it does not count again.

More explainers on YouTube filtering are on the Bleepster blog.