What AI transcription actually extracts from a reel

"The AI reads the video" is the kind of sentence that sounds like an explanation and functions as a shrug. It tells you nothing about what will actually be findable afterwards, which is the only thing you care about when you are trying to dig out a reel you half remember.

What happens in practice is more specific and more useful to know. A reel is not one thing to be read; it is three separate sources of text, extracted by three unrelated techniques, with three different failure modes. A video can be rich in one and empty in the others. Once you know which signal carried which part of the content, the behaviour of every search you run starts making sense.

Signal one: the spoken audio

The audio track goes through a speech-to-text model, which returns the narration as a block of plain text. When a reel has a person talking, this is by far the densest signal available. Thirty seconds of speech is roughly seventy to a hundred words — more text than the caption, more than anything on screen, and more importantly it is the connected, explanatory kind of text. It contains the reasoning, the caveats, the "and the reason this works is", none of which anybody bothers to type into a caption.

Speech-to-text is remarkably good at ordinary connected sentences and noticeably bad at exactly the words you would want to search for. Proper nouns are the classic case: a person’s name, a neighbourhood, a supplement, a framework. The model is choosing the most probable sequence of words given the sounds, and a name it has effectively never encountered in training is never going to be the most probable anything. It emits something phonetically close instead — a real word, spelled confidently, sitting in your transcript looking entirely plausible.

Brand names suffer the same way, and code-switching is worse. A cooking reel where the narrator speaks English but names every spice in Gujarati will transcribe the English cleanly and turn the spice names into noise, which is precisely inverted from what is useful. And then there is the hard floor: a reel with no speech at all. A montage over a music bed produces a transcript that is empty, or that contains a hallucinated fragment of the song lyrics, and no amount of model quality changes that. There was nothing said.

Signal two: text burned into the frames

A large fraction of reels put their real content on screen and never say it out loud. The five-item packing list. The three numbers in the comparison. The name of the plugin, shown for two seconds in the corner while the narrator says "this one". If you only had the audio you would have the vibe of the video and none of its substance.

Reading that means treating the video as a series of images: pick frames, then run optical character recognition over them to pull out whatever text is rendered in the pixels. When it works it is the highest-value signal of the three, because on-screen text is deliberately the summary. Creators put things on screen precisely because they are the parts worth remembering.

It fails in ways that have nothing to do with how interesting the content is. Low-contrast text over moving video — thin white type over a bright kitchen — is the most common one; the characters are legible to you because you are watching them move, and marginal to a system looking at one still frame. Decorative and handwritten fonts, which short-form video loves, are another. So is text that is animated on and off within a few frames, or text that sits inside the platform’s own interface furniture and gets mixed in with the words that were actually part of the video. And OCR returns unstructured strings: a neat two-column table of exercises and rep counts comes back as a run of words with the column relationship gone.

Signal three: the caption

The caption is the only one of the three that was written by a human on purpose. It needs no extraction, no model and no interpretation — it arrives as text and stays as text. That makes it the cleanest signal by a wide margin. Spellings are correct, proper nouns are spelled the way people actually search for them, and there is no confidence score to worry about.

It is also, very often, the least informative. Captions are written to drive engagement rather than to describe content, so a great deal of caption text is a hook, a call to comment a keyword, a block of hashtags, and a credit line. A twelve-step recipe reel routinely carries the caption "you NEED to try this 🤯 recipe below 👇" — accurate, useless, and matching nothing you would ever type into a search box six months later.

The exception is worth noting, because it is common enough to matter: creators who paste the full recipe or the full instructions into the caption. When that happens the caption becomes the best signal in the entire reel, and no processing was needed to get it.

Why frame selection matters more than the OCR model

A thirty-second reel at thirty frames a second is nine hundred images. Practically all of them are redundant — the same shot, the same overlay, one frame later. Running OCR over all nine hundred would cost roughly nine hundred times what it needs to and would return the same sentence hundreds of times over.

So the interesting work happens before the OCR model is ever invoked: scoring frames and keeping a small representative set. The scoring is looking for the moments that carry information — where the frame changed substantially from the one before it, where the image is sharp rather than caught mid-motion-blur, where there appear to be regions of text at all. A handful of well-chosen frames covers everything the reel showed you.

This is why frame selection dominates OCR quality in practice. A superb OCR model handed nine hundred near-identical blurry frames produces an expensive mess of repeated fragments. A modest OCR model handed six well-chosen sharp frames produces a clean, focused block of text. If a reel’s on-screen list never made it into search, the likeliest culprit is not that the characters were misread — it is that the frame showing the list was never picked, because it flashed by in half a second between two visually similar shots.

Putting the three together

Categorization is a separate step that reads all three signals at once. That matters, because any one of them alone is a bad basis for a decision. The caption says "🤯 you need this". The transcript says "so basically what I do is". The on-screen text says "45g" and "no oven". Individually, none of that files anywhere. Together it is fairly obviously a baking reel.

It also explains why a sensible system attaches a confidence score to the category instead of stating a flat label. Confidence is a summary of how much agreeing evidence there was. A talking-head explainer with a descriptive caption gets a high score because all three signals point the same way. A silent aesthetic clip with an emoji caption gets a low one, and it should — there was almost nothing to go on, and the label is close to a guess. A score gives you a way to know which categories to trust and which to skim and correct.

One thing to state plainly about this codebase, since the article is about what actually happens rather than what could: the categorization engine here is pluggable, and under the CATEGORIZATION_ENGINE=LOCAL setting it returns mock data. That is the default, and it exists so the rest of the system — capture, storage, the dashboard, search over whatever text is present — can be developed and run without a GPU attached. Everything above describes what the pipeline does when a real engine is wired in behind that interface. If you are running the local default, categories are placeholders and you should read them as such.

What this means when you search

The practical upshot is that reels are not uniformly findable. Searchability is a property of the format, decided long before you saved it:

• A talking-head explainer is highly searchable. Continuous speech produces the most text, and it is the connected kind that matches the phrases you half remember.
• A text-overlay listicle is searchable only if frame selection caught the right frames. When it works it is excellent, because the on-screen text is already a summary. When it misses, the reel is nearly blank to search despite being full of specifics.
• A silent aesthetic clip is close to unsearchable. No speech, no overlay, a promotional caption — there is simply no text to match. Browsing by category is the only route back to it.
• A demonstration reel with narration and on-screen labels is the best case, because the two signals cover each other’s gaps: the name the narrator mangles is usually spelled correctly on screen.
• Anything where the creator pasted full instructions into the caption is reliably findable regardless of everything else, and costs nothing to index.

None of this is a reason to distrust automatic extraction — it recovers text from videos that would otherwise be entirely opaque, which is the whole point. It is a reason to hold a realistic model of what came out. Three signals, unevenly reliable, covering different parts of the content. When a search works you are usually riding the transcript. When it fails on a reel you know exists, the odds are it was a quiet one.