The audio is already inside your set-top box

A gap that stays open

Broadcasters caption some programmes. The rest of the schedule is the problem.

My starting point was not a technical curiosity. A close friend of mine is hard of hearing, and Italian television captions the evening news and not much else. Live regional windows, sport, talk shows, most of daytime: nothing, or captions so far behind that they describe a sentence that finished half a minute ago. The practical result is watching people move their mouths.

The received wisdom is that fixing this is a broadcaster's job, because live subtitling means a respeaker in a booth or a speech stack in a data centre. That was true for a long time. It stopped being true quietly, when streaming ASR models got small enough to run on a CPU you can buy for the price of a games console — and almost nobody has gone back to check what that makes possible at home.

So I checked. The result runs on his television, adds about a second of delay, never touches the internet, and is switched on with one button on the remote he already had.

The rest of this is the interesting part: not that it works, but where the audio comes from, and what it cost to find that out.

The part that had to be invented

Everything downstream is ordinary engineering. Getting clean programme audio out of a closed box, for free, is not.

A recogniser needs audio. The obvious ways to get it off a set-top box each charge a price, and it took me a while to notice that all three prices were being paid in the same currency: delay.

A microphone near the television gives you the room — including, in this case, the feedback from the listener's own hearing aid. You would be handing the recogniser the worst possible copy of a signal that exists perfectly, uncompressed, three feet away inside a plastic case.

An HDMI audio extractor is cleaner and still wrong. The audio you capture there has already been decoded and presented. It comes out after the decoder's own output buffer — which is precisely the delay you are trying not to pay, bought a second time.

The streaming port that Enigma2 boxes expose on 8001 works on the first try, which is how it earns your trust before taking something. A client attached to that port holds the service. Zap to a channel on another multiplex and the box has no tuner left for the thing you were watching. I measured that, rather than reading it somewhere: it is a hard failure, on screen, in front of the person you built this for.

Then it occurred to me that the box is already demultiplexing the exact audio I want, right now, and Linux has had a standard API for reading it since the 1990s.

Five calls

Every DVB receiver running Linux exposes /dev/dvb/adapterN/demuxM. You open it, install a PID filter, and read. The box's own software is doing this twenty-four times over while it shows you a channel; you are asking to be the twenty-fifth.

/* the whole mechanism */
open("/dev/dvb/adapter0/demux0", O_RDWR | O_NONBLOCK)
DMX_SET_BUFFER_SIZE   1 MiB
DMX_SET_PES_FILTER    pid    = <the audio PID the viewer selected>
                      input  = DMX_IN_FRONTEND
                      output = DMX_OUT_TSDEMUX_TAP
                      flags  = DMX_IMMEDIATE_START
read()                188-byte MPEG-TS packets

No kernel module. No patched firmware. No root exploit. Userspace, on a stock image, with an API older than the box.

DMX_IN_FRONTEND is the line that matters. It says read the stream entering the receiver, not the stream the decoder is presenting. You end up upstream of the picture — which means the recogniser gets a head start instead of a handicap, and it means you are not in the video path at all, so you cannot break it.

What comes out is the selected audio track, still compressed, at about 26 kB/s. That number is the reason the rest of the design is small: you can push it down a cable to a machine that has no idea what a transport stream is, and all that machine needs to do is one AAC decode and a resample.

What it takes to run the recogniser

Less than you would guess, and the thing that matters is not the one you would tune first.

The model is Nemotron 3.5 ASR Streaming 0.6B, quantised to q8_0, running on NeMo-Speech.cpp. On a desktop it transcribes Italian at a real-time factor of 0.476 with a display word error rate of 4.74 % — half of real time with room to spare.

There is a GPU lesson worth having before you go shopping. A Radeon RX 6600 over Vulkan is 5.7× the CPU in raw throughput. Under pacing — fed audio at the rate audio actually arrives, which is the only mode that matters for live — the advantage collapses to 2.2×, because a GPU cannot overlap work it has not been given yet. A live pipeline is not a benchmark, and a throughput number will lie to you about it. What shipped has no GPU in it at all.

The stack is also more portable than it has any right to be: the same sources built on a completely different architecture, with a different compiler and different SIMD kernels, produce a bit-identical transcript — the same 74 words, WER 0.00 %. That is worth knowing before you worry about which box to buy.

The machine that did work, and the one line that made it

What shipped is a fanless Intel N95 mini PC — four cores, no GPU, no fan, the kind of box sold for digital signage. It reaches a real-time factor of 0.4848 over thirty minutes of live television.

I did not optimise a single line of compute code to get there. I changed a governor.

That is 27.6 % of the real-time factor, from one line of sysfs. The reason is specific to this workload and probably applies to yours: streaming ASR runs in bursts — about 105 ms of compute every 160 ms — and intel_pstate in powersave never sees a burst long enough to decide to clock up. It spends its life ramping.

There is also a GPU lesson hiding here. I did benchmark a Radeon RX 6600 over Vulkan, and in raw throughput it is 5.7× the CPU. Under pacing — feeding it audio at the rate audio actually arrives, which is the only mode that matters for live — the advantage collapses to 2.2×, because a GPU cannot overlap work it has not been given yet. A live pipeline is not a benchmark, and a throughput number will lie to you about it.

The second you cannot avoid, and the one I invented

Most of the delay is a decision. One chunk of it was a measurement error I carried for weeks.

Subtitles land about a second after the word. Here is the whole budget, because "about a second" is not an engineering statement:

More than half of that is spent deliberately not showing you something. That is the part I would defend hardest to anyone optimising for a lower number. A subtitle that appears fast and then rewrites itself in front of the reader is worse than one that appears later and stays put — and for a reader who is depending on the text rather than glancing at it, it is much worse. The unconfirmed tail is not displayed at all.

The 2.4 seconds that were 64 milliseconds

For weeks the pipeline held its subtitles back by an extra 1.5 s. I had measured the tap running 2.4 s ahead of the picture, so I was compensating.

I had measured it by screenshotting the decoder's framebuffer and correlating. That method has half a second of systematic doubt baked into it, and it was wrong by considerably more than that. The honest measurement is to ask the decoder directly — read AUDIO_GET_PTS from the audio device and the PTS of the packet the PID filter is handing you, in the same instant, packet in hand:

pairs.append((time.time(), pes_pts(packet), audio_pts(audio_fd)))
# the two ioctls are microseconds apart, so the difference
# between the two clocks IS the offset — no correlation needed

The real figure is 64 ms, spread 32 ms over a hundred samples. I had been wrong by a factor of forty, and in the direction that made the product worse: a second and a half of delay the system did not owe, added to compensate for a number that did not exist.

The person the system was built for had told me the subtitles felt late. I believed my instrument over the ear it was built to serve. The ear was right.

If there is one transferable lesson in this whole project it is that one. Every defect of consequence here was found by doing the real thing rather than its convenient imitation: unplugging a cable found a dead socket that ip link down could not reproduce; pressing buttons on the remote found two faults that a nineteen-case automated matrix had passed; pulling the mains answered a Wake-on-LAN question that months of software-off testing could not.

What it takes to build one

Two consumer boxes, one cable, and no subscription to anything.

The shape of the running system is deliberately boring. The box taps its own demux and pushes 26 kB/s of compressed audio down the private cable. The mini PC decodes it, runs the recogniser, stabilises the words into two lines, and sends those two lines back over a WebSocket. The box paints them on a black band at the bottom of the picture. Nothing else crosses the wire — no video, no transcript history, no telemetry.

Three details that took longer than they should have

• Enigma2's colour format is #AARRGGBB, and AA is transparency, not opacity. White text written as #ffffffff is the skin's own definition of invisible: my glyphs were drawn see-through and took the colour of whatever was behind them — green over a green bar, blue over a blue one. White is #00ffffff.
• Size the window to what you draw. A full-screen transparent overlay looks identical and takes the remote control with it. Volume kept working, because that is handled globally, but MENU did nothing at all. The television looked broken while the subtitles worked perfectly.
• The OSD canvas is 1280 × 720 even on a 1080p service. Lay the band out in 1080p coordinates and it lands below the bottom of the screen, where you will never see it and will spend an evening wondering why.

Making it an appliance, not a project

The difference between a demo and something a person actually uses every evening is that nobody should have to think about the second machine. So: pressing the button on the remote sends a Wake-on-LAN magic packet down the private cable, and the mini PC comes up in 32 seconds. When the television stops sending audio, a watchdog on the mini PC waits out a timer and powers it off. Switching the receiver off sends a farewell datagram so it does not even wait.

The viewer presses one key. The rest of the system appears and disappears around them.

It can be done at home, now

Not as a proof of concept. As the thing that is on in the evening.

What I find worth saying is not that this is clever — most of it is fifteen-year-old APIs used in the obvious order. It is that the capability arrived and the assumption did not move. Live subtitling is still filed mentally under "broadcaster infrastructure", and the actual requirement turned out to be a receiver, a fanless box the size of a paperback, and a cable.

There are tens of millions of people in Europe living with meaningful hearing loss, and a television schedule that is captioned in patches. The gap is not waiting on a research breakthrough. It is waiting on someone plugging the two boxes together.

What I would still like to know: whether this generalises to receivers I have never touched. The tap is standard Linux DVB, so it ought to. If you try it on different hardware, the measurement I would want back is the one from the box you own — first byte out of DMX_OUT_TSDEMUX_TAP, and how far ahead of AUDIO_GET_PTS it runs.