HearAble internals

The shape

Two machines. Compressed audio goes one way, two lines of text come back. Nothing else crosses the wire.

The division of labour follows from what each machine can do. The receiver has the signal and the screen but a 4.4 kernel, a few hundred megabytes of flash and no room for a speech model. The mini PC has four x86 cores and cannot draw a pixel on the television. So the receiver taps and paints; the mini PC listens and recognises; and the link between them carries as little as possible.

From button to subtitle

The full sequence, in order, with the component responsible for each step.

Box runs on the receiver PC runs on the mini PC

1. Box plugin.py · WHERE_SESSIONSTART

   Enigma2 starts and loads the plugin

   The plugin instantiates its dialog, binds three remote-control keys through a global action map, opens its WebSocket and control server on port 8770, and starts a 20 ms poll loop and a 40 ms render loop. HearAble is off. That state is not persisted on purpose: subtitles are something the viewer asks for.

2. Box plugin.py → relay.py

   F4 is pressed

   Key code 62 arrives. The plugin paints the greeting for four seconds, marks the relay enabled, and increments an epoch — a counter that invalidates everything from the previous session, so nothing stale can arrive later and be painted. In the same action it sends a Wake-on-LAN magic packet to the mini PC's MAC, broadcast to 10.77.0.255 on UDP ports 9 and 7, repeating up to fifteen times at twenty-second intervals.

3. PC hearable-t9.service

   The mini PC wakes and starts the pipeline

   The NIC wakes the machine in about 32 seconds. systemd brings up hearable-governor first — it is a hard dependency, not a nicety — then launches scripts/hearable_t9_live.sh, which execs python3 -m hearable.realtime_server with the private-link source and sink configurations.

4. PC sf8008/relay_source.py

   The receiver announces itself

   The mini PC opens a listening socket on port 9010 and starts polling the box's control endpoint every 500 ms:

   GET http://10.77.0.1:8770/control?receiver=10.77.0.2%3A9010
       &want_lead_ms=1500&lead_s=<measured>

   The poll is simultaneously a request, an announcement and a heartbeat. A receiver that stops polling disappears after an 8 s TTL and the relay shuts itself down — which is why an absent mini PC costs the decoder nothing at all: no process is started and no port is held open waiting.

5. Box relay.py → ffmpeg

   The box starts pushing audio

   Seeing a receiver, the relay reads the currently selected audio PID from the service and spawns an ffmpeg process outside Enigma2 that copies that one elementary stream, uncompressed-untouched, into MPEG-TS over a TCP connection out to 10.77.0.2:9010. About 26 kB/s. Nothing is re-encoded and nothing about this touches the user interface thread.

6. PC relay_source.py → ffmpeg

   Audio becomes chunks

   The receiver probes the first packets of each epoch for their timestamp and discards the audio ahead of the intended start — -c copy can only seek to a container boundary, so the stream always begins about 0.65 s early, and that excess is dropped rather than corrected through a feedback loop. What remains is decoded from AAC and resampled to 16 kHz mono, then cut into 160 ms chunks and queued (40 deep).

7. PC realtime_server.py → asr_capi.py

   The recogniser runs

   Each chunk is pushed into the streaming recogniser through a ctypes binding to NeMo-Speech.cpp. The model is cache-aware, so state carries between chunks; rnnt_right_context: 1 means it looks one block ahead before committing to a token. Compute is about 74 ms per 160 ms chunk at p50 — roughly half of real time, which is the whole reason a fanless N95 is enough.

8. PC stabilizer.py → led_subtitles/

   Words are stabilised and laid out

   The raw hypothesis is unstable by nature: a streaming model rewrites its own tail. The stabiliser holds a short history and only promotes a word once several passes agree on it. The LED formatter then packs the confirmed words into two rolling lines of at most 40 characters, with the last word held back briefly so a word still being revised does not flicker in front of the reader.

9. PC subtitle_sinks.py + enigma2_protocol.py

   Two lines go back over the WebSocket

   A subtitle_state message carries the two display lines, a monotonic sequence number and the epoch — and nothing else. The validator actively rejects a cumulative transcript field, because sending the growing transcript would make traffic and memory quadratic in session length. The producer never waits for an acknowledgement.

10. Box wsserver.py → subtitle_session.py → plugin.py

    The panel is painted

    A hand-written WebSocket server — the box has no WebSocket library, and installing one into set-top flash to receive two lines of text would be out of proportion — hands the frame to the session, which keeps exactly one pending state and discards anything from an old epoch or an older sequence number. The render loop paints it at up to 25 fps onto a black band at the bottom of the screen.

11. PC hearable_idle_watchdog.py

    And afterwards, it all goes away

    When the television stops sending audio, the watchdog starts a timer and powers the mini PC off. Pressing F4 again paints a four-second farewell carrying the version number; switching the receiver off entirely sends a farewell datagram so the mini PC does not even wait out its grace period.

The audio path in detail

One PID, off an already-tuned service, compressed all the way to the recogniser's doorstep.

The tap

Enigma2 holds twenty-four PID filters open on demux0 while showing a channel. HearAble opens a twenty-fifth. Userspace only — no kernel module, no patched image, nothing persistent on the box.

open("/dev/dvb/adapter0/demux0", O_RDWR | O_NONBLOCK)
DMX_SET_BUFFER_SIZE   1 MiB
DMX_SET_PES_FILTER    pid      = <selected audio PID>
                      input    = DMX_IN_FRONTEND
                      output   = DMX_OUT_TSDEMUX_TAP
                      pes_type = DMX_PES_OTHER
                      flags    = DMX_IMMEDIATE_START
read()                188-byte MPEG-TS packets, 132.8 kbit/s

The two channels of hearable-audio-relay/1

Control and media are deliberately separate connections, in opposite directions, so a slow or dead receiver can never stall the user interface.

The control response is a stable JSON document: protocol name, source_type (dvb for live television), the service reference, an source_epoch that increments on every discontinuity, the relay state, the selected audio track and PID, and the lead currently in force.

Recognition and stabilisation

The model produces an unstable hypothesis many times a second. Turning that into something a person can read is a separate problem from recognising it.

Model

Nemotron 3.5 ASR Streaming 0.6B, q8_0, 741 548 352 bytes, sha256 a5c435f2…f429ae

Runtime

NeMo-Speech.cpp at 4f96762 on ggml, CPU backend, built natively on the machine

Chunk

160 ms · rnnt_right_context: 1 · language it-IT

Compute

74 ms per chunk at p50 — RTF 0.4848 over thirty minutes of live television

Governor

performance, set by its own unit. Worth 27.6 % of the real-time factor on this bursty duty cycle

The stabiliser

A streaming RNN-T revises its own tail constantly: "andiamo a Ro" becomes "andiamo a Roma" becomes "andiamo a Roma sì". Showing that raw would be unreadable. The stabiliser keeps a short history of hypotheses and promotes a word to stable only once several consecutive passes agree on it at the same position. The unconfirmed tail is not displayed at all.

The two-line formatter

Confirmed words are packed into a roll-up display of two rows, at most 40 characters each — the geometry that reads as one block from across a room. Three parameters govern when a word is allowed onto the screen:

Where the second goes

Measured end to end: p50 1.072 s from audio consumed to subtitle state emitted (n = 1461, p05 0.893 s).

Text back to the screen

A small protocol with two rules, a hand-rolled server, and a renderer that can never accumulate a backlog.

hearable-enigma2/1

One JSON object per WebSocket text frame, over the private link. The producer connects out; the renderer listens. No TLS and no authentication — the cable is the trust boundary, and the port must not be exposed beyond it.

Rule one: never send a cumulative transcript. validate() rejects final_text and raw outright. Rule two: the producer never waits. Acknowledgements are observational. There is exactly one pending state and a newer one replaces it, so a renderer that falls behind loses freshness rather than accumulating a queue.

The renderer

The session drops any state whose epoch is older than the current one, or whose sequence number is older than the last rendered — the two mechanisms that make a channel change, a restart or a reconnection safe. A separate rule takes the panel down when the producer goes quiet for 8 s or disconnects entirely, because a test message once sat on a television for hours.

Drawing on the television

The panel is an Enigma2 skin generated at runtime: two centred white lines on a black band at 85 % opacity, each about 7.5 % of screen height, touching, sitting 7 % above the bottom edge.

The OSD canvas is 1280 × 720 even on an HD service. Five look parameters are configurable at runtime in /etc/enigma2/hearable_look.json — font_scale, line_gap, background_opacity, background_enabled, padding — and a change destroys and re-instantiates the dialog rather than restarting Enigma2, which would blank the picture.

What makes it an appliance

Nobody logs into the mini PC. It arrives when the button is pressed and leaves when the television goes quiet.

The private link

An /etc/network/interfaces.d/hearable stanza gives the NIC a static 10.77.0.2/24 with no gateway and no DNS, so it can never become the default route, and re-arms Wake-on-LAN on every boot because the r8169 driver clears it:

auto enp1s0
iface enp1s0 inet static
    address 10.77.0.2
    netmask 255.255.255.0
    up /usr/sbin/ethtool -s $IFACE wol g || true

Nothing on the subtitle path goes through the house network: no router, no DHCP, no name resolution. The house Wi-Fi going down cannot take subtitles with it.

The idle watchdog

It asks two questions and needs both to act: is anything connected to port 9010 (read from /proc/net/tcp), and is the decoder still watching (its /api/powerstate, falling back to a ping)? With no audio session it starts a timer and powers down; a returning session cancels it.

The status ring

A CH340 serial bridge on the case, named by a udev rule so it does not depend on enumeration order. The protocol is a five-byte frame at 10 000 baud: 0xFA, mode, brightness, speed, checksum — with brightness and speed inverted relative to intuition.

Every file involved

What the system is made of, and what it puts on the two machines.

On the receiver — the plugin

Copied to /usr/lib/enigma2/python/Plugins/Extensions/HearAbleOSD/. Six files plus one shared module deployed from the project rather than duplicated, so the two ends of the protocol cannot drift apart.

On the mini PC — the pipeline

The repository is checked out at /home/hearable/hearable. These are the modules on the live-television path.

Four further modules under hearable/sf8008/ — supervised_reader.py, box.py, byte_position.py, frame_match.py — implement sf8008_usb, an earlier and superseded source type that required the receiving machine to hold a bit-identical copy of the media, so the decoder's play position could be recovered by matching a frame off its screen. Nothing on the current path uses them.

Local media takes the same path

A film played from the receiver's own storage — a USB stick, an internal drive — is subtitled by the same chain, with no second code path. The relay is a second reader of whatever the decoder is playing: the demux for live television, the file for a file. Either way it forwards only the selected audio track, compressed, from ahead of the viewer's position. The receiver sees source_type: file instead of dvb in the control response and carries on.

FILE_STALE_SUBTITLES: 0 across the run — nothing from before a pause, a seek or a change of file was ever painted afterwards.

Measured with the film on the box's USB storage (/media/hdd). An NVMe or internal drive is the same path as far as HearAble is concerned — it reads whatever Enigma2 is playing and never touches the storage itself — but that has not been measured and is not claimed here.

Configuration and deployment

Files the system creates on the machines

Nothing else on either machine is modified. Everything below is written by HearAble itself.

The receiver's own configuration is never edited. No Enigma2 setting is changed, no init script is added, no kernel module is loaded, and the tuner is never opened by HearAble. Removing the plugin directory and the three hearable_*.json files returns the box to stock.