Project 02 / Case study
Applied AI · desktop · fully local
A floating window that captures what is playing on the PC or what the microphone picks up, recognises it, translates it and draws the result on top of everything. Two neural models working on a desktop CPU. Not one network call, no account, no quota: once installed it runs offline.
Watching something in Japanese with Spanish subtitles depends on someone having made them beforehand. Live there are none, and the solutions that do exist take your audio to a third-party server. For a film that hardly matters; for a private meeting, or for whatever is playing on your own laptop, it does. I wanted the result without handing the audio to anyone and without depending on coverage.
Whisper Small quantised to eight-bit integers on CPU, with voice detection and no memory of the previous chunk. Japanese becomes English; if Spanish is requested, a second neural model makes the jump. One thread captures, another translates, and a two-slot queue decides what gets shown when the model cannot keep the pace of the audio.
That the work was not writing the five hundred and ten lines but deciding four things: the size of the chunk, forcing the CPU instead of detecting the hardware, splitting translation into two jumps, and never letting the model touch the interface thread. The value here is in the decisions, not in the volume.
Architecture
Two Python files and six dependencies with pinned ranges. Capture does not know the model and the model does not know the interface: they talk through a bounded queue and through Qt signals.
The source is chosen on screen: the microphone or the system capture loop, that is, whatever the PC is playing. Audio comes in at sixteen thousand hertz, is read in two hundred millisecond frames, is mixed to mono and accumulates until it reaches the chunk size. Four seconds is a measured compromise, not a round number: with two seconds there is less delay, but the model loses context and translates a long sentence worse; with six or eight it gains context at the cost of latency. Four is the point where a conversation stays followable.
The queue has a maximum capacity of two chunks. When it fills, the oldest is dropped and the new one comes in. It is the decision I like most in the project: if the model runs slow, the subtitle does not pile up or fall a whole minute behind, it simply shows the present. A subtitle from forty seconds ago is no use at all.
Recognition runs with faster-whisper on CTranslate2 and three parameters worth more than the whole code: voice detection on with three hundred milliseconds of minimum silence, beam of five, and memory of previous text turned off. That last one is what avoids the classic Whisper failure, where it starts repeating and inventing, clinging to what it said in the previous cut. The model is loaded on the worker thread, never on the interface thread, so the window responds while the model wakes up.
On top of the window: a rounded rectangle drawn by hand with QPainter, dragging to place it where it does not cover anything, and a "click-through" mode that lets the mouse reach what is underneath. Without that, an app like this gets in the way in two minutes.
Decisions
None of them was taken for taste. Every one has a rejected alternative and a cost I paid.
CTranslate2 detects an NVIDIA GPU on many machines, tries to use it and blows up because the compute libraries are not installed. The comment in the code says so with the initials spelled out: cublas64_12.dll. Pinning device="cpu" and compute_type="int8" gives up the speed of a card that might not be there, in exchange for starting on any laptop. Detecting the hardware and not knowing what to do with it is worse than not detecting it.
Whisper translates well into English; its direct output into Spanish is more uneven. So the audio always goes to English and a second neural model makes the final jump only if Spanish is requested. The cost is one more jump and some latency. The benefit is concrete: English keeps proper nouns and nuance better because the model does not have to do both things at once, and on the short route the recognised original remains, which is reviewable.
There are much lighter speech models that fit whole on a modest CPU. I ruled out the minimal variant after comparing it: the difference in quality was too large to swallow in a film with cross talk. I chose the heavy model and quantised it to eight bits, which is how you pay for quality without buying a graphics card. Quality drops; the hardware requirement drops far more.
The application applies no moderation and does not replace insults, and that is written in its documentation. A subtitle that changes what was said stops being a subtitle and becomes something else. It is a product decision, not an oversight: the aim was to reproduce content, not to sanitise it. I preferred to put it in writing rather than leave it for whoever uses it to discover.
Details you can see
Three small things that are exactly what give away whether someone has installed their own program on someone else's machine.
Validating the paths of the model file before unpacking. A translation package is a ZIP that comes from the internet with permission to write to your disk. The installer walks every entry, resolves it and checks that it lands inside the destination; if any points outside, it aborts with a clear message. Treating a downloaded file as trusted is a security mistake with a name of its own, and you do not have to discover it yourself.
Not showing the same line twice. Recognition sometimes returns the same text on consecutive chunks. If it is painted as it comes, the window flickers. The previous line is kept and the repeat is dropped: a detail invisible when it works and very visible when it is missing.
Saying what is happening at each moment. The status bar shows which model is loaded and which path is being used. When a result comes out odd, it can be blamed on a concrete cause instead of on a mystery. And Windows asks for microphone permission even to capture system audio: that too is documented, because if the user does not know it, the first run looks broken.
First, tests. Right now it has not a single one, and they can be written: a known audio chunk with its expected transcription, and a test that verifies the queue and interface chain with the model replaced by a test double. That is exactly the pattern I did apply in Storm the Castle to test logic that depends on a model that does not always answer. Second, cutting by sentence instead of by time: the four seconds split utterances in half and the translator loses context exactly where it needs it most. The right thing is to segment by silence detection or end of utterance, keeping the maximum duration as a ceiling. It is the improvement that would gain most for final quality, and I have had it written down since the first day I used it with a film.