Capture

From ArrayBuffer to accepted text

The browser sends raw interleaved int16 frames on the same WebSocket that carries JSON control messages; the gateway distinguishes them by whether the received message has a "bytes" key. Conversion is one line: np.frombuffer(message["bytes"], dtype=np.int16).astype(np.float32) / 32768.0. From that point on, audio is a float array in [-1, 1] and every threshold in the system is expressed in those units.

Waking on hey mycroft

RemoteWakeDetector (gateway/server.py:195) is a thin, stateful wrapper over an openWakeWord ONNX model. It is worth reading closely because it does three things that a naive wrapper does not.

• It buffers to a fixed inference size. Incoming WebSocket frames are arbitrary lengths; append() accumulates into _pending and only calls model.predict() on exact 1,280-sample chunks — 80 ms at 16 kHz.
• It requires consecutive hits. A single frame over wake_threshold (0.55) is not enough; _consecutive_hits must reach wake_patience (2), and any below-threshold frame resets it to zero. Two frames is roughly 160 ms of sustained detection.
• It keeps a pre-roll ring. A 0.35 second trailing buffer is maintained at all times and trimmed sample-accurately by _trim_pre_roll(). When the word fires, the gateway calls consume_pre_roll(), clears the utterance buffer and re-seeds it with that audio — so the syllables spoken before detection completed are not lost.

required_hits = max(1, int(self.patience or 1))
self.last_fired = False
while self._pending.size >= self.chunk_frames:
    chunk = self._pending[: self.chunk_frames]
    self._pending = self._pending[self.chunk_frames :]
    pred = self.model.predict(float_to_int16(chunk))
    score = float(pred.get(self.model_name, 0.0))
    self.score_frames += 1
    self.last_score = score
    if score >= self.threshold:
        self._consecutive_hits += 1
    else:
        self._consecutive_hits = 0
    if self._consecutive_hits >= required_hits:
        self.last_fired = True
        return True

Detection state is fully observable: trace_snapshot() returns the model name, threshold, patience, required hits, current consecutive count, frames scored, last score and total audio seconds. The gateway logs it on every chunk while _wake_listening_allowed() is true, which is how threshold tuning was presumably done.

Every amplitude threshold in the system

Both ends of the connection measure the same two quantities — RMS and peak — on the same [-1, 1] float scale, and both make decisions from them. Collected on one ruler, the constants show the design: the client is permissive about starting, the server is strict about accepting, and the gap between them is where echo and room noise are meant to die.

# 2) Peak-normalize quiet browser audio so Parakeet doesn't treat real speech
# as silence. Do not boost tiny low-energy clips: that turns room noise into
# plausible one-word hallucinations such as "yeah".
peak = float(np.max(np.abs(audio))) if audio.size else 0.0
if peak > 1e-4 and peak < 0.7 and raw_peak >= 0.08 and raw_rms >= 0.006:
    audio = audio * (0.7 / peak)

A recogniser that pretends to be another recogniser

asr_shim.py is 227 lines and does something slightly mischievous. The gateway is written to speak Qwen3-ASR's API — an OpenAI chat completion whose user message contains an input_audio part. The shim accepts exactly that shape, throws away the text instruction, extracts the audio, forwards it to Parakeet's ordinary /v1/audio/transcriptions multipart endpoint, and reassembles the answer into a chat completion. Its own header says it plainly:

The reason it exists is the second half of the file. Parakeet 0.6B, given a near-silent clip, does not return nothing — it returns a plausible short English phrase. So the shim loads Silero VAD lazily via torch.hub, resamples to 16 kHz by linear interpolation, sums the durations of all detected speech segments, and refuses to call Parakeet at all unless that total reaches VAD_MIN_SPEECH_MS. The default failure mode is notable:

VAD_THRESHOLD      = float(os.environ.get("VAD_THRESHOLD", "0.45"))
VAD_MIN_SPEECH_MS  = int(os.environ.get("VAD_MIN_SPEECH_MS", "150"))
VAD_ENABLED        = os.environ.get("VAD_ENABLED", "1") == "1"
VAD_FAIL_OPEN      = os.environ.get("VAD_FAIL_OPEN", "0") == "1"

def _vad_error_result(reason, duration_s=0.0):
    if VAD_FAIL_OPEN:
        log.warning("VAD %s; fail-open forwards audio to Parakeet", reason)
        return True, duration_s, 0.0
    log.warning("VAD %s; fail-closed returns empty transcript", reason)
    return False, duration_s, 0.0

The shim's health endpoint returns the full gate configuration, including whether the model actually loaded. It is the only way to distinguish “VAD is filtering correctly” from “VAD never loaded and is silently rejecting everything”:

$ curl -s http://127.0.0.1:8788/healthz | python3 -m json.tool
{
    "ok": true,
    "upstream": "http://127.0.0.1:5092/v1/audio/transcriptions",
    "vad_enabled": true,
    "vad_loaded": true,
    "vad_fail_open": false,
    "vad_threshold": 0.45,
    "vad_min_speech_ms": 150,
    "parakeet_model": "parakeet-tdt-0.6b-v3"
}

$ curl -s http://127.0.0.1:8788/v1/models | python3 -m json.tool
{
    "data": [
        {"id": "qwen3-asr-1.7b", "object": "model", "owned_by": "parakeet-shim"},
        {"id": "parakeet",       "object": "model", "owned_by": "parakeet-shim"}
    ]
}

Four ways a returned string is thrown away

Audio-domain gates cannot catch a recogniser that returns confident nonsense. So transcribe_audio_qwen3() ends with a text-domain cascade — four predicates, each of which logs a distinct warning and returns the empty string. Every one names a specific observed failure.