Barge-in

Four states and one arbitration

The gateway tracks four device states — idle, listening, thinking, speaking — and publishes every transition to the browser as a state frame. Ordinary transitions are unremarkable. The interesting edge is the one that leaves speaking, because the gateway cannot know from the audio alone whether it should be taken.

Three frames, and a decay instead of a reset

While the assistant is speaking, the browser raises its bar twice: the RMS requirement goes from SPEECH_THRESHOLD (0.0055) to 0.02, and a separate peak requirement of BARGE_IN_PEAK_THRESHOLD (0.09) applies. Both must persist. The counting rule is the part worth copying:

if (peak >= BARGE_IN_PEAK_THRESHOLD) {
  bargeInFrameCountRef.current += 1
  if (bargeInFrameCountRef.current >= BARGE_IN_FRAMES) {
    console.info(`[voice] barge-in! peak=…  rms=…  frames=…`)
    bargeInFrameCountRef.current = 0
    stopPlayback("interrupt")
    resumeRecognitionAfterPlayback()
  }
} else {
  // Quiet frame — decay count so noise spikes don't accumulate.
  bargeInFrameCountRef.current = Math.max(
    0,
    bargeInFrameCountRef.current - 1
  )
}

The constants are annotated in the source with their derivations, which is unusual and worth preserving here verbatim:

The SILENCE_MS comment is a small piece of architectural history: a client-side timeout was shortened by 900 ms because a server-side filter cascade (Figure 5) took over the job it had been doing. The two layers were tuned against each other.

Four places the assistant's own voice is stopped

Speaker to room to microphone is a physical feedback path that no amount of application logic can remove. Always attacks it at four points, two of them acoustic and two of them semantic.

The implementation is defensive in a way that suggests it has failed in the field: it checks that the destination produced audio tracks, that the hidden <audio> element actually played, and that the AudioContext reached the running state, resuming it twice if necessary. Any of those failing sets aecLoopbackReadyRef back to false and falls back to ordinary playback — with trap 1 silently gone and traps 3 and 4 carrying the whole load.

Comparing what was heard to what was just said

When acoustic cancellation fails, the assistant transcribes itself. _looks_like_self_echo() catches that by comparing the new transcript with self.last_answer. Both strings are reduced to lower-case alphanumeric words, and three rules are tried:

normalized_transcript = cls._normalized_echo_text(transcript)
normalized_answer     = cls._normalized_echo_text(spoken_answer)
if not normalized_transcript or not normalized_answer:
    return False
if normalized_transcript == normalized_answer:          # rule 1 — identical
    return True
if len(normalized_transcript) >= 24 and normalized_transcript in normalized_answer:
    return True                                      # rule 2 — a long substring
transcript_tokens = normalized_transcript.split()
answer_tokens     = set(normalized_answer.split())
if len(transcript_tokens) < 4 or not answer_tokens:
    return False
overlap = sum(1 for token in transcript_tokens if token in answer_tokens)
return (overlap / len(transcript_tokens)) >= 0.85          # rule 3 — bag of words

The check only runs when barge_in_candidate is set, so it costs nothing on an ordinary turn and cannot suppress an unrelated utterance spoken while the assistant is silent. It also runs before the early-transcript callback that shows the user what was heard: an echo that is going to be discarded never appears in the interface at all.