Always signal stack

Reading the source of iangogentic/always

One microphone,
four signal paths.

Always is a single-user, self-hosted voice assistant that runs on one desktop. A browser streams 16 kHz PCM into a FastAPI gateway; a Silero-VAD-gated recogniser turns it into text; a long-lived Rust agent process answers; Kokoro speaks the answer back. The part that is not ordinary is the fourth path — a second, listen-only socket that lets the assistant speak when nobody asked it to, because a subagent it dispatched minutes earlier just finished.

first-party Python 15,908 lines TypeScript/TSX 39,862 vendored Rust 390,138 git history 1 commit package version always 0.1.0

01 — Orientation

What the code actually is

The README calls Always a “self-hosted, voice-first AI assistant.” The source is more specific than that. Four programs, three languages, and one machine:

  • A FastAPI gateway (gateway/server.py, 4,826 lines) that owns the browser WebSocket, buffers microphone audio, calls speech recognition, forwards text to the agent, and streams synthesised audio back. It exposes 26 HTTP routes, one /ws endpoint, and one WebRTC offer endpoint.
  • A voice worker (voice-daemon/jcode_voice_worker.py, 1,010 lines) that keeps exactly one Rust agent process alive behind a five-verb unix-socket protocol so a voice turn does not pay harness startup cost.
  • A vendored Rust agent harness (jcode/, version 0.12.2, 786 .rs files, 390,138 lines, a 50-crate Cargo workspace) that does the actual reasoning and tool use. It is third-party code with a small local patch — see The harness.
  • A Next.js browser client (apps/zola/, a fork of the Zola chat UI) whose voice logic lives in one 3,141-line React component.

Everything talks over loopback or unix domain sockets. Nothing in the repository requires a hosted API; the model endpoints are OpenAI-compatible URLs on 127.0.0.1.

gateway/server.py
4,826lines, one file
HTTP routes
26+1 WebSocket, +1 WebRTC
worker ops
5ping · shutdown · ensure_session · history · chat
WS message types
8 in / 34 outclient → gateway → client
python tests
803 unittest files
systemd units
6all --user

Sourcing

Every number and identifier on this site was read out of the working tree, not the README. Where the documentation and the code disagree — and they do, in eleven places — the code wins and the disagreement is recorded on Drift.

02 — Topology

Four paths through one machine

Audio does not make a single round trip. It makes three, plus a fourth that has no user event at its head at all. Reading the four as separate signal chains is the fastest way to understand the codebase, because each chain is owned by a different file and each has its own failure mode.

End-to-end signal path through the Always stack Four horizontal signal chains: capture from browser microphone through the ASR shim to a transcript; reasoning from the session engine through the voice worker to the jcode agent process; speech from streamed text deltas through Kokoro back to the browser speaker; and the ambient chain, in which a second listen-only socket to the same agent process carries unsolicited narration to the same speaker. Path 1 — capture · browser to transcript browser mic PCM16 · 16 kHz WebSocket /ws gateway :8790 VoiceSession append_pcm16() joined_audio() highpass · normalise asr-shim Silero VAD gate Parakeet tdt-0.6b-v3 :5092 transcript 5-stage filter no speech → empty transcript, Parakeet never called transcript text handed to ConversationSession.run_text_turn() Path 2 — reason · transcript to token stream run_text_turn session_engine.py worker socket one per turn jcode_voice_worker op = chat, stream JcodeServerClient pooled, persistent jcode serve Rust · AF_UNIX host model OpenAI-compatible text_delta streamed events each delta appended to a sentence buffer as it arrives Path 3 — speak · token stream to speaker chunker sentence or 260 ch _flush_foreground_tts under a turn lock Kokoro :8880 · af_heart PCM → WAV 24 kHz mono audio_chunk base64 over WS speaker foreground queue index 0 clears the queue first Path 4 — ambient · no user event at the head of this chain jcode serve same process, same session AmbientListener a second socket on_event() buffer + narratable? _ambient_flush Kokoro again audio_chunk ambient: true ambient queue never clears path 3 same jcode serve process — path 4 attaches a second, listen-only connection to it both queues feed one speaker Amber = the two processes that appear in more than one path. Teal = the ambient chain, which has no request/response shape at all.
Figure 1. The whole stack as four chains. Paths 1–3 are an ordinary request/response voice loop split across three files. Path 4 is the design's actual novelty: the gateway holds a second connection to the same agent session, reads events nobody asked for, and speaks them. Sources: gateway/server.py, always_assistant/session_engine.py, voice-daemon/jcode_voice_worker.py, voice-daemon/ambient_listener.py, asr_shim.py.

Why path 4 needs its own socket

The chat client (voice-daemon/jcode_server_client.py) only reads its socket from inside a chat() call. Events that arrive between turns sit in the kernel buffer until the next turn reads them — and are then attributed to the wrong turn. Rather than demultiplex one socket between two consumers, the shipped design opens a second one. That decision, and the attempt that preceded it, are the subject of Ambient.

03 — Deployment shape

Where it runs

There is no container, no orchestrator, and no cloud dependency in the repository. Six systemd --user units in systemd/user/ start six long-lived processes on a single Linux desktop; three of those processes are model servers whose weights are deliberately not checked in.

Service, port and socket map for a single Always host A dashed boundary marks one Linux host running six systemd user units. Inside, HTTP services are bound to loopback ports: the gateway on 8790, Kokoro text to speech on 8880, a Qwen3-ASR llama.cpp server on 19070, the ASR shim on 8788, Parakeet on 5092 and an EvoCUA vision model on 19110. Unix domain sockets connect the gateway to the voice worker and the voice worker to the jcode agent process. Only four of these components are first-party code in this repository. one Linux host · systemd --user · everything on 127.0.0.1 browser client apps/zola (Next.js 16) WebSocket or WebRTC always gateway uvicorn gateway.server:app :8790 jcode voice worker jcode_voice_worker.py jcode-worker.sock AF_UNIX, one connection per turn jcode serve Rust harness v0.12.2 jcode-server.sock ambient: gateway's own second socket to the same session asr-shim asr_shim.py · Silero VAD · :8788 Parakeet ASR parakeet-tdt-0.6b-v3 · :5092 Qwen3-ASR (llama.cpp) qwen3-asr.service · :19070 Kokoro TTS kokoro-tts.service · :8880 ASR (either) forwards TTS host model endpoint OpenAI-compatible · :8080 default jcode serve calls the model directly desktop-delegate EvoCUA-8B · :19110 not reachable from the voice stack Amber = first-party long-running processes. Everything on the right is a model server whose weights are gitignored; the repository ships only the unit file.
Figure 2. Service and port map, reconstructed from the six unit files in systemd/user/ plus the defaults in always_assistant/config.py. The dashed ASR edge matters: the gateway's default recogniser URL is the Qwen3-ASR llama.cpp server on :19070, and the Parakeet shim on :8788 is opted into by overriding ALWAYS_QWEN3_ASR_URL. The shim exists precisely so the gateway need not know the difference.
Ports and sockets, from the unit files and config defaults
EndpointBoundStarted byCode in this repo?
Always gateway:8790always-satellite-gateway.serviceyes — gateway/
Voice workerjcode-worker.socksupervised by the voice daemonyes — voice-daemon/
jcode agentjcode-server.sockjcode serve, spawned by the workervendored — jcode/
ASR shim:8788 (per its own header)no unit file ships hereyes — asr_shim.py
Parakeet ASR127.0.0.1:5092externalno
Qwen3-ASR127.0.0.1:19070qwen3-asr.serviceno — llama.cpp + GGUF
Kokoro TTS127.0.0.1:8880kokoro-tts.serviceno — Kokoro-FastAPI
EvoCUA VLM127.0.0.1:19110evocua-llama.serviceno — llama.cpp + GGUF
Desktop indicator—pi-voice-indicator.serviceyes — Tkinter overlay

The gateway publishes two liveness endpoints. /health always returns 200 and reports whether the worker answered a ping; /ready returns 503 when it did not. Both echo the audio and auth configuration, which makes them the fastest way to confirm how a running instance is actually configured:

gateway liveness — gateway/server.py:3379 
$ curl -s http://127.0.0.1:8790/health | python3 -m json.tool
{
    "ok": true,
    "worker_ok": true,
    "sample_rate": 16000,
    "tts_enabled": true,
    "auth_required": true
}

$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8790/ready
200
worker_ok is the result of ping_worker(cfg) — a one-second {"op":"ping"} round trip on the worker's unix socket. auth_required reports the gateway's configured authentication state back to the caller, which is what makes /health useful as a deployment check rather than just a liveness one: it describes the instance, not the code.
04 — Inventory

The repository, with the dead wood marked

This is a working personal system rather than a released library, and it reads like one: several modules are the fossil record of earlier harnesses. The naming preserves the lineage — hermes_voice_worker (a Hermes CLI), then pi_voice_worker (a Pi Agent RPC loop), then jcode_voice_worker (the current one). Only the last is wired up; the HERMES_VOICE_* and PI_VOICE_* environment prefixes survive as compatibility aliases throughout always_assistant/config.py.

Top-level layout
PathWhat it holdsState
gateway/FastAPI app (4,826 lines), chat store, static fallback UIlive
always_assistant/reusable backend: config, asr, tts, audio_utils, session_engine, worker_clientlive
voice-daemon/nine modules: four live, one reachable only via the frontend, one orphaned, two legacy harness workersmixed
apps/zola/Next.js 16 chat UI fork; voice logic in one 3,141-line component; four always* API routeslive
jcode/vendored Rust harness, 50-crate workspace, 390,138 lines, plus iOS, Figma and demo mediavendored
asr_shim.py227-line FastAPI adapter that impersonates Qwen3-ASR over Parakeet, gated on Silero VADlive
desktop-delegate/817-line EvoCUA computer-use CLI; screenshots via the XDG portalstandalone
systemd/user/six unit files, all --user; each runs from a checkout rather than an installed wheellive
tests/three unittest files, 80 test methods, 74 of them in test_gateway.pylive
scripts/one end-to-end ambient smoke testcannot pass
bin/hermes-voice (a systemd/PulseAudio wrapper), desktop-delegate (a two-line passthrough)partly stale
“This module avoids all three by being a completely independent process-local concern. No pool. No shared sockets. No threads spawning threads. Pure asyncio, one task per WebSocket session, one socket, straight-line reader loop.”
voice-daemon/ambient_listener.py, lines 41–44 — verbatim

That paragraph is the most useful sentence in the repository. It is the module docstring of the file that replaced a design the documentation still describes, and it names the three specific failures that killed the earlier attempt. The whole of Ambient is an expansion of it.

05 — Scope

What this is not

Reading only the code, several things the surrounding documentation implies are absent from the tree:

  • There is no metrics or tracing backend. The gateway computes per-turn timings into a TurnResult.metrics() dictionary and logs [STAGE] lines; nothing exports them. No Prometheus, no OTLP.
  • There is no multi-tenancy. A device registry exists and mints per-device tokens, but every session shares one agent process, one model configuration and one home directory.
  • There is no waveform or VU display in the product. The browser computes a single RMS scalar and writes it to the width of one <span>. No AnalyserNode, no canvas, no FFT anywhere in apps/zola or gateway/static. The meter motif on this site is a reading device, not a screenshot.
  • The Rust is not the product. jcode/ is 6.7× the size of everything first-party in this repository and is a third-party harness. Always contributes roughly three patched files to it.

What is here, and works, is a complete local voice loop with an unusual fourth path bolted onto it. The remaining pages follow the signal in order.