Neo Developer Documentation

Neo is Matrix's default conversational agent: a recursive LLM tool-calling loop with cortex-backed memory, the shared MCP tool surface, swarm parallelism, autonomous Automatrix execution, and core_execute delegation to the MCL pipeline for rigorous / money-moving tasks.

This documentation is written for people working on Neo itself, extending the control loop, adding memory mechanisms, wiring new tools, or understanding how the conversational HTTP service works.

---

Contents

---

Repository layout

neo/
├── cmd/neo/
│   ├── main.go          # CLI entry: REPL or single -prompt turn
│   └── serve.go         # Production HTTP service entry
├── internal/
│   ├── agent/
│   │   ├── agent.go      # Control loop: Chat, tool dispatch, budget, compaction trigger
│   │   ├── capability.go # Capability surface: API/Is/IsNot/FailurePatterns resident rendering
│   │   ├── compaction.go # Summary generation + transcript trimming + verbatim validation
│   │   ├── prompt.go     # System prompt builder + ground truth injection
│   │   ├── reporter.go   # Say/Status/Notice interface
│   │   ├── validate.go   # High-entropy token verbatim validator
│   │   └── knowledge.md  # Embedded Paxeer grounding facts
│   ├── automatrixlog/
│   │   └── store.go      # Durable Automatrix completion inbox (in-app surprise results)
│   ├── automatrixsettings/
│   │   └── store.go      # Per-user Automatrix opt-in + alarm + per-day counter
│   ├── config/
│   │   ├── config.go     # Config struct, Default(), Load(), env overlay
│   │   └── kvx.go        # .kvx file parser (sectioned key/value)
│   ├── conversation/
│   │   └── store.go      # Durable turn log per conversation_id (JSONL on disk)
│   ├── delegate/
│   │   └── client.go     # core_execute HTTP bridge to the MCL daemon
│   ├── llm/
│   │   ├── client.go     # OpenAI chat-completions client with tools
│   │   └── message.go    # Message, ToolCall, Tool types + constructors
│   ├── memory/
│   │   ├── embedder.go   # Embedding backend selection (gateway, direct, hash)
│   │   ├── pager.go      # Memory controller: pinned, retrieve, procedural, write-back
│   │   ├── pattern.go    # PatternSpec schema + encode/decode/render
│   │   └── writeback.go  # Cortex write helpers (fact, outcome, pattern)
│   ├── notify/
│   │   └── notify.go     # Out-of-app completion ping (ntfy + Apprise)
│   ├── recall/
│   │   └── recall.go     # Conversational recall lane (embedded turn ranking)
│   ├── server/
│   │   ├── automatrix_run.go     # Autonomous opportunity dispatch + settlement
│   │   ├── automatrix_governor.go # Per-user opt-in + Chronos alarm lifecycle
│   │   ├── engine.go     # Process-wide dependencies + core_execute + swarm wiring
│   │   ├── media.go      # GET /media, POST /upload (machine-volume media plane)
│   │   ├── server.go     # HTTP mux: /chat, /events, /conversations, proxy catch-all
│   │   ├── session.go    # Per-conversation agent + supervisor + gate waiters
│   │   ├── sse.go        # Event broker: replay buffer + live fan-out + tap
│   │   ├── surface.go    # Workspace-surface classifier (terminal/browser/editor/search/media)
│   │   └── swarm.go      # Concurrent sub-agent orchestration
│   ├── task/
│   │   └── store.go      # Durable task-supervision ledger (survives restart/suspend)
│   ├── tools/
│   │   ├── surface.go    # Natural vs Escalate classifier
│   │   └── tools.go      # MCP manager: spawn, bind, dispatch, schemas + synthetic tools
│   └── writeback/
│       └── consolidator.go # Background consolidation pass (dual-model)
└── neo.frozen.kvx        # Frozen architecture spec (design contract)

---

The one-sentence contract

Neo takes a user message, runs it through a recursive tool-calling loop with cortex-backed memory, and returns a final answer. Anything that moves funds or needs a wallet signature is delegated to core_execute, which routes through the MCL pipeline with inline user approval. Neo never holds a signing key.

That is the whole point. It is the boundary between the conversational world and the rigorous, replayable, on-chain world.

---

Architecture highlights

Supervisor pattern

Every task dispatched through the HTTP service is wrapped in a persistent supervisor (session.superviseTask). The supervisor keeps at least one agent on the task until it is genuinely complete, interrupted by the user, or hits a hard ceiling. Non-clean exits (model errors, stalls, exhausted step budgets) trigger checkpoint, backoff with jitter, and respawn of a fresh agent over durable state. The ErrIncomplete sentinel distinguishes a task that did not finish from a genuine completion or a transport error.

var ErrIncomplete = errors.New("neo: turn incomplete (task not finished)")

Swarm (concurrent sub-agents)

The spawn_subagents synthetic tool fans a task out to multiple headless sub-agents that run concurrently, each in its own isolated context window with the restricted tool surface (full Natural, no money, no recursion). Heavy tool work stays in the sub-agents' windows; only a compact, model-readable digest returns to the parent. A no-recursion guard (swarmActiveKey) prevents sub-agents from spawning their own sub-agents.

Automatrix (autonomous execution)

Automatrix is Neo's proactive surprise-task system. A Chronos recurring alarm wakes Neo to scan cortex for opportunities. The governor manages per-user opt-in and alarm lifecycle. The runner dispatches a supervised, restricted-surface run on a background goroutine, decoupled from the wake request. Per-day caps and bounded retry (max 3 attempts) prevent churn. Only genuine completions are announced; failures silently re-pend.

Capability surface

The capability surface renders the agent's own architectural truth resident in the byte-stable system prefix: external API surface, is/is-not facts, tool inventory with one-line semantics, and accumulated failure patterns. It is derived from the self-model artifact and live tool schemas, never hand-written prose. Missing sections render as explicit UNKNOWN with a pull path, never fabricated facts.

Media plane

Generated and uploaded media (images, video, audio) live on the agent's own machine volume under MediaDir. The media MCP server writes generated outputs there; users upload inputs via POST /upload; and GET /media/<name> streams either back to the browser. Nothing leaves the per-user machine.

SSE event broker

The broker fans per-run events out to SSE subscribers with a replay buffer (512 events) so a client that subscribes after POST /chat still receives every event. A tap seam persists the durable workspace trace. The wire envelope is byte-compatible with the daemon's sseEvent shape.

---

Key locked decisions

These decisions are frozen in neo.frozen.kvx. Do not re-litigate them without an explicit spec version bump.