Compiler Pipeline

The MCL compiler is a 6-stage pipeline defined in core/pipeline.mtx. It converts intent.draft (prose + slot pre-fills) into intent.compiled (a fully-typed Intent IR). The Go binary mclc is a runtime that interprets the .mtx pipeline; it contains no compile logic itself.

Source files: core/pipeline.mtx, cmd/mclc/main.go, mtx/interpreter/.

Pipeline overview

intent.draft
    │
    ▼
┌─────────────────────────────┐
│  Stage 1: Ingest            │  Parse prose + slot pre-fills
│  core/pipeline.mtx          │  Normalize, extract raw surface
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  Stage 2: Classify          │  Match verb (D7 closed vocab)
│  core/verb.mtx              │  Grammar-constrained LLM call
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  Stage 3: Frame Extract     │  Fill the Frame: objects, constraints, criteria
│  core/frame.mtx + SKILL.mtx │  Grammar-constrained LLM call
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  Stage 4: Pre-Resolve (D13) │  NL refs -> matrix:// URIs
│  core/pre_resolve.mtx       │  Unresolvable -> Unknown
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  Stage 5: Confidence Score  │  Compute confidence (0..1)
│  core/confidence.mtx        │  Below threshold -> clarify questions
└─────────────────────────────┘
    │
    ▼
┌─────────────────────────────┐
│  Stage 6: Emit              │  Assemble Intent IR
│  ir/intent.go               │  Canonical JSON hash (D11)
└─────────────────────────────┘
    │
    ▼
intent.compiled

Stage 1: Ingest

The ingest stage parses the raw prose from intent.draft.prose and any pre-filled slot values from intent.draft.slot_values. This is pure data normalization; no LLM call is made.

Input: IntentDraftBody.Prose, IntentDraftBody.SlotValues

Stage 2: Classify

The classify stage matches the user's intent to one of the D7 closed verbs. This is a grammar-constrained LLM call: the model is constrained to emit exactly one of the 10 verb names or an x: extension.

The verb classifier rules are declared in core/verb.mtx. The compiler model (SlotCompiler) runs at temperature=0 with seed=42 for determinism (D11).

If the caller provides a -verb flag (CLI) or the draft carries a verb hint, this stage is skipped.

Stage 3: Frame Extract

The frame extraction stage fills the Frame's typed slots: objects, constraints, success_criteria, and preferences. This is the primary LLM call where the compiler model produces a structured JSON object matching the intent_frame@1 grammar.

The prompt templates come from the matched skill's SKILL.mtx on-blocks. Each verb branch has its own prompt with slot interpolation:

on verb=build
  prompt
    system="You are the Matrix plan compiler. Fill the Frame for a 'build' intent."
    user="User goal: {prose}. Context: {cortex.bundle}. Known slots: {slots}. Fill the Frame JSON."
  end
end

Slot interpolation variables: {prose}, {verb}, {cortex.bundle}, {slots}, {slot.<name>}.

Stage 4: Pre-Resolve (D13)

The pre-resolution stage resolves all natural-language entity references to matrix:// URIs before user sign-off. This is a mandatory step (D13).

Resolution calls:

Call	Purpose
`cortex.find(type=T, near=expr, limit=n)`	Semantic search over cortex memory
`cortex.resolve(expr)`	Exact resolution by NL hint or partial URI
`cortex.context(verb=v, budget_tokens=n)`	Cold-start bundle (3-tier)

Unresolvable references are declared as Unknown with severity blocking, preferred, or optional.

Stage 5: Confidence Score

The confidence scoring stage computes an overall confidence value (0..1) based on the filled slots, resolved references, and unknown count. The formula is declared in core/confidence.mtx.

When confidence falls below a threshold (typically 0.75), the compiler emits intent.clarify with structured questions for the unmet unknowns. The user responds with intent.answer carrying slot patches (RFC 6902 JSON Patch per D8).

Stage 6: Emit

The emit stage assembles the final Intent IR from the filled Frame, resolved references, declared unknowns, and metadata. The Intent is encoded to canonical JSON with sorted keys (D11), and its sha256 hash is computed.

The CompileMetadata struct records the full compilation trace for replay verification:

type CompileMetadata struct {
    Seed               string  // sha256 of (intent.id || actor || snapshot_hash || mtx_digest || model_digest)
    MtxDigest          string  // sha256 of canonical SKILL.mtx + core/*.mtx ASTs
    ModelDigest        string  // digest of the compiler model
    ModelVersion       string  // model identifier
    Temperature        float64
    Grammar            string  // "intent_frame@1"
    SkillID            string
    SkillVersion       string
    CortexSnapshotHash string  // Merkle root at compile time
}

Determinism (D11)

The compiler seed is computed as:

seed = sha256(intent.id || actor || snapshot_hash || mtx_digest || model_digest)

Given the same inputs, the compiler produces byte-identical output. This is enforced by:

temperature=0 on the compiler model
seed parameter passed to the provider API
Grammar-constrained decoding (JSON schema) so the model can only emit valid IR
AST-hashed .mtx files so comments do not break the seed