Deus - Discovery, Extraction, Embedding, and Index Construction

Overview

The discovery pipeline turns a free-text query into ranked service results. It extracts structured constraints from the query, generates optional semantic embeddings, merges vector and lexical candidates, filters by price and uptime, and blends a final score.

Discovery Service

Source: deus/internal/discovery/discovery.go

The Service owns query execution, candidate merging, and ranking. It is constructed with a store, an embedder, and ranking weights.

Search flow

1. Clamp Limit to 10 (default) when <= 0 or > 100
2. Call ExtractConstraints with query text and explicit filters
3. Resolve kind, max_price_wei, min_uptime_bps from extracted filters
4. Merge candidate rows by ID from semantic and lexical searches
5. Fall back to ListDiscoverCandidates only when both search paths produce nothing
6. Filter by uptime, then by price
7. Score with BlendScoreWithPrice, sort descending, truncate to limit

Result fields

Each DiscoverResult carries: ID, Slug, DisplayName, Summary, Kind, QualityScore, UptimeBPS, Score, and Operations (from the manifest pricing array).

Ranking weights

Configurable via WithRankingWeights. Default values:

Degradation model

The search pipeline degrades gracefully:

• Embedding failures do not stop lexical search
• Vector search failures do not stop lexical search
• Lexical failures do not stop browse fallback
• The only hard failure is when no candidates exist at all

Constraint Extraction

Source: deus/internal/discovery/extract.go

ExtractConstraints turns plain-language queries into structured filters:

Explicit filters win over text extraction. The semantic query is the remaining text after recognized phrases are stripped and normalized.

Embedding Implementations

Source: deus/internal/discovery/embed.go

HashEmbedder (default)

• Deterministic, non-semantic (768-dimensional)
• Semantic() returns false -- disables vector search
• Generates normalized vectors from SHA-256 seeds
• Used as development/test fallback

HTTPEmbedder

• Calls a remote embedding service (JSON POST)
• Semantic() returns true -- enables vector search
• 15-second HTTP timeout
• Default dimension: 768
• Response shape: data[0].embedding

Backend selection

NewEmbedderFromConfig returns HTTPEmbedder when the endpoint string is non-blank, otherwise HashEmbedder.

Search Document Construction

Source: deus/internal/discovery/index.go

BuildSearchDocument concatenates: DisplayName, Summary, Description, each Tags entry, and each Operations[].Name into a single trimmed string.

IndexService validates the manifest, writes the search document to the store (for lexical/tsvector search), embeds the document, and upserts the embedding vector (for pgvector HNSW search). A nil embedder short-circuits after the search document is stored, so lexical search works even without a semantic backend.

Catalog Endpoint

Source: deus/internal/server/handlers_discovery.go

GET /v1/catalog serves the public paginated catalog. It enriches each CatalogItem with headline pricing (first pricing[].price_wei + unit) and tags extracted from the stored manifest JSON. Response:

{
  "services": [...],
  "total": 42,
  "limit": 20,
  "offset": 0
}