#Vector Search

Implementation status (2026-09-13): the optional Discovery navigator reuses the shared vector adapters and adds its own generation/approval lifecycle. Reviewed document coverage and synthetic serving latency gates pass; public semantic activation is subject to the gates below.

Vector search is disabled by default and never replaces SQL. Architecture and privacy invariants are accepted in ADR 0002.

#Explicit Configuration

{
  "search": {
    "vector": {
      "enabled": true,
      "provider": "qdrant",
      "endpoint": "http://127.0.0.1:6333",
      "collection": "k0smos_vectors",
      "allowed_sources": ["documentation:guide"],
      "embedding": {
        "provider": "feature_hash",
        "dimensions": 384
      }
    }
  }
}

Set K0SMOS_QDRANT_API_KEY through the environment/secret manager when Qdrant authentication is enabled. Never put it in tenant JSON. The endpoint is deployable infrastructure configuration; no request-time setting can change it.

The request shapes follow Qdrant's official point upsert, query, collection, and payload-index APIs. Tenant isolation uses indexed payload filtering as recommended for shared multitenant collections: Qdrant API reference and Qdrant multitenancy guide.

Run the bounded rebuild only after migrations/configuration and a health check:

php bin/console search:vector:rebuild --source=documentation --batch-size=16

This example is the Documentation feature-hash baseline. Documentation emits only its reviewed public manifest sources under doc/public; the full current catalog is English and other navigation locales use its configured fallback. General doc/, prompts, tenant configuration, reports, credentials and private notes are excluded. Discovery is a separate, explicitly enabled discovery:chunk consumer with module-owned generations; the generic rebuild command cannot write those generations.

#Offline Evaluation Baseline

The versioned corpus is tests/Fixtures/Search/vector-documentation-evaluation.json: five English queries with one labeled relevant document each. The executable evaluation uses the current 39 public English projections and compares the whole-query substring source with the bundled 384-dimensional feature-hash reference.

The added tenant-configuration disposition document makes the feature-hash reference miss the runbook target for one query (rank 10). The other four labeled targets remain in the top three. This is a small regression corpus, not a production semantic benchmark. Before production activation, record against the chosen real embedding model and live Qdrant deployment:

• Recall@k and nDCG against an expanded reviewed corpus;
• p50/p95 query latency and ingestion throughput;
• storage size and embedding/request cost;
• rebuild, tombstone, model-upgrade, service-unavailable, and stale-index tests;
• cross-tenant retrieval (must be zero foreign hits);
• operational backpressure, pause/resume, lag, and failed-record quarantine.

The five-query Documentation evaluation is accepted as contract evidence for ADR 0002. It is not product approval for tenant content. Discovery may implement explicit public adapters for Ecommerce products, Blog posts, published Pages and selectively approved document evidence. Production semantic activation of these sources requires at least 50 reviewed IT/EN queries, lexical-only versus semantic/graph comparisons, at least 10% relative nDCG@10 improvement on semantic queries, no exact-identifier regression and zero hard-constraint violations. Record denominators and the limitations of synthetic measurements.

Extend the isolation evidence to overlapping ids across two tenants, private and revoked documents, source/module disable, stale jobs and approval withdrawal. No foreign or ineligible content may appear in results, context, snippets, citations or graph paths. Recheck canonical publication before assembling model context and again before returning a response. The default deployment allowlist remains documentation:guide; discovery:chunk has an implemented provider but requires explicit deployment configuration and completed acceptance before use. Internal tenant records, private chat and raw connector payloads remain excluded.

#Pinned Ollama adapter

The core embedding boundary now accepts ollama as well as the explicit feature_hash baseline. Model/endpoint/credentials are resolved from the existing enabled integration scope (search.embedding by default). Generic vector configuration pins embedding.digest, embedding.dimensions and embedding.normalization; changing these identities requires a separate collection/rebuild. Discovery generation management belongs to the module's backoffice/CLI and is not activated by the generic configuration switch.

The adapter checks /api/tags before and after /api/embed, requires the exact configured tag and manifest SHA-256, rejects truncated/oversized or invalid vectors and uses unit normalization. Batches have at most 16 inputs of 4,000 characters, 256 KiB encoded input and 8 MiB response. Dedicated vector HTTP transport bounds time/connect/response size and disables redirects, retries and decompression; generic application HTTP defaults are preserved.

QueryEmbeddingProviderInterface distinguishes asymmetric retrieval queries. plain-v1, embeddinggemma-search-v1 and qwen3-search-v1 freeze exact prompt policies in the embedding descriptor identity. No model is selected, downloaded or enabled by resolving the adapter. Qdrant verifies collection dimensions and Cosine distance, enforces public/tenant/model/source/locale/metadata predicates and rejects mismatched returned payloads. Explicit immutable point identities can be physically removed in batches of at most 100. Exact point retrieval also verifies tenant/source/version/checksum/model/privacy and metadata parity before an owner can validate a generation; matching counts alone are insufficient. The public API contract is Qdrant retrieve points.

VectorDocumentProviderInterface::allowsDirectProjection() separates ordinary providers from owners that require generation/ledger writes. Documentation permits the generic rebuild command and typed handler. Discovery refuses both generic write paths and exposes only exact current immutable point identities through the registry. Generic rebuild batches now default to 16 to fit the bounded embedding transport. Large source documents still require owner chunking before using the Ollama adapter; silent input truncation is not supported.

Official API contracts: Ollama embeddings, model inventory and Qwen retrieval prefixes. The new adapter/configuration/Qdrant unit boundary passes 9 tests (49 assertions); these are mocked contract checks, not real-provider relevance evidence.

#Discovery data ownership and query flow

Qdrant is the implemented vector database adapter. Discovery reuses it instead of adding another datastore provider: it supports the required cosine search, scope filters, exact point verification and physical removal. This is a reuse decision, not a measured claim that Qdrant outperforms every alternative.

The tenant's existing SQL database remains authoritative for canonical content and for Discovery collections, document versions, approvals, graph assertions, generation/point ledgers, preferences and usage accounting. SQL graph edges are derived from current approved assertions; the human review decisions must be retained. No separate graph engine is introduced.

Media owns original files. Shared bounded extraction produces ordered text and locators; Discovery stores versions/chunks, requires selective approval and links them to eligible public content. Connector imports reach Discovery through their canonical Ecommerce/Blog/Page owners and committed content notices. Public search does not call upstream connectors or consume private chat history.

Projection splits canonical text into chunks of at most 2,000 characters and uses the approved structural document chunks. The Ollama provider turns them into vectors; Qdrant stores vectors and identity/provenance metadata, not original files or full chunk text. Discovery creates a separate Qdrant collection for each immutable tenant generation and still validates tenant, locale, source, generation, model identity and publication on retrieval. SQL keeps the exact point manifest and rebuild/cleanup intent. Changed model identities require a new generation, complete inventory, validation and an explicit serving switch.

The real relevance fixture used Qwen3-Embedding 0.6B Q8, 1,024 dimensions and the pinned qwen3-search-v1 prompt policy through Ollama. That measured model is an embedding configuration, not the visitor's chat model or an automatically selected tenant default. The earlier feature-hash reference is a lexical proof.

Visitor interpretation uses a separately configured discovery.interpret scope. It produces validated preference patches, not arbitrary catalog claims or SQL. DiscoverySearch combines lexical, document, vector and reviewed graph candidates; exact/preference bonuses and reciprocal rank fusion order them. Current owners and retained proofs are checked before returning result cards. The discovery.relations scope separately proposes evidence-backed graph links for an operator to review. Each scope has explicit spending controls.

See modules/Discovery/src/README.Discovery.md for operator flows and modules/Discovery/src/AI.Discovery.md for ownership and recovery contracts.

#Discovery measurement and remaining gates

Discovery's expanded semantic checkpoint (2026-09-13) uses a frozen 54-query IT/EN corpus with 36 canonical records, approved document extracts, graph paths and exclusion controls. Real pinned Qwen3 0.6B Q8/Ollama/Qdrant raises semantic nDCG@10 from 0.42384 to 0.69560 in the combined document/vector configuration, with no exact-match regression or observed hard/fixture-privacy violation. Private/foreign canonical controls are prefiltered by the synthetic leaf; owner SQL and provider isolation require the separate contract tests. This is synthetic, in-sample evidence; one unrelated no-result query still admits lexical false positives. The 0.40 cutoff is the measured fixture configuration, not an implicit tenant default. Expected document/parent/quote coverage now passes 4/4 for both vector and full retrieval. Canonical vector hits preserve later document proofs without adding duplicate ranking contributions. The persisted report includes document_evidence_expected, vector_document_evidence_matched, full_document_evidence_matched and degraded_queries; all expected document cases must match and degraded count must be zero. Older stored reports remain readable but cannot authorize activation without those measurements. Numeric ranking alone is diagnostic.

The separate 10,000-content / 100,000-point / 200,000-edge fixture now passes its warm IO serving budgets: 31 ms graph p95 and 367 ms retrieval p95, excluding embedding, after ordered SQL and point-identity index fixes. Canonical reads are synthetic and the scale vectors are reused; this is not production-owner SQL or fresh-model throughput evidence. MySQL separately validates SQL semantics, collation and reversible indexes. Activation still requires the configuration hash, current complete inventory, exact parity and an operator-reviewed report. Executed evidence, failed baselines and deployment checks are recorded in modules/Discovery/src/EVALUATION.Discovery.md.