feat(ingest): resolve project labels with Jev

docs/IMPLEMENTATION_PLAN.md

@@ -85,7 +85,7 @@ M0 exits with executable contracts and a disposable accepted source/note that ca
 | M1.2 Expose one application core | MCP/read HTTP: catalog-backed reads, upload/proposal/status/revision/withdraw/evaluate and run operations, bounded errors/diagnostics. | M0.4–M0.7, M1.1 | Actual transports enforce capabilities and scope; no generic HTTP/CLI knowledge writes or raw queries; accepted reads disclose canonical/index revisions; diagnostics contain no secrets. |
 | M1.3 Bridge selected local files | Connector: stdio MCP to service, explicit-file byte transfer, bounded upload resume, service identity checks and protocol-only stdout. | M1.2 | Selected file reaches service by bytes/hash, including a remote service; arbitrary VPS-local path interpretation fails; byte payloads and credentials stay outside model-visible responses. |
 | M1.4 Deliver scoped lexical reading | Index/query: FTS, typed edges, project/shared filters and `default_includes` resolution, canonical revision reads, neighbors/history, read-time freshness and synthesis dependency warnings, complete citation groups and tokenizer accounting. | M0.7, M1.2 | IDs/redirects/graph/citations obey scope; omitted project includes resolve defaults plus shared for one project context, explicit empty includes suppress defaults, and every expanded project is authorized; stale warnings survive budgets; too-small budget returns a defined error; index deletion/rebuild preserves authority and citations; index lag is disclosed. |
-| M1.5 Qualify hybrid retrieval | Embedding/vector/ranking adapters: model/native-package probes, scoped keyword/vector fusion, capped graph expansion, language fixtures and isolated index generations. | M1.4; retrieval/benchmark decisions | Top-five source recall meets the PRD target on labeled questions; old citations survive model/parser changes; offline assets work; keyword degradation is labeled. Real provider swap and generation rollback pass consumer contracts without search-consumer changes. |
+| M1.5 Qualify hybrid retrieval | Embedding/vector/ranking adapters: install the pinned `intfloat/multilingual-e5-small@614241f622f53c4eeff9890bdc4f31cfecc418b3` candidate and retrieval extra on the GEWU host; activate with `GEWU_EMBEDDING_MODEL_PATH` and verified `GEWU_EMBEDDING_MODEL_SHA256`; scoped keyword/vector fusion; capped graph expansion, language fixtures and isolated index generations. | M1.4; retrieval/benchmark decisions | The MSI/local GEWU deployment can load the model from a verified local path using its artifact digest and environment settings, then run offline. Source vectors record model/profile metadata and can be rebuilt or replaced without changing canonical records. Top-five source recall meets the PRD target on labeled questions; old citations survive model/parser changes; keyword degradation is labeled. Real provider swap and generation rollback pass consumer contracts without search-consumer changes. |
 
 M1.1–M1.4 support the pilot. M1.5 and all four native client flows are required to close full M1/v1. Authentication and enforced data isolation precede any non-disposable shared use.
 
@@ -98,8 +98,27 @@ M1.1–M1.4 support the pilot. M1.5 and all four native client flows are require
 | M2.3 Expand source intake | Fetch/parser adapters: URL/HTML, PDF, supported local files, pasted text and supplied NotebookLM exports; transcript locators where applicable; archived downloads, source identity/revision changes, reference-only failure records. | M0.4, M1.3; format limits | Exact locations verified on real fixtures; changed URL creates a revision; failed capture preserves prior bytes; scanned/encrypted/unsupported material returns honest status. Parser replacement retains old extraction resolution. Required input paths are qualified; unsupported document subtypes are listed explicitly. |
 | M2.4 Complete linking and consistency | Entity/relationship application services plus ingest skills: canonical aliases, duplicate candidates, reversible merge history, evidence/rationale-bound typed links, contradictions and supersession. | M0.5–M0.6, M2.1 | Shared dedupe preserves provenance; substantive links include conditions/support; similarity does not create accepted edges; destructive merges require authority; contradictions retain both claims. |
 | M2.5 Qualify native hosts and quality | Acceptance/client fixtures: Codex, Claude, Antigravity and OpenCode on declared supported OS pairs; five common documents through two agents; sampling and labeled coverage/error evaluation. | M2.1–M2.4; quality/coverage decisions | Each supported pair transfers, rejects invalid evidence, auto-accepts eligible work, retries/resumes and retrieves citations. Convergence compares findings/support errors; no pass based on matching prose or tool listing. |
+| M2.6 Control project facet vocabularies | Ingest and taxonomy: project-scoped canonical labels for tags, audience, use and domain; bounded agent proposals; exact/alias, lexical and embedding-based shortlist retrieval; persist label vectors in the derived SQLite index keyed to canonical label ID and embedding profile; Jev fit and reuse-versus-new decisions; review feedback and alias/merge history. | M1.5, M2.1, M2.4 | Server-enforced proposal limits apply per facet. For each proposal, candidate retrieval stays within authorized project/shared scope and supplies the closest existing labels to Jev alongside a new-label option. Label vectors remain linked to their canonical project labels and can be rebuilt when labels or model profiles change. Decisions distinguish label fit from reuse, and uncertain cases remain reviewable. Existing matches resolve to canonical labels; approved, sufficiently distinct new labels enter the project vocabulary and are reused on later records. Search through canonical names or aliases reaches the same records. Thresholds are tuned on labeled duplicate/near-duplicate/distinct-label examples; adding or merging a label preserves provenance and is reversible. |
 
-For the pilot, complete M2.1–M2.2 and the selected-format/client portions of M2.3–M2.5. Remaining matrix cells and source paths stay NOT RUN or BLOCKED until verified; they remain full-v1 obligations.
+For the pilot, complete M2.1–M2.2, M2.6, and the selected-format/client portions of M2.3–M2.5. Remaining matrix cells and source paths stay NOT RUN or BLOCKED until verified; they remain full-v1 obligations.
+
+#### M2.6 implementation sequence: controlled project labels
+
+The goal is to keep facet vocabulary reusable without preventing a project from adding a genuinely useful concept. Labels are scoped by project and facet (`tags`, `audience`, `use_for`, `domain`). Similarity retrieval creates a shortlist; Jev judges fit and reuse; GEWU applies the decision policy. A similarity score alone never merges labels or authorizes a new one.
+
+| Step | Work | Acceptance evidence |
+| --- | --- | --- |
+| 1. Install and activate local embeddings | On the GEWU host, install the `retrieval` extra and the pinned `intfloat/multilingual-e5-small@614241f622f53c4eeff9890bdc4f31cfecc418b3` model once. Configure `GEWU_EMBEDDING_MODEL_PATH` and `GEWU_EMBEDDING_MODEL_SHA256` in the service environment; keep the artifact digest check, CPU/local-only loading and offline operation. Do not put model installation on the Antigravity/client agent. | Service startup verifies the model digest and loads it; an embedding probe verifies its dimensions. With the local model installed, label and source embedding generation works offline. Unset settings retain lexical mode; incomplete or invalid settings fail clearly. The service never downloads weights at runtime. |
+| 2. Define the project label catalog and persistence | Add stable label IDs and project-scoped catalog entries with facet, canonical name, short definition, aliases, lifecycle status (`active`, `merged`, `retired`), provenance and timestamps. Keep the canonical catalog in GEWU's authoritative project data. Mirror label embeddings into a derived SQLite table keyed by label ID and embedding profile (model ID, artifact digest, dimensions and text digest); the catalog itself is stored with canonical project records, not duplicated as mutable SQLite state. Rebuild vectors from canonical data; never treat a vector as the label's source of truth. | Labels and vectors survive restart. Index rebuilding recreates vectors from canonical project data. A model/profile change cannot mix vectors from different profiles. Reads, proposals and shortlist queries enforce project/shared authorization. |
+| 3. Inventory and migrate current free-text facets | Scan existing knowledge for tags, audiences, uses and domains. Normalize exact duplicates conservatively (Unicode NFC, case and whitespace); preserve original strings as aliases and retain usage counts/provenance. Produce likely semantic duplicate groups for review, but do not automatically merge merely related terms. | Every current facet value maps to a catalog entry without losing node provenance or searchability. Exact duplicate normalization is auditable. Semantic near-duplicates are presented as candidates for owner review before merge. |
+| 4. Retrieve closest existing labels | For each agent proposal, search only the authorized project/facet catalog. Resolve exact names and aliases first, then combine lexical retrieval with cosine similarity over label name + definition + aliases. Embed the proposal's label + short meaning/context and return a bounded top-K shortlist to Jev. Start with a small K (for example, five); tune candidate recall on human-labeled synonym, near-neighbor and distinct-label pairs. Treat vector similarity as candidate retrieval, not equivalence probability. If embeddings are unavailable, lexical matches may be shown, but new-label automation stays off. | The correct existing synonym appears in the shortlist for the labeled evaluation set, including English/Chinese fixtures. Similar-but-distinct labels remain available as separate options. Shortlists never disclose labels outside the caller's allowed scope. |
+| 5. Update ingest skills and contracts | In `kb-ingest`, fetch the project's active vocabulary before classifying. Prefer canonical labels; propose a new label only when the source supports a reusable distinction, with a short definition. Enforce at most five tags and three labels in each other facet (14 total) in both skill and MCP contract. Send candidate labels with the existing source-bound review flag. | Agent pack/schema tells the agent how to read the catalog and submit bounded, definition-bearing proposals. Agent prose cannot mutate the catalog directly. |
+| 6. Extend the Jev call | In the same Decisions API request, ask a fit `noul` for each candidate and a `choice` among that facet's retrieved canonical labels plus a `new label` option. Include short definitions so terms are compared by meaning, not string resemblance. Parse and persist per-option probabilities, selected option and choice confidence alongside existing relevance/lens/facet-fit scores. Jev selects the proposed text or an existing option; it does not generate label text. | One request returns typed, bounded, validated judgments. Invalid/missing answers leave the review flag stored without Jev data. Persisted judgments identify the candidate/options and model so later policy/threshold changes can be audited. |
+| 7. Apply conservative decisions and reviewer feedback | Current policy auto-adds only when Jev selects the new-label option, the proposal ranks in the configured top N for its facet, fit and new-label probabilities each meet their threshold, and choice confidence clears its threshold. Defaults: tags top 5; audience/use_for/domain top 3 each; fit >= 0.90, new >= 0.90 and confidence >= 0.80; local embeddings and Jev are both required. Jev selects from agent-proposed labels and retrieved project labels; `new_label` means the proposal text is accepted, not that Jev creates new wording. Persist the decision on the ingest-review row and the new label, Jev scores and review provenance in the canonical project catalog. Existing-label matches are recorded as reuse decisions. Human feedback can guide future ingest, not model weights. | Decisions and label additions are attributable; the catalog appears in later project context. Tune thresholds against labeled false-new and false-merge outcomes before lowering them. Ambiguous/unsupported cases remain in the review queue. |
+| 8. Make search resolve canonical labels | Store canonical label IDs on accepted nodes (or a migration-safe representation that resolves to IDs). Exact searches for canonical names and aliases resolve through the catalog to the same ID set. Keep label vectors and search aliases synchronized with canonical revision and active embedding profile. | Searching/filtering by a canonical label or any approved alias returns the same authorized records; changing a label definition or model profile triggers a safe index rebuild without changing citations or canonical knowledge. |
+| 9. Pilot and tune automation | Run the configured policy over the Dome documents. Review match/new-label decisions, duplicate proposals, unique-label count, one-record labels, and reviewer corrections by facet. Tune caps, retrieval K and thresholds on these judgments; threshold env vars can tighten or relax promotion. | Pilot report includes per-facet false-new and false-merge rates, shortlist recall, one-off label rate, review burden, latency and offline behavior. Current defaults are conservative initial settings, not measured acceptance claims. |
+
+**Rollout boundary:** existing node facets continue to read during migration. Catalog resolution is introduced compatibly; bulk canonical rewrites and merges occur only after a reviewed migration map exists. Canonical project data remains authoritative, while SQLite catalog/vector tables are derived and safely rebuildable.
 
 ### M3 — Read-only application and pilot evaluation