From b7c11bbe2db459e3b6b0f492e6973d5c9f9c3523 Mon Sep 17 00:00:00 2001 From: Ajasra Date: Mon, 12 Oct 2026 15:45:08 +0800 Subject: [PATCH] Initial commit --- docs/GOAL.md | 23 +++ docs/PRD.md | 461 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 484 insertions(+) create mode 100644 docs/GOAL.md create mode 100644 docs/PRD.md diff --git a/docs/GOAL.md b/docs/GOAL.md new file mode 100644 index 0000000..f728116 --- /dev/null +++ b/docs/GOAL.md @@ -0,0 +1,23 @@ +# Project North Star: KB + +KB is a local-first, multi-project knowledge base for people and AI agents. It turns collected sources into scoped, evidence-backed notes and linked syntheses that remain traceable, reviewable, and reusable across projects and tools. + +## Core Mission + +Help project teams find and reuse trustworthy knowledge without losing its source, scope, freshness, or uncertainty. People can explore and curate the knowledge through a web UI; agents can retrieve cited, budgeted results through typed MCP and HTTP interfaces, and contribute only through validated proposals. + +## Invariant Non-Negotiables + +1. **Evidence before assertion**: Every claim and datapoint must trace to a source and a quote location. Validation checks the evidence against the archived source when available; unsupported interpretations remain clearly identified as interpretations. +2. **Project scope is explicit**: Every node belongs to one registered project or the shared library. Cross-project relevance is represented through explicit typed links and project-specific lens fit, never by silently copying or blending project knowledge. +3. **Humans own project intent**: Project owners control the north star, key questions, and autonomy policy. Agents can propose changes, but cannot directly change those human-owned goals. +4. **Proposals are the write boundary**: Agents submit changes to staging. Shared validators enforce schemas, scope, link rules, and evidence requirements before a proposal can be accepted into the knowledge base. +5. **Git files are the source of truth**: Markdown and YAML records remain directly inspectable and versioned. Search indexes are derived, disposable, and rebuildable from those files. +6. **One core, multiple interfaces**: The UI, CLI, MCP server, and HTTP API use the same scoped service and validation rules. No interface grants agents a privileged write path or exposes raw database queries. +7. **Trust survives change**: Preserve history and audit records; link contradictions instead of overwriting them; retain superseded knowledge; and disclose staleness whenever dated information is retrieved. +8. **Local-first and agent-agnostic**: The system remains useful with local files and local models, runs across Windows, macOS, and Linux, and does not depend on any one agent harness or vendor integration. + +## Definition of Success + +A person or agent can answer a project question with a useful, cited result in the intended time and token budget; merged claims have valid evidence; stale datapoints carry warnings; and knowledge can be reused across projects without losing its origin or confusing its scope. The measurable v1 targets are maintained in [the PRD](PRD.md#success-metrics). + diff --git a/docs/PRD.md b/docs/PRD.md new file mode 100644 index 0000000..285eaa0 --- /dev/null +++ b/docs/PRD.md @@ -0,0 +1,461 @@ +# Multi-Project Knowledge Base for Humans and AI Agents + +Working name: KB | Version 0.2 (draft) | 12 Oct 2026 | Status: for review +## Summary + +KB is a local-first, git-based knowledge base that turns collected material (web pages, papers, reviews, NotebookLM exports, talks, internal notes) into verified, linked, reusable knowledge. It serves two kinds of readers equally: people (through a web UI) and AI agents (through MCP and an HTTP API). One installation hosts many projects, each identified by a project_id (for example interactive-video, dome-performance, agentic-design). Projects keep their own goals, taxonomies and curated notes, and can draw on a shared library of sources, entities and concepts. +The core idea is three layers: immutable sources, atomic notes extracted from them with evidence, and compiled syntheses (briefs, market overviews, timelines, comparison tables) built from the notes. Agents write only by submitting proposals that pass a validator. +## Problem and opportunity + +- Research for a project is scattered across browser tabs, PDFs, NotebookLM notebooks and chat histories, and is hard to reuse by a different person, agent or project. +- AI agents working on development, writing, presentations, budgets or marketing plans start each session without the project's accumulated knowledge, or get it as unverified summaries. +- Existing note tools store text but do not enforce provenance, evidence strength, freshness or typed relationships, so market numbers go stale, contradictions go unnoticed and claims lose their sources. +- Knowledge that applies to several projects (for example agentic design patterns used in both an interactive video tool and a dome show pipeline) is duplicated or lost. +## Goals and non-goals + +Goals +1. One knowledge base, many projects. Every node belongs to exactly one project or to the shared library, and queries can be scoped accordingly. +2. Evidence-backed knowledge. Every claim traces to a source and a quote location that a validator can check. +3. Dual-audience access. Humans browse, learn and search in a UI. Agents retrieve cited, budgeted, structured answers through MCP and API. +4. Agent-agnostic. Works from OpenCode, Claude, Codex and Antigravity through shared files, skills and MCP, with no logic locked inside one tool. +5. Goal-aware ingestion. A local agent pipeline digests new material against the project's goal (three horizons) and proposes notes, links and insights. +6. Multi-audience value. The same nodes serve developers, product managers, project managers and sales through facets and saved lenses. +7. Trustworthy over time. Dated claims, staleness tracking, contradiction links and a full audit trail. +Non-goals (v1) +- Real-time multi-user collaborative editing (git-based collaboration only). +- Hosting copyrighted source texts for redistribution. KB stores notes, short quotes and links, and keeps archived copies private. +- Replacing project management, CRM or code search tools. KB feeds them knowledge but does not manage tasks or deals. +- A general-purpose chatbot. Agents use KB as a tool, and the UI is for navigation, review and reading. +- A story-map authoring editor (a canvas for drawing branching story flows) and an embedded in-UI copilot. Agents reach KB through MCP, and story authoring is a separate product. +## Users and use cases + +User +Typical needs +Example +Developer +Problems and proven techniques, architecture precedents, case studies +How do interactive movies avoid combinatorial explosion of branches? +Product manager +Market shape, competitors, user problems, feature precedents, comparisons +Compare how three titles handled player choice pacing. +Project manager +History, timelines, cost and schedule datapoints, risks +Timeline of interactive film production tooling for a kickoff deck. +Sales +Market overview, positioning, proof points, objections, as-of numbers +Current market size claims and who made them. +Ingest agent +Digest a source against the project goal and propose notes +Process a new paper into proposals with evidence and edges. +Consumer agent +Retrieve cited knowledge during coding, writing, budgeting, marketing +Fetch a brief for a presentation with token budget 2,000. +Curator agent +Find stale, duplicate, contradictory or orphan nodes +Weekly report of stale market datapoints. +Project owner +Define the goal, review proposals, set autonomy levels +Approve new notes, update the north star. +## Multi-project model + +This section defines how one database serves many projects. +### 5.1 Project registry + +Each project has a registry entry projects//project.md with frontmatter: +- project_id: lowercase slug, unique, immutable (for example interactive-video, dome-performance, agentic-design). +- title, description, status (active, paused, archived), owners. +- north_star: the project's main goal (Horizon 3), human-owned. Agents may propose edits through review, never change it directly. +- key_questions, watch_list (competitors, technologies, titles, people), out_of_scope. +- audiences: which audiences the project cares about. +- taxonomy_extensions: project-specific domain tags and node subtypes added to the core vocabulary. +- autonomy_policy: per node type, one of suggest, auto-merge-safe, research-on-gap. +- default_includes: other projects or shared whose nodes appear in this project's searches by default. +Horizons 1 (current request) and 2 (session objective) are not stored in the registry. The agent derives them per task and records them in the run log. +### 5.2 Scope of every node + +Every node has project_id, which is either a registered project or the reserved value shared. +Scope +What lives there +Rule +Project +Notes, syntheses, project-specific cases and datapoints +Owned by one project. Reviewed against that project's lens. +Shared +Sources, entities (companies, people, tools, titles), general concepts, cross-domain techniques +Deduplicated globally. Any project can reference them. +Rules: +1. Sources and entities default to shared, so one paper or company is stored once and referenced by many projects. +2. Notes default to the project that triggered ingestion. A note can be promoted to shared when it is useful beyond the project, through a review action that records who promoted it and why. +3. Cross-project links are allowed and explicit. A project node can link to a shared node or to a node in another project with a typed edge. The edge records the project pair, and the UI and API mark it as cross-project. +4. Relevance is per project. A node has lens_fit keyed by project: a note about agentic design patterns can carry a high score for agentic-design and a medium score for interactive-video, without being copied. +5. Default query scope is the active project plus its default_includes and shared. Other scopes are opt-in (projects=[...] or all). +### 5.3 Identity and layout + +- Node IDs are globally unique and stable: : for project nodes and shared: for shared nodes. Renames create redirects. +- Repository layout (single repo, namespaced): +kb/ + shared/{sources,entities,notes,synthesis}/ + projects//{project.md,notes,synthesis,staging,runs,lens/} + schema/ protocols/ skills/ src/ docs/ +- A separate repo per project is not used in v1, because shared sources and cross-project links would need duplication or submodules. The repo can still be split later because every node carries its project_id. +### 5.4 Isolation and export + +- v1 is a single-owner or small-team setup. Isolation is logical: every API call takes an explicit scope, and the validator rejects nodes with an unknown project_id. +- Per-project export (nodes, linked shared nodes, and an index) must be possible so a project can be handed over or archived. +- Per-project access control (read, propose, merge) is a P2 requirement and the schema must not block it. +## Content model + +### 6.1 Layers + +1. Source (immutable): archived snapshot or reference, hash, bibliographic and reliability metadata. Never edited by agents. +2. Note (atomic): one finding, claim, problem, technique, case, datapoint or event, with evidence quote and location. +3. Synthesis (compiled): topic pages, briefs, market overviews, timelines and comparisons, regenerated from notes. +### 6.2 Node types + +Source, Finding/Claim, Problem, Technique/Solution, Case, Entity, Datapoint, Event, Concept, Synthesis. Projects may add subtypes through taxonomy_extensions. +Project subtype packs. A project adds subtypes and fields through its registry, and the same validator checks them. The interactive-video pack starts with these four, taken from the deep-research report: +Subtype +Based on +Key fields +Production case +Case +release year, studio, delivery medium (FMV, real-time 3D, streaming player, theatrical), budget, reported revenue, units sold, total and average playthrough runtime, shooting ratio, decision points, endings +Narrative mechanism +Technique +paradigm (branch-and-bottleneck, state variable, storylet, database cinema, foldback), complexity class, authoring overhead, agency profile +Technical constraint +Problem +domain (buffering, state storage, bandwidth, platform parity), severity 1-5, mitigation +Market metric +Datapoint +metric type (market size, conversion, ticket premium, completion drop-off), demographic, value, unit, sample size, reporting year +Other projects define their own packs. All subtype fields follow the evidence rules in 6.3. +### 6.3 Metadata + +All nodes: id, project_id, type, title, summary (1-2 sentences for agents to scan), created, updated, status (draft, reviewed, stale, superseded), confidence (low, medium, high), tags, schema_version. +Facets: audience (dev, product, pm, sales), domain (project-extensible core set: tech, design, market, history, legal, production, marketing), use_for (presentation, budget, comparison, architecture decision, pitch). +Lens fit: lens_fit: {: {score, horizon, key_question, reason}}. +Provenance: derived_from: [{source, quote, loc}], run_id of the agent run that proposed it, agent and model identity. +Source-specific: URL, author, publisher, publication date, access date, source type (paper, review, press, vendor, internal), reliability grade, archive path, license note. +Claims and datapoints: evidence_quote (short, with page or timestamp), evidence_type (measured, reported, opinion, vendor marketing), as_of, stale_after. +Datapoint values: value, unit, sample_size, reporting_year and demographic (where relevant) are separate fields from the quote, so numbers can be compared and charted. +Entities: aliases (alternate names) and merged_from (IDs merged into this node, so a merge can be undone). +### 6.4 Typed edges + +solves, causes, supports, contradicts, instance_of, part_of, precedes, supersedes, compared_with, derived_from, plus applies_to (node to project) for cross-project relevance. Edges live in frontmatter and are validated: the target must exist and the edge type must be allowed between the two node types. +Projects add extension edges in their registry, each with allowed source and target types. The interactive-video pack adds implements (case to mechanism), introduces (mechanism to constraint, a project-level form of causes), mitigates (technique to constraint), has_metric (case to datapoint) and incurs_cost (problem to datapoint). documented_in is the core derived_from edge. The core set stays small so other projects are not forced to carry these. +## Functional requirements + +Priority: P0 required for v1, P1 important, P2 later. +### 7.1 Project management + +ID +Requirement +Pri +FR-P1 +Create, archive and list projects from a registry entry, and the validator rejects nodes with unregistered project_id. +P0 +FR-P2 +Every read and write API takes a project scope, with the documented default scope. +P0 +FR-P3 +Shared library with global dedupe of sources and entities. +P0 +FR-P4 +Promote a note to shared, and mark links as cross-project in API and UI. +P1 +FR-P5 +Per-project lens (north-star, key questions, watch list) read by ingest and curation. +P0 +FR-P6 +Per-project export and import. +P1 +FR-P7 +Per-project access control. +P2 +### 7.2 Ingestion + +ID +Requirement +Pri +FR-I1 +Capture from URL, PDF, file, text paste and NotebookLM export through an inbox/ folder, CLI and API. +P0 +FR-I2 +Archive source snapshot, hash, normalize metadata, dedupe by hash and URL across the whole base. +P0 +FR-I3 +Triage against the project lens: relevant or skip, with score and reason. +P0 +FR-I4 +Digest extracts schema-valid notes with quote and location for every claim. +P0 +FR-I5 +Analyze writes learnings and points of interest per horizon, labeled as the agent's interpretation and kept separate from source claims. +P1 +FR-I6 +Link resolves entities to a canonical name plus aliases (name normalization and embedding similarity find candidates, the agent judges borderline matches), logs every merge so it can be undone, proposes typed edges and flags contradictions. +P0 +FR-I7 +Verify re-checks quotes, numbers and dates against the archived source and assigns evidence grade. +P0 +FR-I8 +All writes go to staging/ as proposals, with a run log. +P0 +FR-I9 +Tag each source with the projects it was ingested for, so one source can serve several projects. +P0 +FR-I10 +Task brief: agent derives H1 and H2 for each run and stores them in the run record. +P1 +### 7.3 Validation and review + +ID +Requirement +Pri +FR-V1 +kb validate checks schema, required fields, link resolution, edge-type rules, project scope and that every quote appears in the archived source. A proposal that fails cannot be committed. +P0 +FR-V2 +Review inbox with accept, edit, reject and merge-duplicate actions. Decisions are logged and reused as examples for the skills. +P0 +FR-V3 +Autonomy policy per node type and per project, with graduated promotion from suggest to auto-merge once acceptance rate is demonstrated. +P1 +FR-V4 +Contradictions are linked, never overwritten. Superseded nodes stay with superseded_by. +P0 +### 7.4 Retrieval and agent interface + +ID +Requirement +Pri +FR-A1 +MCP server and HTTP API expose: search, get_node, neighbors, timeline, compare, brief, list_projects, propose_note, flag_stale. All take project scope. +P0 +FR-A2 +Hybrid retrieval: keyword plus embeddings, then graph expansion by typed edges. +P0 +FR-A3 +Responses are structured JSON with citations, confidence, as_of and project of origin, bounded by a token budget parameter. +P0 +FR-A4 +brief(topic, audience, purpose, budget) returns a compiled, cited summary for one audience. +P1 +FR-A5 +Agents cannot write to notes/ or synthesis/ directly. They can only submit proposals. +P0 +FR-A6 +Single canonical AGENTS.md, with thin CLAUDE.md and GEMINI.md pointers, a shared skills folder, and an install script that links skills and MCP config into OpenCode, Claude, Codex and Antigravity. +P0 +FR-A7 +Optional hook integration so a coding harness can inject relevant KB nodes at turn start, with high-confidence full items and low-confidence hints. +P2 +FR-A8 +MCP resources (schema, project registry, node by ID) and prompts (per-project roles, for example an architecture advisor) alongside the tools. +P1 +FR-A9 +Typed tools only. No raw query language (Cypher, SQL) is exposed to agents, and every write goes through a proposal. +P0 +### 7.5 Compilation and curation + +ID +Requirement +Pri +FR-C1 +Regenerate affected syntheses when notes change and show a diff for review. +P1 +FR-C2 +Curator agent reports stale nodes, duplicates, orphans, contradictions and cross-project promotion candidates. +P1 +FR-C3 +Staleness: nodes past stale_after are flagged in the UI and returned with a warning by the API. +P0 +FR-C4 +Optional topic-cluster summaries: detect clusters in the graph and write a summary per cluster for broad questions. Enabled only when a project outgrows its syntheses. +P2 +### 7.6 Web UI + +ID +Requirement +Pri +FR-U1 +Project switcher, with an "all projects" view and clear marking of shared and cross-project items. +P0 +FR-U2 +Global search with filters for type, audience, domain, date, confidence and project. +P0 +FR-U3 +Node pages with summary, evidence, source links, typed neighbors and history. +P0 +FR-U4 +Problem to solutions to cases matrix, market datapoint table with as-of badges, timeline view, comparison view. +P1 +FR-U5 +Graph explorer. +P1 +FR-U6 +Review inbox and staleness list. +P0 +FR-U7 +Audience lenses: saved filters plus brief templates for developers, product, project managers and sales. +P1 +FR-U8 +Export a synthesis or timeline to slides or markdown. +P2 +## Architecture summary + +1. Source of truth: markdown and YAML files in git. +2. Derived index: SQLite (full text and vectors) or Postgres plus pgvector, with an edges table. It is disposable and can always be rebuilt from the files. +3. KB service: one core with MCP, HTTP and CLI transports derived from a single registry of tools. It is the only code path that reads the index and commits to the repo. +4. Skills: judgment procedures (capture, triage, digest, analyze, link, verify, compile, report), kept as one canonical set with a shared concerns to protocol mapping. +5. UI: a client of the same API with no privileged access. +6. Principle: deterministic strictness lives in code (validator, indexer), not in prompts, so different agents and models are held to the same floor. +Reuse from existing work: the one-core-with-thin-adapters and registry pattern, confidence tiers, observe-then-enforce promotion and acceptance receipts from Jev, and the skill template, state-machine pipelines and symlink-based cross-agent setup from Winnow. +Storage decision (v1). Files in git plus SQLite (full text, vectors and an edges table). A graph database such as Neo4j was evaluated and is not used in v1: it adds a server to run and back up, and it moves the source of truth out of git, which loses diffs, undo and direct file access for agents. A graph engine can be added later as a second derived index. Revisit when the base passes the 100,000-node target, when multi-hop path queries become too slow in SQLite, or when live multi-user editing is required. Also left out of v1: cloud document parsers (local-first), separate entity-tagging models and heavy entity-matching pipelines. Agents get typed tools only, never a raw query language. +## Non-functional requirements + +Area +Requirement +Performance +Search under 300 ms for a project of 10,000 nodes on a laptop. brief under 5 s excluding model time. +Scale targets +10,000 nodes per project and 100,000 total, with full index rebuild under 10 minutes. +Local-first +Everything runs offline with local models as an option. Cloud models are configurable. +Portability +Plain files, no lock-in. Works on Windows, macOS and Linux. No required symlinks. +Reliability +Index is rebuildable. Every write is a git commit. Ingest runs are resumable from file artifacts. +Auditability +Every node records run, agent and model. Every proposal decision is logged. +Security +Source text is treated as data, never as instructions. Ingest agents have no write access beyond proposals. Secrets are kept out of the repo. +Legal +Store notes, short quotes and links. Archived copies stay private. License field on every source. +Versioning +schema_version on every node and run record, with migration scripts. +Agent-agnostic +No required use of tool-specific hooks, plugins or subagents in the core pipeline. +## Success metrics + +Metric +Target +Citation validity (quotes found at stated location) +100% of merged notes +Proposal acceptance rate after tuning +At least 70% for notes, tracked per skill and per agent +Retrieval quality on a per-project question set of 20-30 questions +Correct source in the top 5 for at least 80% of questions +Cross-agent convergence +Same 5 documents through two different agents give validator-clean, mostly overlapping notes +Freshness +No datapoint older than its stale_after served without a warning +Reuse +At least one project reuses shared nodes from another project in its first month +Time to answer +A person finds a cited answer to a typical question in under 2 minutes +## Milestones + +Milestone +Scope +Exit criteria +M0 Foundations +Node schema, edge vocabulary, project registry, validator, 20 hand-guided sources across two projects, starting with the deep-research report as the first ingest test (graded as a secondary, AI-synthesized source) +Validator catches seeded errors. Notes validate. Verify flags unsupported claims in the report. +M1 Service +Index, hybrid search, MCP and HTTP with project scoping, proposals to staging/ +Agents can search and propose from all four tools. +M2 Ingest +capture, triage, digest, verify, link, run logs, task brief +Cross-agent convergence test passes. +M3 UI +Project switcher, search, node pages, review inbox, staleness +A person reviews and navigates without touching files. +M4 Synthesis +analyze, compile, brief, matrix, timeline, curator report +Briefs produced for each audience. +M5 Multi-project hardening +Promotion to shared, cross-project edges, export and import, acceptance receipts per agent +Three projects live on one base without cross-contamination. +## Risks and mitigations + +Risk +Mitigation +Different agents extract inconsistent notes +Shared schema, validator gate, golden test documents, short example-driven skills +Hallucinated or misquoted claims +Quote-location validation, separate verify step, second agent for verification where possible +Low-value auto-captured notes +Lens-based triage, relevance thresholds, review-first autonomy +Projects pollute each other +Required project_id, scoped queries, per-project lens fit, explicit cross-project edges, promotion only through review +Duplicate sources and entities across projects +Global dedupe on hash, URL and entity resolution in the shared layer +Prompt injection from ingested sources +Treat source text as data, restrict ingest agent permissions, no direct writes +Market data goes stale +as_of and stale_after required for datapoints, curator reports, API warnings +Vendor or tool lock-in +Files plus MCP, no dependence on any one agent's hooks or plugins +Index drift +Index is derived and rebuilt from files, with a doctor command to check it +Over-building the stack (graph server, extra pipeline stages) +Start with files, SQLite and typed tools. Add a graph engine, clustering or heavier entity matching only when a revisit trigger in section 8 is hit. +## Open questions + +1. Should KB live in its own repo or inside one of the existing agent repos? +2. Embedding model: local or API, and how to handle re-embedding when the model changes? +3. UI stack and hosting: local web app only, or also hosted for a team? +4. Do projects ever need hard isolation (separate access rules), or is logical separation enough? +5. Should shared notes require two-step review (project owner plus a shared curator)? +6. How should NotebookLM exports be captured: manual export, or an automated step? +7. Which node subtypes does each of the three starting projects need beyond the core set? +## Appendix A. Example project registry entry + +--- +project_id: interactive-video +title: Interactive video knowledge +status: active +north_star:
+key_questions: + - How do others control branch explosion at feature-film scale? + - Who pays for interactive video, and how much? +audiences: [dev, product, pm, sales] +default_includes: [shared, agentic-design] +autonomy_policy: {source: auto-merge-safe, note: suggest, datapoint: suggest} +--- +Other starting projects: dome-performance, agentic-design. +## Appendix B. Example note + +--- +id: interactive-video:state-variables-vs-branching +project_id: interactive-video +type: technique +title: Track state variables instead of multiplying branches +summary: Store choices as flags or values and vary scenes by state so paths do not multiply with each choice. +audience: [dev, product] +domain: [tech, design] +confidence: medium +status: reviewed +solves: [interactive-video:combinatorial-explosion] +lens_fit: + interactive-video: {score: 0.9, horizon: H3, key_question: branch explosion} + agentic-design: {score: 0.4, horizon: H2, reason: state tracking in agent workflows} +derived_from: + - {source: shared:src-example-talk, quote: "...", loc: "t=14:32"} +schema_version: 1 +--- +## Appendix C. Agent tool signatures (illustrative) + +search(query, project="interactive-video", scope="project+shared", filters={type, audience, domain, since}, limit=10) +get_node(id) +neighbors(id, edge_types=[], depth=2, scope="project+shared") +timeline(topic, project, from, to) +compare(ids) +brief(topic, project, audience, purpose, token_budget) +list_projects() +propose_note(project, node_bundle, run_id) +flag_stale(id, reason) +Resources: kb://schema, kb://projects/{project_id}, kb://nodes/{id}. Prompts: per-project roles, for example architecture_advisor(project_type, budget_tier) for interactive-video. +