GEWU/docs/architecture/ARCHITECTURE.md

GEWU architecture

Status: proposed baseline v0.3, 13 October 2026. The independent review covers the v0.2 acceptance foundation; v0.3 adds the owner-required local agent and module replacement protocol. Product authority: GOAL and PRD. Related: acceptance contract.

Decision

Build a modular Python service and a React reading application in the GEWU code repository. Deploy one authoritative knowledge repository per installation, shared across projects. The knowledge repository and private runtime data live outside the code checkout and outside agent-writable mounts.

Local agents reason about sources and submit proposals. The service validates their structure, source locations, permissions and policy eligibility, then records accepted changes. People can search and follow evidence without an agent running. The service contains no reasoning agent, autonomous research loop or general-purpose chat endpoint.

Provide a custom GEWU local agent in the same workspace-profile form as Winnow/Praxis in the RESEARCH folder: a role, shared protocols, modular skills and coordinating pipelines running inside an existing host. The source pack is maintained separately from development instructions. Host/model and transport adaptations can change without duplicating the skill logic; no new standalone reasoning runtime is required.

The starting deployment is one host, one service instance and one canonical writer. This fits the five-person pilot and makes the acceptance boundary inspectable. Capacity remains a measurement gate; five people does not specify simultaneous agent load.

Technology choices and reasons

These are proposed baselines, not installed versions. Use Python 3.12 as the initial compatibility target, Pydantic 2, React 19 and TypeScript strict mode. Resolve supported releases of the other packages at scaffolding, pin exact versions in uv.lock and package-lock.json, and record Python, Node and SQLite runtime versions. Do not deploy floating latest dependencies. Any version conflict reopens the relevant choice before implementation.

Boundary	Choice	Why for GEWU	Alternative and reconsideration trigger
Service	Python, FastAPI, Pydantic, Uvicorn; uv for environment/locking	Typed validation and the document-processing ecosystem; consistent with useful AAA/Jev patterns.	A TypeScript-only service saves a language but complicates Python parser integration. Revisit if parser requirements disappear.
Domain core	Plain typed Python modules, explicit dependency injection and public ports	The same scope, evidence and proposal rules serve MCP, reads and maintenance; implementations can be replaced behind tested contracts.	No generic plugin framework or process per module. Introduce distribution mechanisms only for a demonstrated extension need.
Custom local agent	Canonical workspace profile, SKILL.md modules and versioned pipeline procedures	Matches the owner's RESEARCH workflow; reuses existing hosts/models while keeping GEWU-specific reasoning procedures portable.	A standalone agent engine adds a runtime to maintain and is outside the selected form.
MCP	Official Python MCP SDK; local stdio connector, service Streamable HTTP	One contract across agent harnesses; connector can read selected local files and upload bytes.	Direct remote connections remain a tested option for clients that support the chosen auth profile; a remote path is never a file upload.
Canonical knowledge	UTF-8 Markdown with YAML frontmatter, YAML records, native Git CLI	Reviewable records, exact history, portable exports.	A database as canonical storage conflicts with the present PRD.
Private artifacts	Content-addressed filesystem by SHA-256	Retain originals without growing Git with PDFs/media. Git records hashes and manifests.	Object storage only after one-host capacity/backup measurements justify it.
Search/graph	SQLite FTS5, ordinary edges tables, sqlite-vec behind a vector adapter	Rebuildable on one host; no graph/vector service to operate.	sqlite-vec is a gated candidate because its upstream describes pre-v1 compatibility risk. If portability/performance fails, revise the adapter choice; full v1 still needs hybrid search.
Embeddings	Local Sentence Transformers adapter; multilingual-e5-small is the first evaluation candidate	Offline execution and a multilingual path without hosting a reasoning agent.	Final model/artifact hash depends on the project's labeled language corpus, license and CPU/RAM measurements. API providers are optional and must be explicitly configured.
SQL access	Standard sqlite3, parameterized SQL, explicit repository methods	FTS/vector queries are easier to audit directly; the index is disposable.	SQLAlchemy/PostgreSQL become candidates only if operational state or deployment topology warrants the cost.
Web	React, TypeScript, Vite, React Router	Search, evidence drawers and linked reading views; static production assets require no Node server.	Next.js SSR adds a second server without a present SEO or server-rendering requirement.
Browser state	TanStack Query for server data; URL for scope/filters/selection; React local state for transient controls	Cache ownership and cancellation are explicit.	Do not copy AAA's bespoke telemetry stores into a reading application. Introduce another store only for demonstrated shared client state.
Visual implementation	CSS custom properties + CSS Modules; Radix primitives where needed; Lucide icons	Small, application-owned visual vocabulary with accessible interaction foundations.	Tailwind/shadcn is viable, but adds conventions without a present component system to reuse. No mixed styling systems by default.
Source/rendering adapters	pypdf for initial PDF extraction, markdown-it-py for Markdown structure, trafilatura for HTML text, react-markdown + GFM + sanitized math rendering in the UI	Keep parser-specific behavior behind versioned extraction contracts.	OCR and more complex parsers require a quality/cost probe. PDF extraction success is not locator or reading-order correctness.
Quality tools	pytest, Ruff, mypy; ESLint, TypeScript, Vitest/Testing Library, Playwright and axe	Cover contracts, recovery and actual browser reading workflows.	No arbitrary coverage percentage in place of behavioral acceptance.
Hosting	Linux VPS, Docker Compose, Caddy HTTPS; native local development	Repeatable single-host service and private volumes; Python serves built UI assets.	Kubernetes, Redis and distributed queues are deferred until measured need.

The exact dependency inventory is a scaffolding deliverable. Model selection and native vector packaging are explicit prototype gates, not reasons to pretend the stack is already verified.

Components and dependencies

flowchart LR
    H[Human reader] --> W[React website]
    W --> R[Read HTTP adapter]
    PAK[GEWU profile, skills and pipelines] --> A[Existing local agent host]
    A --> C[Local stdio MCP connector]
    A --> R
    C --> M[Service MCP adapter]
    O[Human owner client] --> M
    R --> Q[Scoped query services]
    M --> Q
    M --> P[Proposal and acceptance services]
    P --> V[Schema, evidence and policy validators]
    P --> G[Exclusive Git writer]
    G --> K[Canonical knowledge repository]
    P --> J[Durable staging and run records]
    P --> B[Private source artifacts]
    K --> I[Index worker]
    I --> S[Derived SQLite index]
    Q --> S
    Q --> K

Imports point inward: transport -> application -> domain. Adapters implement storage, Git, clock, parser and embedding interfaces used by application services. Domain code imports neither FastAPI nor MCP nor React. A small lifespan-owned composition root creates services and closes resources. Each application service owns a complete use case; routes translate protocol messages and enforce transport admission.

Every substantial module exposes a versioned public contract and owns its implementation/state. Consumers cannot import its private internals or provider-specific SDK types. The modularity protocol defines module descriptors, dependency checks, compatible replacement tests and migration/rollback. This applies equally to service components, UI features, skills and pipeline coordinators. Compatible replacement changes composition/configuration rather than unrelated business logic; a breaking replacement declares a consumer migration.

Recommended code layout:

src/skb/
  domain/          # node, evidence, scope, policy, proposal contracts
  application/     # intake, evaluate, accept, query, runs, maintenance
  adapters/        # git, artifacts, records, sqlite, parsers, embeddings, auth
  transports/      # MCP, read HTTP, operator/maintenance CLI
  bootstrap/       # configuration and lifespan assembly
connector/         # local stdio client bridge; uses shared contract package
agents/gewu/       # actual source pack; installed into a research workspace
  AGENTS.md        # research role, separate from repository development rules
  manifest.json    # pack/skill/pipeline versions and verified compatibility
  protocols/       # evidence, scope and contribution concerns
  skills/          # kb-ingest, kb-query, kb-curate, kb-synthesize
  pipelines/       # host-executed coordination and checkpoint procedures
web/src/
  app/             # routes, providers, shell
  features/        # search, reading, sources, problems, history
  components/      # shared accessible primitives
  api/             # generated types, typed client, query keys
  styles/          # semantic tokens and reading typography
schema/            # generated/versioned public schemas and fixtures
tests/             # unit, contract, integration, recovery, fixtures
docs/              # product, architecture, protocols, decisions, acceptance

Do not create empty layers or adapter classes for every file. A module must own a real responsibility. Share public schema definitions between service and connector; generate browser types from the read API. Type generation complements runtime validation.

Persistence classes

Store	Contents	Rebuildable?
Private knowledge Git repository	Projects, nodes, source manifests, accepted evidence/assessments, policies, redirects, acceptance manifests	Authoritative; restore from backup.
Private artifact tree	Original bytes and immutable extraction/location maps, keyed by hashes	Required source evidence; restore from backup. A parser rerun does not replace a cited extraction version.
Durable staging/run tree	Proposal revisions, validation/decision events, uploads, external-run progress and receipts	Authoritative workflow history, written atomically; restore from backup.
Private control SQLite	Human accounts/password hashes, sessions, agent credential hashes, role/ownership grants, owner challenges and security audit	Operational authority; not the disposable search index. Backup with restricted access.
Derived SQLite index	Searchable nodes, FTS, edges, vectors, index metadata	Delete and rebuild without losing acceptance, evidence or identity.

Distinguish the code repository from the knowledge repository. The PRD's single-repository rule means all knowledge projects share one canonical repository; it does not require committing private team sources into the application's code repository.

Git paths use safe storage keys rather than raw logical IDs. IDs such as shared:paper-name remain metadata, avoiding Windows filename restrictions. Use bounded safe YAML loading, reject duplicate keys and unknown fields, and emit stable ordering/UTF-8/LF. References use logical IDs and revisions, never caller-selected filesystem paths.

Reads and retrieval

Every content query carries scope. Resolve registered projects and their explicit includes before search or ID resolution; return resolved_scope. All-team visibility does not authorize silently widening relevance scope. Redirects and graph traversal recheck destination scope.

Search runs scope-filtered FTS and vector candidates, merges them with reciprocal rank fusion, and performs bounded typed-edge expansion. Start with a maximum one-hop expansion and explicit candidate/edge caps in the public contract. Return ranking explanations separately from evidence status. Related suggestions never become accepted relationships by indexing.

The local service stays in keyword mode unless GEWU_EMBEDDING_MODEL_PATH and GEWU_EMBEDDING_MODEL_SHA256 are both set. The configured path must point to an already-installed model tree; inference is local-only with remote code disabled. GEWU_EMBEDDING_MODEL_ID defaults to the pinned multilingual E5 candidate and GEWU_EMBEDDING_DIMENSIONS defaults to 384. Install the retrieval extra to enable the optional runtime. Missing or failed vector generations produce a labeled keyword response. See the probe and setup evidence. This opt-in runtime is not a full-v1 acceptance claim.

Use normalized English token search plus language fixtures. unicode61 alone is not a claim of Chinese word segmentation; test CJK queries and use a documented segmentation or trigram adapter when needed. Evaluate short queries separately. Search acceptance is the PRD's labeled per-project question set, not a synthetic speed result.

Index updates publish all node/edge/FTS/vector changes for a chosen Git revision in one SQLite transaction. Queries use a read transaction pinned to an index revision. A model change builds a separate index generation, with model ID, artifact hash, dimensions, normalization, chunking and parser versions. Publish the new generation only after integrity and retrieval gates pass. Avoid replacing an open database file on Windows: switch the service's generation pointer and retire old connections before cleanup. The acceptance contract specifies the immutable extraction artifact and rebuild watermark procedure.

get_node may read a requested accepted revision directly from Git before indexing. Search reports its indexed revision and lag. Embedding outages permit clearly labeled keyword-only reads; they do not relax acceptance rules or count as full-v1 hybrid acceptance.

Budget whole serialized result groups, including citations and warnings. Proposed counting profile: a pinned tiktoken package with the o200k_base encoding and hash-verified encoding assets, identified in each response and preinstalled for offline use. This is an API accounting convention, not a promise to equal every harness model's tokenizer. Define the error-envelope minimum separately. Never cut a quote or discard a stale warning to meet a budget.

Process and concurrency model

Run one Uvicorn application worker initially. A process-lifetime OS lock excludes a second canonical writer on the same local filesystem; a repository head compare-and-swap check rejects stale publication. Do not deploy two writers with independent in-memory locks or accept shared NFS storage as equivalent.

Acceptance requests enter a bounded queue whose pending work is recoverable from staging records. Critical acceptance is not a fire-and-forget FastAPI background task. Blocking disk/Git calls cross one bounded worker boundary; extraction and model work execute in capped child processes with timeouts. Workers return artifacts with input hashes; only the service can publish canonical state. The contract defines ordering and recovery.

A revoked credential, changed policy or changed dependency invalidates a prepared result before publication. Cancellation of a caller does not undo an already accepted proposal; status lookup resolves uncertainty.

Authentication and deployment

Human browser sessions are expiring HttpOnly, Secure, SameSite cookies. The read API accepts those sessions with read capabilities only, including for an admin. It also accepts agent bearer keys for scoped, budgeted retrieval under the same read contracts used by MCP. Authentication endpoints may establish/end sessions but cannot author knowledge. Agent API keys are unique, revocable, server-bound to an agent and responsible human, stored as hashes at the service and outside Git at the connector.

For the initial controlled-team connector, document a GEWU provisioned-key authentication profile over TLS. Do not describe this as generic OAuth interoperability. HTTP MCP authorization has its own specification; supporting arbitrary remote OAuth clients requires a separately implemented and tested authorization-server/resource-server profile. The local stdio connector avoids assuming all four harnesses support the same remote authentication flow.

Protected changes use the human-authenticated owner MCP client and a digest-bound decision described in the contract. The proposed owner profile uses a short-lived opaque session obtained by password and TOTP reauthentication in a separate human terminal; it is a GEWU-controlled client profile, not a claim of generic OAuth support. Production separates the human owner credential from the agent's OS account/credential store. Owner-level unrestricted filesystem agents cannot be claimed to be isolated by prompt instructions.

Local operation uses the same core and directory layout as the VPS. Development may run on the user's account with disposable data; enforced pilot/team operation uses a separate service identity. VPS publication requires an independent restore test, HTTPS, private volumes and client workflow evidence. Moving authority to the VPS freezes local writes first; there is no independently writable local replica synchronization in v1.

Delivery and open decisions

Gate	Work	Observable exit
M0 contract slice	Schemas, state machine, safe file records, scope/policy fixtures; pin dependencies	Seeded invalid evidence/authority rejected; public schemas generated; decisions below settled for affected boundary.
Pilot end-to-end	Proposed starting point: Windows/Codex, text/Markdown, two projects plus shared, lexical search first	Upload -> proposal -> automatic acceptance -> committed read -> citation; crash/replay tests; read-only browser. Client/format choice remains recommended.
Full P0 service	Hybrid index, all required input types, four harnesses, portability and protected decisions	Per-client transfer/rejection/acceptance/retry/retrieval receipts; no scope or authority bypass.
Custom agent and module gates	Install GEWU profile/skills into an isolated research workspace; enforce dependency boundaries and pinned module/skill versions	Native host ingestion/query and interrupted-run recovery; compatible module swap passes contract/consumer tests. Instruction-pack presence alone cannot close this gate.
Team release	VPS migration, backup/restore, quality sampling, capacity/security acceptance	All selected PRD P0 gates plus measured retrieval, evidence and human reading tasks.
P1 follow-up	Rich comparisons, graph canvas, promotion/import/export and synthesis/curation	Explicitly selected requirements and their own acceptance evidence.

Still needs owner input: first admin and project/shared owners; production auto-accept policy; sampling owner/cadence/error limits; representative sources and expected answers; supported initial client/format; VPS capacity, concurrent load, archive retention, recovery time and recovery-point targets. Production credentials and policies cannot use silently invented answers.

Technical probes needed: native vector package on Windows/macOS/Linux; parser locator fidelity including scanned/math PDFs; final embedding model and language set; cold rebuild under the PRD's ten-minute target. Measure cached-vector index rebuild and cold re-extraction/re-embedding separately; do not rename a warm rebuild to claim the full target passed.

Research basis

Checked 12 October 2026. These sources establish available mechanisms, not GEWU performance or compatibility:

• SQLite WAL: WAL allows reader/writer concurrency but retains one writer; motivates short index transactions and one-host operation.
• SQLite FTS5: tokenizers and substring search options; language quality needs our fixtures.
• sqlite-vec upstream: embedded vector implementation and explicit pre-v1 warning; requires an isolated adapter and pinned packaging.
• Official Python MCP SDK, versioned MCP transports, and HTTP authorization: use negotiated compatible versions; do not implement draft changes by accident.
• FastAPI process workers: multiple workers are separate processes; an in-process lock cannot serialize their writes.
• Pydantic strict validation, Vite, TanStack Query: typed boundaries, static assets and server-state tooling supporting the choices above.
• pypdf extraction limitations: extraction and OCR are different capabilities; quality must be measured.
• Sentence Transformers model loading supports pinned revisions and local-only loading. A disposable Linux/Python 3.14 CPU probe loaded the pinned multilingual-e5-small model card snapshot and sqlite-vec 0.1.9; see the probe record. The adapter is implemented, but vector storage/fusion, offline distribution, native cross-platform compatibility and labeled recall remain unqualified. sqlite-vec remains pre-v1 and is not yet approved as the production index backend.
• tiktoken provides named encodings for a reproducible budget convention. Git update-ref supports an expected-old object check for publication.