vasily/GEWU
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add documentation, specifications, and pipeline configurations for GEWU agent system

Browse Source
This commit is contained in:
Ajasra committed 2026-10-13 12:36:21 +08:00
1 parent b7c11bbe2d
commit 7fb257095f
31 files changed
+1770 -317
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture/ACCEPTANCE_CONTRACT.md
+157
View File
@@ -0,0 +1,157 @@
# Acceptance, authority and evidence contract
Status: proposed v0.2 after [independent review](../reviews/ARCHITECTURE_SENIOR_REVIEW.md). Drafted 12 October, revised 13 October 2026. Implements the boundaries in the [architecture](ARCHITECTURE.md); schemas and executable tests are still required at M0.
## Record separation
A source capture records bytes. An extraction records a parser's reading of those bytes. An assessment records an agent or human judgment about a particular assertion. A decision records the authority and checks that permitted a change. Keep separate identities and immutable revisions for each. Repeated claims do not manufacture independent evidence.
All public records carry `schema_version`; all stored timestamps are timezone-aware UTC. Domain IDs are globally unique, bounded and never reused. Typed models reject extra fields, non-finite numbers, unsafe paths and oversized collections. The following shapes name the required boundaries; they are not executable JSON Schemas yet.
| Record | Required bindings |
| --- | --- |
| Source revision | Stable source ID, artifact SHA-256, media type, byte length, capture time, original URL/version where known, license/retention metadata, archive availability. |
| Extraction | Source revision, parser/version/config digest, extraction hash, ordered blocks, page/timestamp metadata, normalization version, warnings/status. |
| Assertion evidence | Node/field path, assertion digest, source revision, extraction ID, locator, exact quote and normalization profile. |
| Support assessment | Immutable assessment ID, assertion digest, evidence digests, assessor principal/run, method/rubric, supported/unsupported/unavailable, rationale. |
| Proposal revision | Proposal ID, monotonic revision, initiating project, payload digest, base commit, touched node revisions, affected policy domains, operations, evidence/assessment references, authenticated proposer and run. |
| Validation result | Proposal revision/digest, validator version, evaluated policy versions, dependency revisions, check results and actionable findings. |
| Owner decision | Decision/challenge IDs, proposal/revision/digest, human principal and owner-session ID, affected domains, policy and grant versions, decision, authorization/expiry times, decision digest. No secrets. Immutable decision snapshot; lifecycle tracked separately. |
| Acceptance manifest | Proposal/revision/digest, before-revision, changed paths/content hashes, accepted node revisions, actor/agent/key IDs, decision snapshot/ID/digest when required, policy/validator versions, decision time. |
| Receipt | Immutable proposal/revision/digest, outcome, commit ID when accepted, node revisions, acceptance-manifest path/hash and findings. Current index revision/status appears in the response envelope, separately from this immutable receipt. |
Proposer identity is derived from authentication. A self-reported model or harness label is useful provenance but grants no capability. Assessment authorship is likewise authenticated; the agent cannot submit a record pretending to be a human reviewer.
## Public transport contract
One versioned operation catalog owns request/response schemas and capabilities. Each content request includes `api_version`, `request_id`, explicit `scope`, and operation-specific fields; reads add `budget_tokens`, bounded `limit` and optional opaque `cursor` where applicable. A cursor binds the normalized query, scope and index revision; a mismatched or retired snapshot returns `CURSOR_EXPIRED`, never silently mixes pages.
Successful reads return `api_version`, `request_id`, `resolved_scope`, `data`, complete `citations`, `warnings`, `omissions`, `canonical_revision`, `index_revision`, `retrieval_mode`, `budget_profile`, `tokens_used` and `next_cursor` where applicable. Fields irrelevant to a direct canonical read are explicitly null with a reason. Error envelopes use `code`, safe `message`, `findings` and `retryable` plus request identity. HTTP status and MCP error representation map to the same domain result. Query `data` contains only accepted records; draft status belongs to proposal operations.
| Surface | Authentication | Allowed capabilities |
| --- | --- | --- |
| Read HTTP API | Read-only browser session or agent bearer key | Scoped search, node/source/neighbor/history reads; agent responses have the same budgets and citations as MCP. No proposals or owner decisions. |
| Agent MCP | Provisioned agent key through the controlled connector profile | Permitted reads, upload/staging/run operations and typed knowledge proposals; no human owner decision. |
| Owner MCP | Dedicated short-lived human owner session | Read the challenge/diff and authorize/reject that bound protected proposal within current owner grants. No unconditional merge. |
| Authentication/control plane | Human login or restricted operator channel | Session establishment and credential/grant management only; no knowledge-authoring bypass. |
Named initial operations are `search`, `get_node`, `neighbors`, `list_projects`, `source_upload_begin/chunk/finalize`, `proposal_submit/get/revise/withdraw/evaluate`, `run_get/checkpoint` and `flag_stale` (a proposal). Exact schema files and HTTP path mappings must be generated and tested at M0 before adapters are implemented. The content/authority semantics above are fixed design recommendations, not left for each adapter to invent.
## Scope and policy domains
Reads resolve requested projects plus their declared includes and shared. An `all` request is explicit. Direct IDs, redirects, source citations, graph neighbors and suggestions honor the resolved scope. When a target is outside scope, return an omission reason and a way to request wider scope. Never silently follow a link into another project.
Mutation authority follows every changed record. A project owner does not gain control of shared records or another project's records through a read include. Shared has an explicit owner grant and policy.
Proposed bundle rule: all-or-nothing acceptance in one Git commit under every affected policy. A source in shared and a project note may travel together only when both domains authorize their changes. Otherwise retain the entire bundle and return domain-specific findings. The agent may submit a shared-source proposal first, then a dependent project-note proposal; each receives its own receipt. Never report an incomplete bundle as accepted.
## Proposal lifecycle
| Persisted state | Entered by | Allowed next step |
| --- | --- | --- |
| `staged` | New proposal or corrected immutable revision | Evaluate; withdraw. |
| `validating` | Service records the evaluation attempt and input bindings | Accept, return to staged with findings, hold for review, or reject. A crashed attempt is recovered and revalidated. |
| `needs-review` | Missing protected decision, policy exception or unresolved ambiguity | Append assessment/decision and revalidate; submit corrected revision; reject or withdraw. |
| `accepted` | Canonical commit published and receipt reconciled | Terminal for this proposal. Corrections use a new linked proposal. |
| `rejected` | Authorized explicit rejection with rationale | Terminal. A new linked proposal may address the rejection. |
| `withdrawn` | Authenticated proposer/authorized owner withdraws before acceptance | Terminal. An accepted proposal cannot be withdrawn. |
`validation-failed`, `conflict`, `busy`, `dependency-unavailable` and `index-pending` are outcomes, not competing lifecycle states. Invalid or conflicting proposals return to `staged` with an immutable failed-attempt record. `index-pending` accompanies accepted state and does not request a second acceptance.
Correction appends a proposal revision and invalidates previous validation. Changed assertions or evidence invalidate their support assessments. An unchanged payload cannot inherit a validation result under a changed policy, schema or dependency revision. Withdrawal and acceptance serialize through the same writer; whichever completes first determines the permitted next action.
## Exact replay and conflict behavior
The client supplies `proposal_id`, revision and payload digest. The service recomputes the digest over canonical serialization. Exact retries return the original durable receipt, with current index status in a separate envelope. Reusing the same ID/revision with different content returns `IDEMPOTENCY_CONFLICT`; correction requires a new revision. Retrying an already accepted proposal never creates a second commit.
Proposal edits use expected revision checks. In the initial implementation, a changed canonical base commit returns a conflict, even if only an unrelated record changed. This conservative rule avoids implicit rebases. The client can fetch the new base and submit a new proposal revision, reusing only assessments whose assertion/evidence bindings remain identical. Record conflict frequency; move to dependency-level optimistic concurrency later if measurements justify it.
Authenticated status reads are allowed after key rotation through the same stable agent identity and current read capability. Revocation blocks further use of the old key; it does not erase its historical attribution.
## Publication and crash recovery
The acceptance commit itself contains the acceptance manifest at the deterministic path `acceptance/<proposal-storage-key>/<revision>.yaml`. It cannot contain its own Git commit ID. The external receipt resolves that ID from the committed manifest. The manifest path is unique and immutable; its first introducing commit on the authoritative ref's ancestry supplies the accepted commit ID. An orphan commit or another branch cannot establish acceptance.
Publication progress is a separate durable journal field, not a second proposal lifecycle: `evaluated`, `publishing` (expected head, manifest path/hash, prepared commit ID), `published-unreconciled` (published commit), then `reconciled` (receipt path/hash). The ref may advance before the journal records `published-unreconciled`; recovery always checks Git rather than trusting a stale journal field.
1. Persist the proposal revision and evaluation intent using temporary-file write, flush and atomic replace. Journal events are separate immutable files with a recoverable sequence; an interrupted tail is detectable.
2. Validate outside the writer's critical section where possible. Prepare safe content-addressed artifacts and bounded file changes. Quarantine uploads cannot become canonical references before integrity checks succeed.
3. Acquire exclusive writer access. Recheck current authentication/grants, policy versions, base commit, dependency hashes and protected decision bindings. Serialize credential/grant changes with the final authorization check and publication so revocation cannot race that check. Do no parsing, model inference or external network I/O while holding this boundary.
4. Build the exact Git tree using a private temporary index and argument-vector Git commands. Include only changed canonical records and the acceptance manifest. Generate the commit, then publish the canonical branch with `git update-ref <ref> <new> <expected-old>` while holding the process-lifetime writer lock. Only a successful ref update makes the change accepted; orphan commit objects do not.
5. Persist the immutable external receipt with the published commit ID; reconcile stage state and any consumed owner decision in the control store. If a required reconciliation write fails, freeze all new canonical publications and return an indeterminate/recovering result until reconciliation succeeds. Do not call the committed mutation rejected.
6. Schedule derived indexing from the published Git revision. The response envelope reports `index-pending`, `indexed` or `index-failed` alongside the receipt. Direct accepted-revision reads remain available.
Canonical readers use a committed tree, never partially modified worktree files. A service-owned checkout may be materialized for inspection after publication. Manual edits to that checkout are unsupported; detect drift and do not import it automatically.
| Failure point | Recovery requirement |
| --- | --- |
| Before proposal durable write | Return failure; retry can submit again. |
| Artifact stored, no accepted reference | Keep quarantined/unreferenced until retention cleanup; no accepted knowledge exists. |
| Prepared tree or orphan commit, ref unchanged | Ignore for acceptance; retry against current head. |
| Ref updated, receipt absent | Locate the committed manifest, verify digest/authority bindings, recreate receipt and reconcile stage state. Never reapply the bundle. |
| Receipt exists, index behind | Replay index updates or rebuild; status remains accepted. |
| Index generation corrupt | Serve explicit unavailable/degraded results or canonical ID reads; rebuild from authoritative records and artifact maps. |
| Unknown/manual canonical mutation | Hold writes for operator diagnosis. Do not manufacture a receipt or silently reset history. |
Startup reconciles manifests and workflow records before accepting new writes. Every proposal retry and status lookup also checks its deterministic manifest path against a captured authoritative ref when the receipt is absent or inconsistent. If present, verify proposal/digest bindings and the introducing commit, reconstruct the identical receipt and consume the bound decision idempotently under the global publication/recovery gate. If required persistence still fails, return `RECOVERY_REQUIRED` and keep new publications frozen. If absent and the writer has no active attempt, resume/revalidate staged work. This applies before restart as well as afterward, including requests from a different principal.
Fault injection must kill the process at each boundary, including filesystem write failures and receipt failures. Test live retries, unrelated queued proposals and challenge replay during the recovery window. Process-crash tests do not establish power-loss durability; storage flush behavior and backup recovery require separate acceptance on the intended filesystem. Network/shared filesystems and concurrent writers are outside this baseline.
## Human ownership without per-note approval
Routine auto-acceptance is controlled by explicit production policy. Proposed initial candidates are additive source captures and factual notes/datapoints with valid locations and supported assessments. Same-agent assessment is permitted for routine factual notes, as confirmed in the PRD. This candidate list is awaiting owner selection; a confidence threshold alone never authorizes acceptance.
Policy separately enumerates shared-source metadata, entities, typed links, interpretations and syntheses. Missing rules hold affected changes. Destructive merges, promotion, protected project intent and autonomy policy need attributable exceptional authority. Link correctness must not be inferred merely from both endpoint nodes existing.
Proposed protected-action flow:
1. An agent stages a proposal and receives a public challenge ID and exact digest. The challenge is not a bearer approval.
2. The human launches the owner client outside the agent's execution account. It obtains the dedicated session defined below, retrieves the diff directly from MCP, and displays affected projects, policy effects, hashes and expiration.
3. The human confirms there. The owner client submits a decision through MCP; the service checks project/shared ownership and records a single-use authorization bound to proposal revision, digest, policy versions and expiry.
4. The service revalidates and accepts when every affected domain permits it. Changed content, changed authority/policy, expiration or withdrawal invalidates the decision. The agent receives status only and never receives owner credentials.
An admin bootstraps accounts, key issuance and owner grants through a separate operator control plane. This authority alone cannot bypass a protected knowledge decision. Owner decisions and their audit references survive index deletion. The account/bootstrap commands are proposed tooling, not currently runnable commands.
Isolation is a deployment fact. If the agent can execute as the owner and read the owner's secrets, this mechanism cannot prove human-only authorization. Use a separate OS principal/device or a protected interactive authenticator before claiming that gate passed. Do not invent human approval from conversational text supplied by the agent.
### Proposed owner authentication profile
Use the controlled GEWU owner CLI over HTTPS to the same service's Streamable HTTP MCP endpoint. Before MCP initialization, the CLI calls a separate `POST /auth/owner-session` control endpoint with human account ID, password, TOTP and challenge ID. Password/TOTP are read directly from the human terminal without echo, never arguments/environment variables, and discarded after use. The server verifies a maintained Argon2id password hash and TOTP with a pinned library (proposed `argon2-cffi` and `pyotp`), rate limits failures and prevents reuse of an accepted TOTP timestep. Enrollment/reset requires restricted operator identity and records a security event. TOTP seeds are encrypted under a service key kept outside the database and ordinary data-backup set. Maintain that key in a separately protected, encrypted, access-controlled recovery backup. A restore verifies the key before enabling owner operations; if unavailable, owner login stays disabled until an attributable operator reset and human re-enrollment complete. Test both recovery paths before team release.
On success, generate a random 256-bit opaque token, store its hash in the private control database, and return it once with a five-minute expiry. Store the human principal, current grant version, service audience, allowed challenge ID and capabilities `owner:inspect`/`owner:decide`. The CLI keeps the token in memory and presents it as the MCP bearer credential; it is rejected by ordinary browser sessions and agent-key validation paths. The token cannot create arbitrary proposals or authorize another challenge. Reauthentication is required after expiry. Network operations verify TLS and the configured service identity. This custom profile is explicitly limited to GEWU's controlled clients and makes no general OAuth-interoperability claim.
MCP operations: `owner_challenge_get(challenge_id)` returns proposal revision/digest, affected domains/policy versions, expiry and status; `owner_diff_get(challenge_id)` returns the exact bounded diff and its digest; `owner_decide(challenge_id, expected_proposal_revision, expected_payload_digest, expected_diff_digest, decision, rationale)` accepts an attributable authorize/reject decision after rechecking current grants. Oversized diffs must be fully paginated and hash-bound before confirmation; they cannot be silently truncated.
Challenges live in the control database with states `issued`, `authorized`, `consumed`, `expired` or `invalidated`, plus immutable transition events. Only an unexpired `issued` challenge can acquire an immutable owner decision. Exact decision retries return the same decision; altered retries fail. Multiple owners can supply separate decisions for a bundle, each bound to its domains. An authorization is consumed by one acceptance manifest containing the full non-secret decision snapshot and digest. The global writer/recovery gate ensures the manifest is checked before any decision can be reused. A crash before the SQLite consumed flag is updated is recovered from that manifest; a missing flag cannot permit another acceptance. Rejection invalidates the challenge and records its decision without publishing the proposed knowledge.
Contract tests must cover owner session used as an agent token, browser/agent token used for owner operations, expired/wrong-audience token, revoked grant, TOTP reuse, altered diff, duplicate decision and crash-before-consumed-state. The profile remains proposed until implemented and reviewed on the actual client/service deployment.
Library basis checked 13 October 2026: [argon2-cffi password verification](https://argon2-cffi.readthedocs.io/en/stable/howto.html) and [PyOTP](https://pyauth.github.io/pyotp/). PyOTP documents replay rejection, secret protection and throttling as application responsibilities. Neither library establishes human/agent OS isolation or makes this profile phishing-resistant; a hardware-backed authenticator is a possible later replacement for the owner login adapter.
## Source transfer and evidence locations
The stdio connector reads only an explicitly selected path within configured roots. Reject devices, directories and reparse/symlink escapes; revalidate the opened handle/file identity to avoid path races. It transfers bytes to service MCP upload-session/chunk/finalize operations. Never echo source bytes or credentials into model-visible upload responses.
Proposed initial limits for probing: 50 MiB per artifact, 512 KiB binary chunks, two concurrent uploads per credential, ten-minute idle upload expiry, 100 operations per proposal and 1 MiB proposal JSON. The JSON wire limit accounts for base64 expansion. Advertise effective limits in diagnostics; tune only from measured fixtures and operational limits. These values do not constitute owner-approved archive capacity.
Chunk retries bind session/offset/content hash. A duplicate matching chunk succeeds idempotently; changed content at the same offset fails. Finalization verifies full length and SHA-256 before creating an immutable artifact. Upload completion is staging, not source acceptance. Resume reports verified offsets; expired sessions fail explicitly.
Locators use zero-based Unicode code-point offsets and half-open spans into a specified immutable extraction block. PDF page numbers are one-based. Text/HTML locators name a block ID; timestamped transcripts additionally name the time interval. A quote must match at that locator after the declared normalization profile. Normalization may standardize line endings and Unicode form, but cannot silently rewrite numbers, units or qualifiers. Record the normalized text alongside the original map.
The server derives or verifies extraction maps from captured bytes with its pinned parser. Agent-supplied extracted text cannot substitute for an archived PDF's location map. Encrypted, scanned, multi-column or mathematical sources may require a specialized parser/OCR adapter; return an explicit unsupported/unavailable extraction state until that path is verified. Preserve originals and every cited extraction version when parsers change.
### Immutable extraction format and rebuild procedure
Store one canonical UTF-8 JSON extraction artifact with `schema_version`, source artifact hash, parser name/version/config digest, normalization profile, extraction status/warnings, and ordered `blocks`. Each block has `block_id`, ordinal, original extracted text, normalized text, page number or time range, and a normalized-to-original code-point range map. This map relates text representations; it does not claim byte offsets into compressed PDF bytes. PDF page/block association is mandatory; geometry is optional and explicitly absent when unavailable.
Within one extraction artifact, derive `block_id` from the block ordinal and hash of original extracted text. IDs are stable only within that immutable extraction version. Hash canonical serialization of the complete artifact excluding its self-ID/hash field to obtain `extraction_id`; store under that hash. Record hashing/serialization version and test vectors. A citation resolves `(source_revision, extraction_id, block_id, start, end)` against these stored bytes. Normalization maps must resolve every quoted normalized span to original-text ranges. Parser upgrades create new artifacts and never reinterpret old locators.
A full rebuild captures a canonical Git head `H`, enumerates only its accepted record manifests and referenced source/extraction IDs, validates schemas and hashes, and builds an isolated index generation. Read exact stored extraction artifacts; never rerun a parser to resolve an existing citation. Record `H`, index schema, extraction profiles, embedding artifact/profile and corpus inventory hash as the generation watermark. Missing/corrupt authoritative artifacts fail the rebuild visibly rather than dropping affected nodes. After integrity/retrieval checks, publish the generation pointer under the service's index-publication gate, then apply complete commits after `H` in order. A newly published generation may report lag, but must never claim a later head before those commits are indexed. Crash recovery uses the generation watermark and replays unapplied commits; incomplete generations are never selected.
Reference-only sources remain discoverable with availability warnings. Their unsupported assertions stay outside the accepted evidence-backed corpus. A quote-location pass establishes occurrence; semantic support, source reliability, confidence and freshness remain separate fields.
## Contract errors and acceptance fixtures
Errors carry a stable code, safe message, request/proposal identity, field findings and retry guidance. Initial codes include `SCOPE_REQUIRED`, `OUT_OF_SCOPE`, `UNAUTHORIZED`, `OWNER_DECISION_REQUIRED`, `POLICY_MISSING`, `EVIDENCE_MISMATCH`, `SUPPORT_UNAVAILABLE`, `BASE_CONFLICT`, `IDEMPOTENCY_CONFLICT`, `BUDGET_TOO_SMALL`, `UPLOAD_EXPIRED`, `BUSY` and `RECOVERY_REQUIRED`. Internal paths, raw SQL and secrets remain in neither client errors nor routine logs.
M0 must implement adversarial fixtures for wrong-location matching quotes; wrong numeric units with valid quotes; changed assertions with old assessments; cross-policy bundles; stale owner approval; revoked key at publication; changed payload replay; scope escape through redirects; crash-after-ref-before-receipt; index lag; and whole-group token budgets. These fixtures are required work, not tests claimed to exist.
docs/architecture/ARCHITECTURE.md
+157
View File
@@ -0,0 +1,157 @@
# GEWU architecture
Status: proposed baseline v0.2 after [independent review](../reviews/ARCHITECTURE_SENIOR_REVIEW.md). Drafted 12 October, revised 13 October 2026. Product authority: [GOAL](../GOAL.md) and [PRD](../PRD.md). Related: [acceptance contract](ACCEPTANCE_CONTRACT.md).
## Decision
Build a modular Python service and a React reading application in the GEWU code repository. Deploy one authoritative knowledge repository per installation, shared across projects. The knowledge repository and private runtime data live outside the code checkout and outside agent-writable mounts.
Local agents reason about sources and submit proposals. The service validates their structure, source locations, permissions and policy eligibility, then records accepted changes. People can search and follow evidence without an agent running. The service contains no reasoning agent, autonomous research loop or general-purpose chat endpoint.
The starting deployment is one host, one service instance and one canonical writer. This fits the five-person pilot and makes the acceptance boundary inspectable. Capacity remains a measurement gate; five people does not specify simultaneous agent load.
## Technology choices and reasons
These are proposed baselines, not installed versions. Use Python 3.12 as the initial compatibility target, Pydantic 2, React 19 and TypeScript strict mode. Resolve supported releases of the other packages at scaffolding, pin exact versions in `uv.lock` and `package-lock.json`, and record Python, Node and SQLite runtime versions. Do not deploy floating `latest` dependencies. Any version conflict reopens the relevant choice before implementation.
| Boundary | Choice | Why for GEWU | Alternative and reconsideration trigger |
| --- | --- | --- | --- |
| Service | Python, FastAPI, Pydantic, Uvicorn; `uv` for environment/locking | Typed validation and the document-processing ecosystem; consistent with useful AAA/Jev patterns. | A TypeScript-only service saves a language but complicates Python parser integration. Revisit if parser requirements disappear. |
| Domain core | Plain typed Python modules, explicit dependency injection | The same scope, evidence and proposal rules serve MCP, reads and maintenance. | No generic pipeline/plugin framework until distinct implementations need one. |
| MCP | Official Python MCP SDK; local stdio connector, service Streamable HTTP | One contract across agent harnesses; connector can read selected local files and upload bytes. | Direct remote connections remain a tested option for clients that support the chosen auth profile; a remote path is never a file upload. |
| Canonical knowledge | UTF-8 Markdown with YAML frontmatter, YAML records, native Git CLI | Reviewable records, exact history, portable exports. | A database as canonical storage conflicts with the present PRD. |
| Private artifacts | Content-addressed filesystem by SHA-256 | Retain originals without growing Git with PDFs/media. Git records hashes and manifests. | Object storage only after one-host capacity/backup measurements justify it. |
| Search/graph | SQLite FTS5, ordinary edges tables, `sqlite-vec` behind a vector adapter | Rebuildable on one host; no graph/vector service to operate. | `sqlite-vec` is a gated candidate because its upstream describes pre-v1 compatibility risk. If portability/performance fails, revise the adapter choice; full v1 still needs hybrid search. |
| Embeddings | Local Sentence Transformers adapter; multilingual-e5-small is the first evaluation candidate | Offline execution and a multilingual path without hosting a reasoning agent. | Final model/artifact hash depends on the project's labeled language corpus, license and CPU/RAM measurements. API providers are optional and must be explicitly configured. |
| SQL access | Standard `sqlite3`, parameterized SQL, explicit repository methods | FTS/vector queries are easier to audit directly; the index is disposable. | SQLAlchemy/PostgreSQL become candidates only if operational state or deployment topology warrants the cost. |
| Web | React, TypeScript, Vite, React Router | Search, evidence drawers and linked reading views; static production assets require no Node server. | Next.js SSR adds a second server without a present SEO or server-rendering requirement. |
| Browser state | TanStack Query for server data; URL for scope/filters/selection; React local state for transient controls | Cache ownership and cancellation are explicit. | Do not copy AAA's bespoke telemetry stores into a reading application. Introduce another store only for demonstrated shared client state. |
| Visual implementation | CSS custom properties + CSS Modules; Radix primitives where needed; Lucide icons | Small, application-owned visual vocabulary with accessible interaction foundations. | Tailwind/shadcn is viable, but adds conventions without a present component system to reuse. No mixed styling systems by default. |
| Source/rendering adapters | `pypdf` for initial PDF extraction, `markdown-it-py` for Markdown structure, `trafilatura` for HTML text, `react-markdown` + GFM + sanitized math rendering in the UI | Keep parser-specific behavior behind versioned extraction contracts. | OCR and more complex parsers require a quality/cost probe. PDF extraction success is not locator or reading-order correctness. |
| Quality tools | pytest, Ruff, mypy; ESLint, TypeScript, Vitest/Testing Library, Playwright and axe | Cover contracts, recovery and actual browser reading workflows. | No arbitrary coverage percentage in place of behavioral acceptance. |
| Hosting | Linux VPS, Docker Compose, Caddy HTTPS; native local development | Repeatable single-host service and private volumes; Python serves built UI assets. | Kubernetes, Redis and distributed queues are deferred until measured need. |
The exact dependency inventory is a scaffolding deliverable. Model selection and native vector packaging are explicit prototype gates, not reasons to pretend the stack is already verified.
## Components and dependencies
```mermaid
flowchart LR
H[Human reader] --> W[React website]
W --> R[Read HTTP adapter]
A[Local agent harness] --> C[Local stdio MCP connector]
A --> R
C --> M[Service MCP adapter]
O[Human owner client] --> M
R --> Q[Scoped query services]
M --> Q
M --> P[Proposal and acceptance services]
P --> V[Schema, evidence and policy validators]
P --> G[Exclusive Git writer]
G --> K[Canonical knowledge repository]
P --> J[Durable staging and run records]
P --> B[Private source artifacts]
K --> I[Index worker]
I --> S[Derived SQLite index]
Q --> S
Q --> K
```
Imports point inward: transport -> application -> domain. Adapters implement storage, Git, clock, parser and embedding interfaces used by application services. Domain code imports neither FastAPI nor MCP nor React. A small lifespan-owned composition root creates services and closes resources. Each application service owns a complete use case; routes translate protocol messages and enforce transport admission.
Recommended code layout:
```text
src/skb/
domain/ # node, evidence, scope, policy, proposal contracts
application/ # intake, evaluate, accept, query, runs, maintenance
adapters/ # git, artifacts, records, sqlite, parsers, embeddings, auth
transports/ # MCP, read HTTP, operator/maintenance CLI
bootstrap/ # configuration and lifespan assembly
connector/ # local stdio client bridge; uses shared contract package
web/src/
app/ # routes, providers, shell
features/ # search, reading, sources, problems, history
components/ # shared accessible primitives
api/ # generated types, typed client, query keys
styles/ # semantic tokens and reading typography
schema/ # generated/versioned public schemas and fixtures
tests/ # unit, contract, integration, recovery, fixtures
docs/ # product, architecture, protocols, decisions, acceptance
```
Do not create empty layers or adapter classes for every file. A module must own a real responsibility. Share public schema definitions between service and connector; generate browser types from the read API. Type generation complements runtime validation.
## Persistence classes
| Store | Contents | Rebuildable? |
| --- | --- | --- |
| Private knowledge Git repository | Projects, nodes, source manifests, accepted evidence/assessments, policies, redirects, acceptance manifests | Authoritative; restore from backup. |
| Private artifact tree | Original bytes and immutable extraction/location maps, keyed by hashes | Required source evidence; restore from backup. A parser rerun does not replace a cited extraction version. |
| Durable staging/run tree | Proposal revisions, validation/decision events, uploads, external-run progress and receipts | Authoritative workflow history, written atomically; restore from backup. |
| Private control SQLite | Human accounts/password hashes, sessions, agent credential hashes, role/ownership grants, owner challenges and security audit | Operational authority; not the disposable search index. Backup with restricted access. |
| Derived SQLite index | Searchable nodes, FTS, edges, vectors, index metadata | Delete and rebuild without losing acceptance, evidence or identity. |
Distinguish the code repository from the knowledge repository. The PRD's single-repository rule means all knowledge projects share one canonical repository; it does not require committing private team sources into the application's code repository.
Git paths use safe storage keys rather than raw logical IDs. IDs such as `shared:paper-name` remain metadata, avoiding Windows filename restrictions. Use bounded safe YAML loading, reject duplicate keys and unknown fields, and emit stable ordering/UTF-8/LF. References use logical IDs and revisions, never caller-selected filesystem paths.
## Reads and retrieval
Every content query carries scope. Resolve registered projects and their explicit includes before search or ID resolution; return `resolved_scope`. All-team visibility does not authorize silently widening relevance scope. Redirects and graph traversal recheck destination scope.
Search runs scope-filtered FTS and vector candidates, merges them with reciprocal rank fusion, and performs bounded typed-edge expansion. Start with a maximum one-hop expansion and explicit candidate/edge caps in the public contract. Return ranking explanations separately from evidence status. Related suggestions never become accepted relationships by indexing.
Use normalized English token search plus language fixtures. `unicode61` alone is not a claim of Chinese word segmentation; test CJK queries and use a documented segmentation or trigram adapter when needed. Evaluate short queries separately. Search acceptance is the PRD's labeled per-project question set, not a synthetic speed result.
Index updates publish all node/edge/FTS/vector changes for a chosen Git revision in one SQLite transaction. Queries use a read transaction pinned to an index revision. A model change builds a separate index generation, with model ID, artifact hash, dimensions, normalization, chunking and parser versions. Publish the new generation only after integrity and retrieval gates pass. Avoid replacing an open database file on Windows: switch the service's generation pointer and retire old connections before cleanup. The acceptance contract specifies the immutable extraction artifact and rebuild watermark procedure.
`get_node` may read a requested accepted revision directly from Git before indexing. Search reports its indexed revision and lag. Embedding outages permit clearly labeled keyword-only reads; they do not relax acceptance rules or count as full-v1 hybrid acceptance.
Budget whole serialized result groups, including citations and warnings. Proposed counting profile: a pinned `tiktoken` package with the `o200k_base` encoding and hash-verified encoding assets, identified in each response and preinstalled for offline use. This is an API accounting convention, not a promise to equal every harness model's tokenizer. Define the error-envelope minimum separately. Never cut a quote or discard a stale warning to meet a budget.
## Process and concurrency model
Run one Uvicorn application worker initially. A process-lifetime OS lock excludes a second canonical writer on the same local filesystem; a repository head compare-and-swap check rejects stale publication. Do not deploy two writers with independent in-memory locks or accept shared NFS storage as equivalent.
Acceptance requests enter a bounded queue whose pending work is recoverable from staging records. Critical acceptance is not a fire-and-forget FastAPI background task. Blocking disk/Git calls cross one bounded worker boundary; extraction and model work execute in capped child processes with timeouts. Workers return artifacts with input hashes; only the service can publish canonical state. The [contract](ACCEPTANCE_CONTRACT.md) defines ordering and recovery.
A revoked credential, changed policy or changed dependency invalidates a prepared result before publication. Cancellation of a caller does not undo an already accepted proposal; status lookup resolves uncertainty.
## Authentication and deployment
Human browser sessions are expiring HttpOnly, Secure, SameSite cookies. The read API accepts those sessions with read capabilities only, including for an admin. It also accepts agent bearer keys for scoped, budgeted retrieval under the same read contracts used by MCP. Authentication endpoints may establish/end sessions but cannot author knowledge. Agent API keys are unique, revocable, server-bound to an agent and responsible human, stored as hashes at the service and outside Git at the connector.
For the initial controlled-team connector, document a GEWU provisioned-key authentication profile over TLS. Do not describe this as generic OAuth interoperability. HTTP MCP authorization has its own specification; supporting arbitrary remote OAuth clients requires a separately implemented and tested authorization-server/resource-server profile. The local stdio connector avoids assuming all four harnesses support the same remote authentication flow.
Protected changes use the human-authenticated owner MCP client and a digest-bound decision described in the contract. The proposed owner profile uses a short-lived opaque session obtained by password and TOTP reauthentication in a separate human terminal; it is an GEWU-controlled client profile, not a claim of generic OAuth support. Production separates the human owner credential from the agent's OS account/credential store. Owner-level unrestricted filesystem agents cannot be claimed to be isolated by prompt instructions.
Local operation uses the same core and directory layout as the VPS. Development may run on the user's account with disposable data; enforced pilot/team operation uses a separate service identity. VPS publication requires an independent restore test, HTTPS, private volumes and client workflow evidence. Moving authority to the VPS freezes local writes first; there is no independently writable local replica synchronization in v1.
## Delivery and open decisions
| Gate | Work | Observable exit |
| --- | --- | --- |
| M0 contract slice | Schemas, state machine, safe file records, scope/policy fixtures; pin dependencies | Seeded invalid evidence/authority rejected; public schemas generated; decisions below settled for affected boundary. |
| Pilot end-to-end | Proposed starting point: Windows/Codex, text/Markdown, two projects plus shared, lexical search first | Upload -> proposal -> automatic acceptance -> committed read -> citation; crash/replay tests; read-only browser. Client/format choice remains recommended. |
| Full P0 service | Hybrid index, all required input types, four harnesses, portability and protected decisions | Per-client transfer/rejection/acceptance/retry/retrieval receipts; no scope or authority bypass. |
| Team release | VPS migration, backup/restore, quality sampling, capacity/security acceptance | All selected PRD P0 gates plus measured retrieval, evidence and human reading tasks. |
| P1 follow-up | Rich comparisons, graph canvas, promotion/import/export and synthesis/curation | Explicitly selected requirements and their own acceptance evidence. |
Still needs owner input: first admin and project/shared owners; production auto-accept policy; sampling owner/cadence/error limits; representative sources and expected answers; supported initial client/format; VPS capacity, concurrent load, archive retention, recovery time and recovery-point targets. Production credentials and policies cannot use silently invented answers.
Technical probes needed: native vector package on Windows/macOS/Linux; parser locator fidelity including scanned/math PDFs; final embedding model and language set; cold rebuild under the PRD's ten-minute target. Measure cached-vector index rebuild and cold re-extraction/re-embedding separately; do not rename a warm rebuild to claim the full target passed.
## Research basis
Checked 12 October 2026. These sources establish available mechanisms, not GEWU performance or compatibility:
- [SQLite WAL](https://www.sqlite.org/wal.html): WAL allows reader/writer concurrency but retains one writer; motivates short index transactions and one-host operation.
- [SQLite FTS5](https://www.sqlite.org/fts5.html): tokenizers and substring search options; language quality needs our fixtures.
- [sqlite-vec upstream](https://github.com/asg017/sqlite-vec): embedded vector implementation and explicit pre-v1 warning; requires an isolated adapter and pinned packaging.
- [Official Python MCP SDK](https://github.com/modelcontextprotocol/python-sdk), [versioned MCP transports](https://modelcontextprotocol.io/specification/2025-11-25/basic/transports), and [HTTP authorization](https://modelcontextprotocol.io/specification/2025-11-25/basic/authorization): use negotiated compatible versions; do not implement draft changes by accident.
- [FastAPI process workers](https://fastapi.tiangolo.com/deployment/server-workers/): multiple workers are separate processes; an in-process lock cannot serialize their writes.
- [Pydantic strict validation](https://docs.pydantic.dev/latest/concepts/strict_mode/), [Vite](https://vite.dev/guide/), [TanStack Query](https://tanstack.com/query/latest/docs/framework/react/overview): typed boundaries, static assets and server-state tooling supporting the choices above.
- [pypdf extraction limitations](https://pypdf.readthedocs.io/en/stable/user/extract-text.html): extraction and OCR are different capabilities; quality must be measured.
- [Sentence Transformers model loading](https://www.sbert.net/docs/package_reference/sentence_transformer/model.html) supports pinned revisions and local-only loading. The [multilingual-e5-small model card](https://huggingface.co/intfloat/multilingual-e5-small) was available in search results; direct retrieval failed, so its full configuration/license remains a mandatory probe rather than a verified selection.
- [tiktoken](https://github.com/openai/tiktoken) provides named encodings for a reproducible budget convention. [Git update-ref](https://git-scm.com/docs/git-update-ref.html) supports an expected-old object check for publication.
docs/architecture/LOCAL_AGENT.md
+77
View File
@@ -0,0 +1,77 @@
# GEWU local agent
13 October 2026. Owner requirement: a custom local agent built like the agents in the RESEARCH workspaces. Delivery form confirmed by inspecting Winnow in SCAR and Praxis in CV: a workspace profile, shared protocols, modular skills and coordinating pipeline instructions executed by an existing host application.
## Architecture decision
Ship a GEWU research-agent pack that runs inside a supported local agent host. The host supplies reasoning, model access and tool execution. GEWU supplies its role, task-specific skills, pipeline procedures and MCP connection contract. A new standalone agent runtime is outside this addition.
The pack is a first-class GEWU client and follows the same permissions as other clients. The knowledge service remains independent: it validates proposals, applies acceptance policy and stores records; the agent interprets sources and drafts knowledge. Reading and evidence navigation continue when no agent is running.
Local execution describes where the host and working files live. A configured cloud model may still receive content. Offline reasoning requires a host with a configured local model; neither the pack nor a local folder guarantees that data stays on the computer. Keep model/provider selection in the host configuration and record the actual model where available.
## Source and installation boundary
The source pack lives at [agents/gewu](../../agents/gewu/README.md) in this repository. Its own `AGENTS.md` defines the research role; it is not the development repository's global persona. This avoids turning every coding session into a research ingestion session.
The intended installed layout resembles the RESEARCH workspaces:
```text
<research-workspace>/
AGENTS.md # thin discovery pointer to canonical profile
.agents/
AGENTS.md # GEWU role and routing
manifest.json # pack/skill/pipeline versions and readiness
protocols/ # shared evidence, scope and proposal rules
skills/<name>/SKILL.md
pipelines/ # coordinating procedures and checkpoint fields
Inbox/ # explicitly selected local source inputs
Drafts/ # unaccepted draft artifacts
Runs/ # private local checkpoints/receipts
```
The checked-in source pack can be linked or installed into the canonical `.agents` directory. Host-specific configuration/pointers and a copy fallback must preserve other installed skills and agent profiles. A partial single-skill copy must also include its referenced protocols/pipelines. Do not overwrite an existing root persona or claim host discovery merely because files exist. No host configuration, symlink or user-global installation is performed by this change.
The live reference inspection found SCAR `.agents` points to `.opencode`; CV owns `.agents` directly. Both use workspace rules and reusable skills. GEWU adopts the organization, while changing their direct-file contribution behavior to its MCP proposal contract. Source reads were limited to profiles, skill templates and representative pipelines; their runtime behavior was not retested.
## Responsibilities and replaceable modules
| Part | Owns | Replacement boundary |
| --- | --- | --- |
| Agent profile | Role, routing, project context and collaboration behavior | Update profile version without copying workflow logic into it. |
| Protocols | Shared evidence, scope and contribution guidance | Referenced by concern; authoritative server validation still lives in service code. |
| Skills | Ingest, retrieve, curate or synthesize one coherent kind of task | Versioned inputs/outputs, tool needs and acceptance criteria. |
| Pipelines | Order skills/stages, identify checkpoints, route corrections and report coverage | Separate files reference skills; a changed coordinator need not rewrite every skill. |
| Host adapter | Discovery, permitted tools and model/runtime configuration | Preserve the same public skill and MCP contracts across hosts. |
| Local connector | Safe file transfer, MCP transport and diagnostics | Shared connector contract; contains no semantic decision authority. |
The [modularity protocol](../development/MODULARITY_PROTOCOL.md) applies to all these parts. Pipeline files are procedures interpreted by the host agent, not an executable scheduler or durable engine. MCP run records and receipts provide durable service-side progress. Local checkpoints support recovery but cannot prove acceptance.
## Initial workflows
| Skill | Pipeline | Output |
| --- | --- | --- |
| `kb-ingest` | Context -> source capture -> triage -> evidence-bound extraction -> support assessment -> proposal -> receipt | Accepted IDs/receipts, held drafts, coverage/skips and actionable findings. |
| `kb-query` | Scope -> retrieve -> inspect evidence -> bounded answer | Cited answer with scope, as-of dates, uncertainty and omissions; no writes. |
| `kb-curate` | Scoped audit -> inspect candidates -> propose corrections/relations -> receipts | Traceable changes or a report, preserving disagreements and old revisions. |
| `kb-synthesize` | Accepted inputs -> dependency inventory -> compare/compose -> evidence audit -> optional proposal | Labeled local synthesis draft, or accepted synthesis after a real receipt. |
The four skills are instruction artifacts in this change. Service schemas, executable tools and client acceptance remain unimplemented. The manifest declares no verified service compatibility yet. The pack can organize explicitly requested local drafts; it must report the missing service instead of fabricating tool calls, source revisions or accepted knowledge.
## Run and resume contract
At run start, record local run ID, authenticated service/agent identity when connected, explicit scope, task objective, pack/skill/pipeline versions and content hashes, host/model label, input hashes and budgets. Resolve tools and schemas from service diagnostics. Unsupported tools or schemas produce a clear blocked stage. Stage records name their inputs, outputs, disposition and last durable receipt.
For a connected run, service run identity, proposal revisions and acceptance receipts are authoritative. Resume queries them before retrying an uncertain upload/proposal result. Model output is not deterministic: preserve completed output artifacts and their hashes rather than asking the model to regenerate the same content and assuming identical IDs.
Pin the installed pack snapshot for an active run. If that version is unavailable after an upgrade, pause with a compatibility finding or explicitly start a successor run. Reuse previous outputs only when their hashes, evidence bindings and new stage contracts still hold; revalidate downstream stages after changes. Install/version manifests alone cannot enforce this in arbitrary hosts; native workflow tests must demonstrate it.
A pipeline has a bounded work/correction budget, explicit coverage and a resumable stop. Two unchanged validation failures are a default stop for diagnosis, not permission to relax checks. Network uncertainty triggers status lookup, not a fresh proposal identity. Protected decisions leave the agent workflow for the separate human owner client. No continuous watcher, automatic external messaging or unbounded research loop is introduced.
## Acceptance and sequencing
First finish the service/tool contracts, then install the pack into a disposable research workspace and run a complete ingestion/read flow with the selected pilot host. Verify invalid-evidence rejection, uncertain-result retry, run resumption, same-agent assessment attribution and shared-source reuse. Query must still work with a read-only credential.
Before full-v1 client claims, repeat the relevant workflow with Codex, Claude, Antigravity and OpenCode on the declared platform matrix. The custom profile is an additional deliverable using those hosts, not an additional fifth runtime. Synthesis/advanced curation remain under their existing P1 service feature priorities; pack files do not promote those features to implemented P0.
Record structural skill validation separately from installed discovery, tool connection, semantic quality and end-to-end acceptance. No live client, server or model workflow has passed as part of this design addition.