docs: clarify local file connector setup

This commit is contained in:
vasily committed 2026-10-17 13:20:52 +08:00
1 parent 511e74d1a3
commit 9252015572
7 files changed
+62 -38

No files matched your search

+2 -2
View File
@@ -1,11 +1,11 @@
# Ingestion pipeline
Version 0.5.1. Host-executed procedure for `kb-ingest`; not a background scheduler. Follow the contribution protocol for budgets, checkpoints and retry identity.
Version 0.5.2. Host-executed procedure for `kb-ingest`; not a background scheduler. Follow the contribution protocol for budgets, checkpoints and retry identity.
| Stage | Input | Output / checkpoint | Gate |
| --- | --- | --- | --- |
| Context | User-selected material, explicit project and `get_project_context` lens | Objective, authorized project lens including four default research lenses and saved guidance, scope, run identity and pinned pack versions | Required tools/schema are available; otherwise report blocked. |
| Capture | Explicitly selected local files, user-pasted text or an allowlisted public URL | Verified artifact digest, stable source ID, content-bound revision ID, metadata candidate and extraction availability | Use the connector for local files, `source_capture_text` for explicitly supplied UTF-8 text/Markdown (maximum 1 MB), or `source_capture_url` for a configured public host. Pasted text requires a stable capture ID and source key, and returns an archived artifact, extraction and candidate marked `pasted-text`; retry with the same ID and identical inputs after uncertainty, since changed inputs conflict. The service retains at most 1,024 private capture receipts. The capture does not register the source. The connector returns a typed `source_candidate`; `.vtt` and `.srt` files are parsed into cue-timed blocks and tagged `transcript`. Cite the precise cue times together with the quote span. The connector can record a declared NotebookLM export provenance for supported text, Markdown or PDF files, but does not verify that provenance; omit `source_revision_id` to derive a revision from file identity, content digest and its durable retry checkpoint, or supply a stable revision ID when resuming an existing workflow. Submit candidates through `proposal_submit`; staging alone does not register a source. Failed fetches return a `capture-failed` candidate without artifact bytes; register it only when a bookmark is useful. PDF extraction currently requires the supported Linux sandbox. HTML citations refer to visible extracted text, not markup-byte offsets. |
| Capture | Explicitly selected local files, user-pasted text or an allowlisted public URL | Verified artifact digest, stable source ID, content-bound revision ID, metadata candidate and extraction availability | For local files, require the separate local `gewu-connector` MCP server, discover `list_selected_files`, then upload a listed file ID with the authorized project ID. The service HTTP MCP cannot read local paths; the connector streams and checkpoints file bytes internally. If it is unavailable, report the connector/file setup blocker. Never send a whole-file base64 payload through conversational MCP or invent a one-shot upload script. Use `source_capture_text` only for text/Markdown explicitly supplied by the user (maximum 1 MB), or `source_capture_url` for a configured public host. Pasted text requires a stable capture ID and source key, and returns an archived artifact, extraction and candidate marked `pasted-text`; retry with the same ID and identical inputs after uncertainty, since changed inputs conflict. The service retains at most 1,024 private capture receipts. The capture does not register the source. The connector returns a typed `source_candidate`; `.vtt` and `.srt` files are parsed into cue-timed blocks and tagged `transcript`. Cite the precise cue times together with the quote span. The connector can record a declared NotebookLM export provenance for supported text, Markdown or PDF files, but does not verify that provenance; omit `source_revision_id` to derive a revision from file identity, content digest and its durable retry checkpoint, or supply a stable revision ID when resuming an existing workflow. Submit candidates through `proposal_submit`; staging alone does not register a source. Failed fetches return a `capture-failed` candidate without artifact bytes; register it only when a bookmark is useful. PDF extraction currently requires the supported Linux sandbox. HTML citations refer to visible extracted text, not markup-byte offsets. |
| Budget checkpoint | Current source stage and run budget | Per-source coverage checkpoint, remaining file/section plan and resume point | Process one source at a time. Use metadata to target relevant bounded slices; retain concise notes and exact evidence excerpts, not full source text. Checkpoint after each source. If quota or tool budget is near exhaustion, save the exact resume point and report remaining coverage without dropping a research lens. |
| Triage | Source plus project lens and four default lenses (technical, conceptual, cultural-historical, financial-market) | Included sections, explicit clear-exclusion reasons and review flags | Treat lenses as coverage prompts, not quotas. Historical, philosophical, cultural or commercial evidence is not automatically out of scope. Exclude only clearly unrelated or unavailable material, or user-excluded topics. Continue normal evidence-backed proposal processing for plausible material; add `ingest_review_create` as a follow-up notification/filter flag when human relevance review would help. |
| Duplicate check | Candidate findings and explicit readable project/shared scope | Existing-node candidates, reuse/update decision, or explicit unresolved-review flag | Before creating a node, run multiple lexical searches over its title, distinctive terms and likely synonyms, then inspect plausible matches with `get_node`. Reuse the stable node ID and preserve supported content for a true update; do not create a second node when it represents the same concept. Keep materially distinct conditions as distinct records. The backend flags exact entity name/alias matches only and does not deduplicate other node types. |