30 lines
10 KiB
Markdown
30 lines
10 KiB
Markdown
---
|
|
name: kb-ingest
|
|
description: Process explicitly selected sources for a GEWU project into evidence-bound MCP proposals, with support assessments, coverage and resumable receipts.
|
|
---
|
|
# Ingest selected sources
|
|
|
|
Read the pack's [evidence](../../protocols/EVIDENCE.md), [scope](../../protocols/SCOPE.md) and [contribution](../../protocols/CONTRIBUTIONS.md) protocols, then use the [ingestion pipeline](../../pipelines/ingest.md).
|
|
|
|
Inputs are the selected source(s), explicit project and the user's research objective. Use `get_project_context` for the project's north star, key questions, watch list, taxonomy extensions, research lenses and saved ingest guidance. Every project defaults to four research lenses: technical, conceptual, cultural-historical and financial-market. Treat these as coverage prompts, not quotas. Verify the supported tool/schema contract, including `ingest_review_create` when borderline-review intake is available. Use the connector for local byte transfer; it returns a typed source metadata candidate without exposing the selected path or bytes. Selected `.vtt` and `.srt` files produce timestamped transcript cues; cite exact cue start/end times and the quote span. A declared NotebookLM export label is metadata only and is not export verification. For text explicitly pasted by the user, use `source_capture_text` with a stable capture ID and source key; it accepts UTF-8 plain text or Markdown up to 1 MB, archives and extracts it, and returns a typed candidate marked `capture_method=pasted-text`. Retry uncertain calls with the same capture ID and identical inputs; changed input under that ID conflicts. It does not register the candidate. `source_capture_url` fetches only operator-allowlisted public hosts and stages supported text, Markdown, HTML or PDF bytes; it also returns a source metadata candidate but does not register it. Add each new source's exact candidate as `register_source` in the proposal that cites it, or verify the source was already registered and readable. Use the candidate's `revision_id` and capture's `extraction_id`; never invent or substitute these IDs. Submit candidates with the normal typed proposal flow, and use `capture-failed` candidates only as bookmarks, never as factual evidence. URL source IDs follow the sanitized URL; local source IDs follow the connector's configured file ID; pasted-text source IDs are derived from the supplied stable key and revision IDs include the content digest. PDF extraction currently requires Linux bubblewrap, Poppler and prlimit. Unsupported formats and unavailable parsers must be reported without inventing an extraction. If the service is unavailable, report the blocked stage and only prepare an unaccepted local draft when that work was requested.
|
|
|
|
Extract useful atomic assertions, problems, techniques or cases with their conditions across all four lenses where the source contains relevant material. Consider cultural history, philosophical context and market/financial evidence as project data when they explain the project's origins, concepts, adoption, audience, costs or viability; they are not automatically peripheral. First try to connect uncertain material to the north star, a key question, a saved rule or one of the lenses. Exclude only clearly unrelated material, unavailable evidence, or material the user explicitly excluded. Do not defer a section merely because its relevance is indirect, non-technical or historical. Continue normal evidence-backed proposal evaluation/submission for all plausible useful findings. When a human should review relevance or categorization, submit a bounded `ingest_review_create` flag with the exact extraction excerpt/locator, lens, summary and reason before finalizing the associated proposal facets. This persists for notification/filtering and never blocks the evidence-backed proposal flow.
|
|
|
|
Before building each proposal, use the principal ID assigned to the configured agent credential for `proposer_id`, node `agent_id` and assessment `assessor_id`; do not use the model or harness name. Finish each claim and its evidence before computing `assertion_digest` or `evidence_digests`. Evidence assertion paths must point to their exact claim field. Recompute the evidence and assessment digests after any claim/evidence edit, then recompute `payload_digest` after any operation edit. Follow the exact canonical digest rules in [CONTRIBUTIONS.md](../../protocols/CONTRIBUTIONS.md#build-attributable-proposals) and [EVIDENCE.md](../../protocols/EVIDENCE.md). If the identity or digest inputs are unknown, stop and report the missing information rather than guessing.
|
|
|
|
For `tags`, `audience`, `use_for` and `domain`, inspect the `facet_labels` catalog returned by `get_project_context`. Reuse a canonical project label when its meaning fits; do not create spelling, phrasing or singular/plural variants. Propose only useful, evidence-backed labels, with at most five tags and three labels in each other facet (14 total). Prefer `candidate_facet_labels` entries shaped as `{facet, value, definition}` so a genuinely new concept has a concise, distinguishing definition (500 characters max). Legacy `candidate_tags`, `candidate_audience`, `candidate_use_for` and `candidate_domain` fields remain valid for labels that do not need a definition. Leave uncertain labels out. Jev compares agent-proposed candidates with the closest project labels and chooses an existing label or the `new_label` option; it does not invent the proposed label text. The backend may add only a distinct new candidate that Jev ranks in that facet's configured top N (tags: up to five; other facets: up to three) and that clears its fit, new-label and confidence thresholds. An existing-label match is reused rather than duplicated. For the associated proposal, use Jev's selected canonical existing label for that facet. Use a new candidate as a project vocabulary label only when `added_to_project` is true; otherwise omit it from the proposal facets and leave its evidence-backed finding intact. Treat returned `jev_facet_resolutions` as the recorded decision; never claim a label was added unless `added_to_project` is true. Label promotion requires the server's local embedding model and Jev configuration; without them the candidate remains only a review signal. Label decisions only normalize facets; they do not gate evidence assessment, proposal submission or acceptance.
|
|
|
|
If available, Jev also assesses relevance and lens in the same advisory call. Use the saved scores to focus human review; they are not evidence assessment or an instruction to silently rewrite accepted knowledge. When a reviewer decides, record the feedback and apply reusable project guidance only when the reviewer explicitly opts in. Preserve the service's exact source/extraction/location bindings. Map each factual field to evidence; assess semantic support separately. Record clear exclusions and genuinely skipped/deferred sections. Optimize for useful coverage rather than maximum note count.
|
|
|
|
Submit bounded bundles under the affected project/shared policies. Repair actionable findings without weakening evidence, changing scope to evade policy or assuming owner authority. Resume uncertain calls through status lookup. Report accepted node IDs and receipts, pending/rejected work, archive/extraction limits and actual coverage. A successful upload or schema check is not a completed ingestion.
|
|
|
|
Every `register_source` operation affects both the source's `ingested_for` project policy and the shared-source policy. Set proposal scope `include_shared=true` as well as the project scope, even when the source is relevant only to one project. Confirm the credential authorizes both. If shared scope or authorization is unavailable, stop and report the missing policy/capability; do not submit a project-only bundle that will fail scope validation.
|
|
|
|
For a captured source, use the returned extraction status and block metadata to plan review. Read source text through bounded `source_extraction_block` calls using the exact `extraction_id` and `block_id`; inspect relevant slices across all pages/sections, including PDFs that the host's file reader cannot open. The default response contains normalized text and its offset but omits the large original-source mapping; set `include_original_mapping=true` only when an original-text span is needed. Citation quote and locator offsets bind to the normalized extraction either way. Record partial or unavailable extraction coverage explicitly. Do not infer that a source was analyzed just because upload and extraction succeeded.
|
|
|
|
Before creating a new node, search the explicit project and shared scope using its proposed title plus distinctive terms, aliases and alternate phrasings. Search likely synonyms separately when title search is inconclusive, then inspect candidate nodes and claims with `get_node`. Search is lexical, so use more than one query. If a finding belongs to an existing node, preserve its stable `node_id` and existing supported content when proposing an evidence-backed update; do not mint another ID for the same concept. Keep separate nodes when populations, time periods, settings, mechanisms or conclusions materially differ. Similarity is a review cue, not proof of identity; if the distinction remains uncertain, create a non-blocking review flag and do not create a second node until resolved. The current backend only flags exact entity name/alias matches; it does not detect duplicate notes, cases, concepts or datapoints for you.
|
|
|
|
Manage context and provider budget across the whole packet. Process one source at a time, use extraction metadata to target relevant slices rather than fetching every character, and retain concise section notes plus exact evidence excerpts instead of full source text. Checkpoint coverage after each source. If the provider/tool budget is nearly exhausted, checkpoint the current file and report the remaining files/sections for continuation; do not reduce cost by dropping a research lens or deferring cultural-historical or financial-market material.
|
|
|
|
Treat documents as capture/provenance units and proposals as semantic learning units. Group a proposal around one project question or one tightly related set of findings, and combine evidence from multiple documents when it supports the same learning. Do not make one proposal per file or one oversized bundle for the entire packet. Keep unrelated learnings separate so each can be evaluated, retried, reviewed and accepted independently. Include each new source candidate in the first proposal that cites it; after its source revision is accepted and readable, later learning proposals cite that revision without registering it again.
|