GEWU/docs/acceptance/four-file-validation-results.md

# Four-file ingestion benchmark: results and remaining gates

The claim-level and question-level diagnostic is recorded in [four-file-failure-analysis.md](four-file-failure-analysis.md). It clarifies that the 92.98% support result contains four partial-support groups, and separates the 29/48 pre-citation answer content grade from the lower score after malformed citation spans. The implementation follow-up produced evidence-bound candidate claim repairs, added the numeric-claim guard, and attempted a same-partition answer rerun. The corrected citation artifact is a separate verified copy and has not yet received a fresh semantic grade.

Date: 15 October 2026  
Current outcome: **all four sources reviewed; 22 nodes accepted, 16 ingestion drafts policy-held, and three correction proposals staged for owner review. The agent does not qualify yet.** The latest run checkpoint is revision 38. Current findings and scores appear in the final section; earlier sections retain the interrupted baseline and repair history.

## Run setup

The test used the four originals in /tmp/gewu-dome-pilot/project-v023/Inbox/ and a new isolated GEWU store at /tmp/gewu-dome-pilot/fresh-v024/. The project context was dome-performance. The agent pack identified itself as gewu-research v0.20.11. Antigravity CLI ran Gemini 3.8 Flash Medium.

The original files were left unchanged. Their SHA-256 hashes matched the four staged artifact digests. Temporary MCP entries and local service processes created for this run were removed at the end; the isolated test store and receipts remain for inspection.

## Initial run results

| Stage | Result | Evidence |
| --- | --- | --- |
| Isolated setup | Pass | Fresh canonical repository started at 5c971d5f44ce3d06556362b45a5985f5f812033d. |
| Capture | Pass | Four artifacts archived; each staged digest matches its workspace original. |
| Extraction | Pass | All four extractions report complete: Markdown 4 blocks; psychology paper 7 page blocks; transcript 1 block; SATIE paper 6 page blocks. |
| Extraction spot checks | Pass | The Markdown extraction contains all three cycles, synthesis, notes and export metadata. The psychology PDF contains the immersion/presence discussion. The transcript extraction contains the sampled historical discussion. The SATIE page 4 extraction preserves the three Table 1 rows, values (337/288, 4288/1984, 2964/1520) and machine configuration; page 5 retains the reported 128-speaker limitation. |
| Durable coverage checkpoint | Partial | Run run-dome-performance-20261015-01 reached revision 1, stage capture, with all four files selected and all four source revisions recorded as extracted. coverage.included is empty; no reviewed sections are recorded. |
| Digestion/proposals | Not completed | No proposal records, accepted knowledge commits, receipts or review flags exist in the fresh store. Canonical head remained the initialization commit. |
| Retrieval and answer benchmark | Not run | No accepted nodes were available to query. |
| Retry, mutation, interruption and scope cases | Not run | These cases follow a completed baseline ingestion. |

## Agent behavior

Antigravity print mode initially denied MCP access. A second attempt also failed because this account does not support the CLI's automatic permission mode. A third attempt used the CLI permission-bypass flag within this disposable workspace and service; it created the run and captured/extracted all four files. After 422.56 seconds, the run remained active at the capture checkpoint and had produced no proposal or later checkpoint. The operator stopped it. Its final result reports ERROR/interrupted, with 269,126 input tokens, 25,712 output tokens and 4,114,819 cache-read tokens. The checkpoint alone did not reveal the ongoing source reads.

This is an interrupted, incomplete baseline: upload and extraction work, while the agent did not finish review, evidence-backed proposal submission, acceptance or reporting. It does not establish that the service or model was stuck. The run is incomplete, not a successful ingestion. No prior pilot outcomes were used to score this run.

## Gate summary

| Gate | Status |
| --- | --- |
| Four originals archived with matching digests | PASS |
| Extraction reports complete for all files | PASS |
| Sampled extraction facts and table rows retained | PASS |
| Every relevant section has a terminal coverage disposition | INCOMPLETE; run interrupted |
| Essential findings represented in accepted knowledge | NOT RUN |
| Citation validity for accepted claims | NOT RUN |
| Retrieval Recall@5 and answer quality | NOT RUN |
| Zero false completion claims in final agent report | NOT ASSESSABLE; agent returned no final report |

The proposed 95% claim-support, 90% essential-finding recall and retrieval/answer thresholds cannot be scored without proposals and accepted records.

## Reproduction and remaining work

The isolated service data, upload checkpoint state, service run record and CLI logs are under /tmp/gewu-dome-pilot/fresh-v024/. The source originals and agent workspace remain under /tmp/gewu-dome-pilot/project-v023/. Use the service run/status operation for the existing run before resuming; do not create a replacement run to conceal its incomplete state. If execution starts fresh, allocate another empty data root and record that it is a new run.

To complete the benchmark, supply the missing operator principal ID and use tool-call progress as well as checkpoints when monitoring the host. Resume from the exact run and source identities, checkpoint section coverage, submit bounded evidence-backed proposals, then run the frozen question set and remaining behavioral cases. Do not qualify the agent on capture/extraction alone.


## Follow-up diagnosis

The saved Antigravity SQLite conversation contains 61 tool executions, including 14 source-extraction reads. Its last action was another successful read of a new transcript slice. The host was making source-reading progress when the operator interrupted it; the original claim that it was stalled was premature.

The benchmark prompt omitted the authenticated principal ID required by the installed pack. Before reading evidence, the agent searched unrelated host configuration, environment, benchmark files and service storage to discover it, including a disposable token file. This violated the pack's intended setup boundary and consumed much of the run. The disposable credential was rotated before retrying, preserving its agent-pilot principal and grants.

The source pack now checks for the operator-supplied principal before source transfer, stops if it is absent, forbids credential/configuration discovery, and describes a bounded resume/checkpoint sequence. An inconsistent installer-state instruction now explicitly permits reading only the required compact run_pins object. The retry handoff supplies principal_id=agent-pilot and preserves the original run and pack pins. The retry records tool names, slice offsets, HTTP status and timing without logging credentials or source bodies.

## Repairs and verification

Two service defects were reproduced and fixed: source_extraction_block now returns its extraction normalization_profile, and invalid run_create/run_checkpoint requests now return status=invalid with actionable field paths without advancing state. This lets a caller construct evidence without guessing and correct a checkpoint rather than receive a generic tool failure. The contribution guide now documents coverage shapes and the proper proposals[].receipt field, rather than a nonexistent last_receipt field.

The checked-in scripts/prepare_ingest_handoff.py requires a non-secret principal, project and selected connector file IDs, supplies a tool budget, and generates an explicit kb-ingest handoff. It supports multiple sources in one learning unit and existing-run resumption. Missing or token-shaped identities are rejected before any handoff is produced. Installer test fixtures were updated to contain the run-pin metadata now required by the installer.

The disposable server fixture also used a static validation context, so accepted source revisions were invisible to subsequent evaluations. It now uses the production LocalContextProvider to load canonical source/node/extraction state. This fixture correction is separate from the two service fixes.

| Native-host attempt | Outcome |
| --- | --- |
| Antigravity / Gemini Flash Medium, explicit principal supplied | Setup no longer searched credentials, but the model probed preparation with incomplete payloads and later read old benchmark logs/invoked a shell. The operator stopped this attempt after 223.81 seconds. It is not a passing workflow. |
| Antigravity / Claude Sonnet Medium | Followed setup, then failed with RESOURCE_EXHAUSTED / individual quota reached after 19.29 seconds. Acceptance was not tested on this model. |
| Codex CLI, isolated MCP connections and read-only shell sandbox | Reproduced the missing normalization profile and opaque checkpoint errors; after the repairs, prepared/evaluated/submitted one supported SATIE unit and checkpointed actual coverage. This is a separate host smoke test, not Antigravity qualification. |

The Codex smoke test preserved the original run ID and v0.20.11 pack pins; the v0.20.14 source-pack safeguards and compact contract were supplied as operator guidance without replacing pinned files. This does not qualify the entire new pack on any host.

### Accepted learning unit

- Node: satie-sound-object-spatialization-pipeline — SATIE mono-source spatialization pipeline.
- Claim: SATIE feeds a sound source's mono output into a spatializer that sends multichannel audio to the output.
- Evidence: 2017bouillot.pdf, page 2, normalized span [1033,1162).
- Proposal/receipt: proposal-dome-satie-pipeline-20261015-01, revision 1.
- Acceptance commit: c0d6ad37ea3b3d98d2abf42a1e604387bc69133a.
- Agent coverage checkpoint: revision 3, with all other sources and unexamined SATIE sections explicitly pending.
- Operator verification checkpoint: revision 6, including the typed accepted-proposal receipt; original run pack snapshot preserved.

### Verification evidence

| Check | Result |
| --- | --- |
| Complete preparation → eligible evaluation → acceptance | PASS; exact prepared proposal JSON forwarded unchanged. |
| Canonical node read and exact archived citation | PASS. |
| SATIE search returns the accepted node in the top five | PASS; single-query smoke test, not corpus Recall@5. |
| Exact proposal replay | PASS; identical immutable receipt, no extra canonical commit. |
| Altered quote with otherwise valid evidence bindings | PASS; evaluation rejects with EVIDENCE_MISMATCH alone. Invalid work was not submitted. |
| Unauthorized project read | PASS; rejected without disclosing the node. |
| Invalid checkpoint correction | PASS; returned field issue identified last_receipt; corrected checkpoint persisted. |
| Focused regression suite | PASS; 21 tests covering MCP contributions, durable runs, handoff validation, installer behavior and generated contract. |
| Changed-file Ruff, focused mypy and pack manifest verification | PASS. |

Raw source content, traces and machine-readable verification results remain private under /tmp/gewu-dome-pilot/fresh-v024/. The checked-in report includes no bearer credentials or full source bodies.

## Remaining qualification

The complete four-file semantic review, independently frozen 40-question set, recall/support scoring and fresh repetitions are still outstanding. One accepted learning unit and its regression checks do not establish full corpus or native Antigravity acceptance. The source-pack host acceptance markers remain unqualified. Resume the existing run using its current status and intact source identities; use bounded learning units and do not infer complete coverage from the artifact inventory.

## Continued implementation: frozen reference and long-extraction repair

The evaluator reference is frozen at SHA-256 `82e5a538f71c0e81b699cb5431e8bdc211227a6dfe662fb1620a4fd45ecf2a08`: 40 source-bound findings and 40 questions, split into 24 tuning and 16 reserved questions. All 40 exact normalized passages and page/block locators validate. All four originals match their backup copies. Private reference material is outside the ingestion workspace at `/tmp/gewu-dome-pilot/evaluation-v025/`; ingestion prompts contain no expected answers. The primary assistant prepared the reference separately from the ingestion CLI; no human reference review has been performed. The earlier SATIE pipeline smoke-test finding was already exposed before this freeze, so this continuation is not a pristine blind baseline. A full-document omission audit remains separate from passage validation.

A reproduction using the actual Markdown extraction found that its 4,007,569-byte canonical JSON exceeded the context provider's 1 MiB ordinary-record limit. Accepting that source would have made subsequent context reads fail. Extraction records now have a separate 16 MiB bound; ordinary registry/node limits remain unchanged. The actual Markdown reproduction now loads successfully. A regression test also confirms that the extraction-specific bound still rejects oversized records. The focused suite passes 26 tests; changed-file Ruff and registry mypy checks pass.

`scripts/run_retrieval_benchmark.py` captures scoped top-five MCP results, question-file hashes, response metadata and timings into an exclusively created JSONL file. It requires independent relevance judgments and deliberately assigns no semantic score from keyword overlap. Example, using the disposable authenticated loopback proxy:

```bash
.venv/bin/python scripts/run_retrieval_benchmark.py \
  --questions /tmp/gewu-dome-pilot/evaluation-v025/questions-blind.json \
  --output /tmp/gewu-dome-pilot/evaluation-v025/retrieval-results.jsonl \
  --project dome-performance --mcp-url http://127.0.0.1:8767/mcp/
```

The first resumed Codex source review evaluated SATIE's architecture finding as `needs-review` under the existing policy without submitting it. Performance conditions, control lifecycle and mapper/swarm techniques received accepted receipts. The completed continuation follows.

## Completed source review and tuning evaluation

All 411,468 normalized characters were delivered through successful service reads, with no gaps in any block's read intervals. This verifies reading coverage, not comprehension by itself. Original/archive integrity passes 4/4. Full text normalization matches the Markdown and transcript originals; a separate Poppler run matches every PDF page. Visual spot checks confirm the SATIE table and psychology paper's title/abstract. The PDF rerun shares the parser and is not an independent visual audit of every page.

| Source | Normalized characters read | Accepted nodes | Policy-held ingestion drafts |
| --- | ---: | ---: | ---: |
| SATIE paper | 42,310 / 42,310 | 7, including the earlier pipeline unit | 2 |
| Psychology review | 50,212 / 50,212 | 3 | 6 |
| Transcript | 63,398 / 63,398 | 5 | 5 |
| Research export | 255,548 / 255,548 | 7 | 3 |
| Total | 411,468 / 411,468 | 22 | 16 |

The canonical store contains four source records and 22 nodes: 50 main claims, seven separately assessed limitations and no assumptions. All **58/58 citation bindings** match their archived source revision, extraction, page/block and normalized span. Every submitted CLI proposal forwarded its prepared JSON unchanged. The accepted canonical head is `55fbebd115cad0d7d54da6d7e5063fc0f3ce839c`.

The four primary source-review turns used 65, 63, 65 and 85 MCP calls respectively, within their supplied budgets. CLI usage, tool counts and query timings are retained privately. Exact CLI model identity was not independently pinned; authored node model labels do not establish that identity. Monetary cost was not measured.

### Support defects and review-ready corrections

The separate evaluator reviewed all 57 assertion groups against their own attached evidence. Four groups have incomplete support: two psychology claims omit passages needed for some clauses, a SATIE claim turns inability to perform a test into an established causal explanation, and a transcript claim's year is absent from its attached excerpt. The other 53 groups pass this review. This is a strict group-level result, **53/57**, not an atomically weighted precision score or human audit.

An attempted same-ID correction was rejected with `NODE_ID_EXISTS`. The service's `upsert_node` creates stable IDs; it cannot update an existing record. Three append-only replacement proposals, each retaining an explicit `supersedes` edge, are now durably staged at revision 1:

| Correction proposal | Change | State |
| --- | --- | --- |
| `proposal-cmmm2015-research-control-corrected-20261015` | Attach complete claim-owned page-2 evidence for both research-control claims. | Evaluated, needs-review. |
| `proposal-satie-performance-conditions-corrected-20261015` | Preserve the authors' inability-to-test wording without asserting its cause. | Evaluated, needs-review. |
| `proposal-transcript1-attention-corrected-20261015` | Add the source passage establishing the production date. | Evaluated, needs-review. |

Their evidence bindings validate, but **none is accepted**. Supersession requires an owner decision under the current policy; the disposable harness has no owner-decision store configured and created no owner challenges. Original accepted records remain in the scored store. The 16 ordinary ingestion holds are evaluated drafts retained in traces/checkpoints, not submitted proposals. Together, the private review packet contains 19 review items, clearly distinguishing these states.

Private review artifacts: `/tmp/gewu-dome-pilot/evaluation-v025/held-review-queue.md`, `correction-review.md`, `correction-proposals.json` and `all-review-items.json`. Prepared base commits must be rechecked before a later decision or submission. No relationship was removed and no node type was changed to evade a policy hold.

### Retrieval and answer scores

Only the **24 tuning questions** were executed. The **16 reserved questions remain unused**. All retrieval variants use the same canonical head. The hybrid candidate uses the existing production construction with the configured local embedding model and a separate derived index; this is a fixture change, not a new search algorithm.

| Full-question retrieval configuration | Queries returning any rows | Conditional evidence Recall@5 |
| --- | ---: | ---: |
| Keyword, 8,000-codepoint budget, first page | 0/24 | 0% |
| Hybrid, 8,000-codepoint budget, first page | 1/24 | 0% |
| Hybrid, 50,000-codepoint budget, first page | 24/24 | 48.6% |
| Hybrid, 50,000-codepoint budget, advancing pagination | 24/24 | **88.9%** |

Recall is separately adjudicated over **12 questions with usable accepted evidence**, including relevant archived passages attached to accepted claims. Eight other content questions have partial or missing accepted material; they are reported separately, not removed from the answer score. Two negative and two governing-rule questions do not require a positive retrieval hit. Some usable passages are collateral to a node's main finding; retrieving them does not establish that the corresponding finding was digested into an accepted assertion. Relevance judgments were made by the primary assistant against frozen expected points, without human review.

The paginated candidate delivers five results for every tuning query without an unchanged-cursor loop. Its observed median query duration was 0.329 seconds over 24 questions, including pagination. This is a small warm-run observation, not a tail-latency or end-to-end model-performance claim.

The response budget counts Unicode codepoints of canonical JSON, not model tokens. Large evidence-bearing records were omitted under the smaller budget, with `BUDGET_TRUNCATED`. An omitted result is not evidence of source absence. Keyword-only full-question searches separately fail to match these long natural-language questions. Both limitations are retained in the raw results.

| Blind answer configuration | Content points before citation penalty | Raw score with citation penalty | Exact answer citations |
| --- | ---: | ---: | ---: |
| Hybrid, 8,000-codepoint budget | 15/48 | 15/48 (31.3%) | 5/5 |
| Hybrid, 50,000-codepoint budget and bounded follow-ups | 29/48 | **21/48 (43.8%)** | **1/16** |

The larger-budget model shortened 15 quote strings while retaining the original wider locators. Its answer content improved, but the raw output fails the 100% citation-validity gate. The evaluator gives at most one point to an otherwise correct answer with invalid citation bindings. Both executed negative questions avoid invented guarantees or ROI; the other two negative questions are reserved. Governing provenance/scope answers correctly distinguish source content from project authority.

`scripts/verify_answer_citations.py` now rejects quote/span mismatches and evidence borrowed from a different claim, source, page or range. With explicit `--repair-quotes`, it copies the exact requested span from independently verified, claim-owned evidence into a separate artifact. It changes no answer prose, IDs or locators and does not assess semantic support. Three regression tests pass. The repaired artifact has **16/16 exact citation bindings** and 29/48 answer points, still below the 85% target. This evaluator repair does not turn the original model run into a passing result or qualify native output enforcement.

### Improvement implementation started

The improvement plan is now tracked in the [validation plan](four-file-validation-plan.md). The citation verifier has been tightened to fail closed on malformed answer records, duplicate answer IDs, missing citation arrays, uncited answers without an explicit evidence-gap/governing-rule classification, and citations missing their required identity or locator fields. Evidence-gap answers may omit accepted-node citations; protocol-based answers must identify their basis so they are not silently scored as source-supported findings. The verifier still checks exact claim ownership and source spans and still makes no semantic entailment claim.

`scripts/build_benchmark_manifest.py` now creates a run manifest containing the backend commit, clean-worktree assertion, complete agent-pack tree digest, four source hashes, frozen-reference hash, extraction IDs/parser versions/config hashes, concrete model ID, retrieval mode and response budget. It refuses dirty backend checkouts, model aliases such as `default`/`latest`, missing inputs and output overwrite. This prevents a run from being labeled reproducible when the checkout or model is not pinned.

Validation for this increment: **75 focused tests pass**, including seven citation-verifier cases and two manifest cases; the separate agent-contract test passes against the current catalog. Ruff and mypy pass for the new tooling. Running the stricter citation schema on the saved benchmark output flags Q37 and Q39 because they contain citation-free governing-rule prose without an explicit `answer_basis`; the raw output remains unchanged. The current GEWU backend exposes evidence to an external answering agent and has no answer-generation endpoint, so this is a reusable pre-publication verifier rather than integrated backend middleware. Native host integration remains open.

The 16 held drafts and three correction proposals still require an authorized owner to decide accept/revise/exclude. The existing fixture has no owner-decision store; no decisions were fabricated. The manifest builder is ready, but no qualification manifest was created from the dirty shared checkout, and no new model run was started because its concrete model ID is not available in the saved run metadata. Reserved questions and fresh repetitions remain unrun until tuning gates pass in a pinned candidate environment.

Example commands for the selected retrieval candidate and separate citation repair:

```bash
.venv/bin/python scripts/run_retrieval_benchmark.py \
  --questions /tmp/gewu-dome-pilot/evaluation-v025/questions-tuning-blind.json \
  --output /tmp/gewu-dome-pilot/evaluation-v025/new-paged-results.jsonl \
  --project dome-performance --mcp-url http://127.0.0.1:8767/mcp/ \
  --budget-tokens 50000 --collect-top-five

.venv/bin/python scripts/verify_answer_citations.py \
  --answers /tmp/gewu-dome-pilot/evaluation-v025/answers-50k.json \
  --records /tmp/gewu-dome-pilot/evaluation-v025/canonical-audit.json \
  --repair-quotes \
  --output /tmp/gewu-dome-pilot/evaluation-v025/new-verified-answers.json
```

Both tools preserve existing output files. The citation verifier requires trusted, independently validated canonical records as its reference input.

### Behavioral checks and qualification decision

| Case | Result |
| --- | --- |
| Exact accepted-proposal replay | PASS; same receipt, canonical head unchanged. |
| Corrupted quote or offset | PASS; `EVIDENCE_MISMATCH`, not submitted. |
| Wrong extraction or source revision | PASS; rejected, not submitted. |
| Valid quote with changed table number, 337 → 999 | **Backend semantic detection fails:** evaluation returns eligible. The mutant was not submitted. This proves that structural validation trusts the supplied support assessment; no incorrect mutant entered canonical knowledge. Separate agent detection of this injected draft was not tested. |
| Unauthorized node read and scoped search | PASS; tool errors, with no tested source-content fragments returned. |
| Resume across source turns | PASS; original run, source identities, pack snapshot and receipts preserved through revision 38. |
| Startup without service tools | Safe stop; one psychology attempt stopped before transfer. Retry retained as a separate trace. |
| Bounded query reporting | PASS in tested turns; omitted evidence is reported as a retrieval limit and no full-packet success is claimed. |
| Full semantic re-ingestion, changed-file derivative, adversarial source instructions, live delayed-index case | NOT RUN end to end. Existing component tests cover several upload/restart/index behaviors; those are not substitutes for these live cases. |
| Reserved questions and three fresh repetitions | NOT RUN; candidate has not cleared support, answer/citation and owner-review gates. |

Accepted-finding recall cannot meet the 90% target in the current state: at least nine of the 32 essential reference items are policy-held across the psychology review and transcript alone. Even crediting every other essential item gives an upper bound of 23/32 (71.9%). This is a bound, not a completed exact recall score; metadata/exclusion dispositions and the remaining reference items still require full adjudication. Conceptual, historical and market holds must not be misreported as successful accepted coverage.

The earlier continuation check passed 69 focused tests but its agent-contract test failed during catalog drift. On 16 October the current tree passes **75 focused tests**, and the separate agent-contract test passes after the catalog update. The new manifest and citation tools also pass Ruff/mypy and whitespace checks. The workspace remains dirty with unrelated concurrent browser/API edits, so it still cannot produce a qualification manifest; the prior CLI run also lacks an exact model pin. These results do not constitute three clean comparable runs.

The run remains active at revision 38, stage `semantic-support-corrections-staged-needs-owner-review`, with 41 proposal references and its original v0.20.11 pack snapshot. The v0.20.14 source pack now documents claim-owned clause support, immutable node IDs/supersession, budget-aware querying and exact answer quotes. These source-pack improvements are not yet natively host-qualified.

The next gates are owner review of the 16 ingestion drafts and three correction bundles, then an isolated backend/model-pinned tuning run using the manifest builder. Enforce citation verification in the answering host’s output path before using the reserved questions. Preserve the 16 reserved questions until tuning clears the agreed gates; then perform the remaining adversarial cases and three fresh repetitions. The review packet is at `/tmp/gewu-dome-pilot/evaluation-v025/all-review-items.json` and its readable summaries are `held-review-queue.md` and `correction-review.md`. Machine-readable scores, judgments, raw traces and the run checkpoint remain in the same private evaluation directory and `/tmp/gewu-dome-pilot/fresh-v024/`.

## 16 October benchmark retest

The frozen reference still validates at SHA-256 `82e5a538f71c0e81b699cb5431e8bdc211227a6dfe662fb1620a4fd45ecf2a08`: 40/40 source passages and locators, 40 questions split into 24 tuning and 16 reserved, and 4/4 original files byte-match their backups. Full extraction comparison passes for all four inputs. The two PDFs match local Poppler page text; this remains a parser cross-check, not a full visual audit.

The canonical audit was regenerated and its JSON hash matches the pre-retest audit exactly. Canonical head remains `55fbebd115cad0d7d54da6d7e5063fc0f3ce839c`, with 22 accepted nodes, 57 assertion groups, 58 exact citation bindings, and all four archived artifacts matching their input digests.

## 16 October continuation: candidate support audit and Antigravity CLI

The user clarified that Antigravity model limits were unused. I checked the CLI directly: Antigravity 1.3.6 lists available models and completes model calls. The semantic review was rerun with pinned models `gemini-3.8-flash-medium` and `gemini-4-argon-medium` in sandboxed plan mode. Thus the prior quota error was not a general Antigravity limit. The four-question answer call still timed out, and the Q13 retry ended in a provider API EOF; neither produced a comparable benchmark answer set.

The four traced support defects now have an evidence-bound candidate treatment. The CMMM2015 assertions were split into atomic claims with exact page spans, the transcript quote now includes the date and Black Holes context, and the SATIE wording was reviewed as written in the source. The candidate structural audit passes 8/8 evidence bindings. Two independent semantic judges mark all eight claims in the CMMM/transcript candidate packet supported, with confidence from 0.90 to 0.98. The earlier two-judge audit marks the SATIE claim supported at 0.98 from each judge. These are advisory findings only. The original canonical support score remains 53/57 (92.98%) because none of the proposed corrections has been accepted; no owner decision was recorded and the canonical head remains `55fbebd115cad0d7d54da6d7e5063fc0f3ce839c`.

Candidate and review artifacts are retained under `/tmp/gewu-dome-pilot/evaluation-v025/`: `claim-repair-candidates-v052.json`, `claim-repair-review-items-v052.json`, `claim-repair-structural-review-v052.json`, `claim-repair-v052-binding-audit.json`, and `semantic-review-claim-repair-v052/`. The semantic runner records the CLI version, binary digest, model pins, batch count, and raw/parsed judge outputs. The first v052 judge call returned malformed structured output; the runner preserved it, retried as attempt 2, and completed the review with both judges.

The per-question diagnosis of 19 lost points is in [four-file-failure-analysis.md](four-file-failure-analysis.md). It maps all missing expected points to questions and distinguishes content gaps from malformed citation penalties. The improved source-selection checklist is saved in `/tmp/gewu-dome-pilot/evaluation-v025/answer-tuning-guidance-v043.md`. The same 24 tuning questions have **not** yet been rerun as a comparable model answer benchmark: the AGY batch timed out, its single-question retry failed at the provider, and the Codex CLI MCP fallback could not obtain the connector under the configured read-only approval policy. Direct Luna smoke answers cover four evidence gaps and a citation-verified Q13 only; they are not a substitute for a full score.

The agent-level behavioral plan is also still incomplete. Existing isolated service checks preserve canonical head during upload-resume and same-digest re-ingest, archive an injected-instruction source without proposal submission, and observe five unsuccessful delayed-index polls. They do not prove agent resistance to prompt injection or agent interruption/resume. A true changed-file re-ingestion, successful delayed indexing, and a useful response to constrained-budget retrieval still need an end-to-end agent run. No reserved question or fresh repetition was run: the last comparable answer score is 29/48 (60.4%) and canonical support is 53/57, both below their 85% and 95% gates.

No canonical records or owner-protected decisions were changed during this continuation. The isolated evaluation service and temporary Antigravity connector have been stopped/removed; no test listener remains on ports 8766–8769.

A new paginated hybrid retrieval run executed all 24 tuning questions at a 50,000-codepoint response budget. It used the same canonical head and frozen independent relevance judgments. Results: **88.9% conditional Recall@5**, **11/12 hit rate** across 12 evidence-answerable positive questions, results returned for 24/24 queries, zero pagination blocks, 0.212 second observed median query time, and 0.248 second mean. These are warm-run retrieval timings only. The separately reported partial-ingestion questions remain Q17, Q25 and Q31; missing accepted material remains Q01, Q02, Q09, Q10 and Q18. Raw capture: `/tmp/gewu-dome-pilot/evaluation-v025/retrieval-paged-retest-20261016.jsonl`; scored output: `retrieval-paged-retest-20261016-scores.json`.

The first stricter citation pass rechecked both existing raw answer sets without changing them. At that point the 50k output had **15 `QUOTE_SPAN_MISMATCH` errors** and two citation-required errors (Q37 and Q39, with no classification for their governing-rule basis). The 8k output's five citations were exact, but it had the same two uncited answers. The prior quote-repaired 50k artifact had no span mismatches but still had those two uncited answers. The later protocol-catalog check is recorded below. No new model answers were generated because the prior run did not record a concrete model ID. The 16 reserved questions remain unused.

Current validation passes: **78 focused tests**, the separate agent-contract test, Ruff, mypy and whitespace checks. The disposable retrieval services were stopped after the run; ports 8766 and 8767 are closed. This retest confirms repeatability of the retrieval measurement on the existing canonical snapshot; it is not a clean-checkout qualification run or a new ingestion repetition. The shared checkout remains dirty, the benchmark manifest builder therefore refuses to certify it, and owner review is still outstanding.

## Owner review and answer-basis implementation follow-up

An editable owner decision sheet was generated at `/tmp/gewu-dome-pilot/evaluation-v025/owner-review-form-20261016.md`. It consolidates the 16 policy-held drafts and three correction proposals, shows claim text, complete attached evidence excerpts, source locators, lifecycle state and correction supersession edges, and provides separate decision choices for held drafts and replacements. It records no decisions and changes no service state. The underlying service harness has no owner-decision store, so approval must use the real authorized owner workflow after decisions are recorded.

The answer output contract now distinguishes accepted-evidence citations, explicit evidence gaps, and governing-rule answers. A governing-rule answer must supply `answer_basis.kind = "governing_rule"` and one or more `{rule_id, quote}` references. The verifier checks those IDs and exact quotes against `scripts/governing_answer_rules.json`, whose protocol source file digests and line text are checked before use. The query pipeline now instructs the agent to emit this structure; the GEWU source pack is v0.20.15. This validates rule identity and quotation, not whether the answer's interpretation follows from the rule. Pack changes have not been host-qualified.

The recheck against pinned local protocol files found the same answer defects under the new structured codes: 15 quote-span mismatches and two `ANSWER_BASIS_REQUIRED` failures in the 50k output; the 8k output has only the two basis failures; the old repaired 50k artifact also has those two basis failures. Raw answers remain unchanged. The updated contract report is `/tmp/gewu-dome-pilot/evaluation-v025/answer-contract-retest-20261016.json`.

After these additions, **83 focused tests pass**, the separate agent-contract test passes, and Ruff, mypy and the GEWU pack manifest verification pass. The answer host integration, owner decisions and pinned-model tuning rerun are the next gates. Reserved questions remain untouched.

## 16 October clean-checkout retrieval baseline

To separate search behavior from the dirty shared development checkout, the 24-question tuning packet was rerun from a detached clean worktree at backend commit `a3156f4e08fac5b34739f82b0c72b28dbcf11349`, with the agent pack at its recorded submodule commit (`gewu-research` v0.20.13). The accepted four-file store was copied to `/tmp/gewu-dome-pilot/retrieval-clean-v026-data`; all retrieval requests used that isolated copy and the hybrid SQLite FTS5/vector index. The run manifest pins the four original file digests, frozen reference and extraction parser identities, local embedding model revision/weights hash, hybrid mode and 50,000-codepoint budget.

The clean-checkout run reproduced the prior retrieval score: **88.9% conditional Recall@5**, **11/12 hit rate** over the answerable positive questions, 24/24 queries completed, and zero pagination blocks. Median observed time was 0.299 seconds (mean 0.288 seconds), on canonical revision `55fbebd115cad0d7d54da6d7e5063fc0f3ce839c`. The difference from the prior 0.212-second median is consistent with a separate cold process/index warm-up and this small sample; it is not evidence of a performance regression. The baseline confirms that the prior retrieval result is reproducible from the clean backend revision and accepted data snapshot. It does not qualify ingestion, semantic answer generation, owner decisions, reserved questions or the uncommitted improvements in the shared checkout.

Private reproducibility artifacts are `/tmp/gewu-dome-pilot/evaluation-v025/retrieval-clean-v026-manifest.json`, `retrieval-clean-v026.jsonl`, and `retrieval-clean-v026-scores.json`. The clean worktree reports no modifications, and local services on ports 8766/8767 were stopped after capture.

## Automated implementation pass

The deterministic benchmark review and bundle runner are implemented in `scripts/review_benchmark_candidates.py` and `scripts/run_four_file_automation.py`. They verify source hashes, audit claim-owned quote/locator/extraction bindings, score paginated retrieval, validate answer citations and governing-rule basis, and write append-only reports in a private output directory. They never mutate canonical knowledge or owner decisions. `scripts/governing_answer_map.json` pins the rubric classification for Q37 and Q39; exact quote repair and rule-basis attachment are written only to separate verified copies.

The first full bundle is `/tmp/gewu-dome-pilot/evaluation-v025/automated-bundle-v035/`. All four original source digests match the pinned manifest. The deterministic audit checked 42 held/correction claims and 45 evidence bindings; all bindings match their extraction text, source revision and locator. All 19 items remain `defer` because their current lifecycle requires protected owner review. The 3 correction proposals preserve their supersession edges in the review ledger.

The 8k answer artifact required two rubric-bound governing-rule references. The 50k artifact required 15 exact quote repairs plus the same two rule references. The verified copies pass the structural citation contract; raw answer files remain unchanged. Semantic support is separate. The existing scored evaluation gives the 50k answers 29/48 points before citation penalties (60.4%), below the 85% target. Accepted assertion support is 53/57 fully supported groups (92.98%), below the 95% threshold. Retrieval passes at 88.9% Recall@5 and 11/12 hit rate. The combined gate report therefore marks qualification **not qualified** and correctly leaves reserved questions and three repetitions unrun.

The disposable live behavioral check in `/tmp/gewu-dome-pilot/behavioral-v029-results/behavioral-results.json` replayed an accepted proposal without changing the canonical head, confirmed the service rejects corrupted quote/offset/extraction/revision bindings, and confirmed unauthorized node reads and scoped searches return no source content. An exact-quote number distortion remained eligible, documenting the semantic-validation gap. The original and candidate stores retained the same head; local services were stopped. A further 35 application/API tests covering runs, uploads, scope, contribution/acceptance and hybrid search passed, but the listed live re-ingest, changed-file, source-instruction, delayed-index and budget/interruption scenarios are still missing.

The dual-model semantic-review runner is implemented with pinned CLI model IDs, schema validation, evidence-ID checks, private raw outputs and resumable batches. In the live Gemini pass, the first batch returned valid structured judgments for 13 claims (12 supported, one partially supported). The next batch did not return within five minutes and was interrupted; no two-model consensus exists, and that partial result is excluded from scoring. The partial run is preserved at `/tmp/gewu-dome-pilot/evaluation-v025/semantic-review-v036/`; it can resume with the same input and model pins using `scripts/run_semantic_review.py --resume`. The external answer host exposes no final-output integration point in this workspace, so citation enforcement currently runs as a required offline gate rather than host middleware. Machine recommendations, semantic consensus, and accepted knowledge remain separate states.

To resume the partial advisory pass in its isolated sandbox:

```bash
.venv/bin/python scripts/run_semantic_review.py \
  --input /tmp/gewu-dome-pilot/evaluation-v025/all-review-items.json \
  --output-dir /tmp/gewu-dome-pilot/evaluation-v025/semantic-review-v036 \
  --resume --dangerously-skip-permissions
```

The clean-run preparation script `scripts/prepare_four_file_run.py` now creates an exclusive private run directory, verifies a clean pinned source checkout, prepares a detached worktree, initializes submodules, copies and hashes the four inputs, and records the benchmark manifest and run state. Its smoke run at `/tmp/gewu-dome-pilot/prepare-smoke-v042/` produced a clean checkout (zero status entries), pinned all four input digests, and copied the 25 MB data snapshot. It fails closed on a dirty source checkout or an existing output root.

Validation for this implementation pass: **68 selected tests pass** (33 benchmark/evaluator/preparation tests and 35 application/API tests), Ruff and mypy pass for the benchmark tools, and `git diff --check` passes. The latest owner bundle is `/tmp/gewu-dome-pilot/evaluation-v025/automated-bundle-v040/`. This completes the reproducible deterministic gate automation and isolated-run preparation, not benchmark qualification. Remaining gates are completing the two-model semantic review and candidate revision pass, answer-host enforcement, outstanding live behavioral cases, authorized owner decisions, and then conditional reserved evaluation plus three fresh full repetitions.

## Candidate quality repair and regression pass (v043–v046)

The original claim-level judgment artifact (`assertion-support-judgments.json`) identifies four partial-support groups, not four unsupported groups. A candidate-only repair file is at `/tmp/gewu-dome-pilot/evaluation-v025/claim-repair-candidates-v043.json`. It splits the two compound CMMM2015 claims and adds study-specific page-2 spans, retains the transcript claim provisionally with its attribution, and narrows the SATIE statement from “the version prevented the test” to the source’s actual “unable to perform” wording. The companion binding audit `/tmp/gewu-dome-pilot/evaluation-v025/claim-repair-binding-audit-v043.json` checked six added evidence spans: all six exactly match their extraction block and page. This verifies binding only, not entailment. The four claims remain unchanged in canonical storage; all policy-protected items remain deferred.

The query skill and pipeline now require per-dimension/source retrieval checklists, targeted follow-up when search returns `BUDGET_TRUNCATED`, and a final check of each number, unit, comparison, attribution, negation and qualifier against its claim-owned evidence. The candidate instructions are in the updated GEWU pack query pipeline. The numeric regression in `tests/domain/test_validation.py` changes a source-backed “400 visitors” assertion to “999 visitors” while retaining the exact original quote. GEWU now returns `needs-review` with `NUMERIC_CLAIM_UNGROUNDED`, preventing automatic eligibility for a changed numeric literal absent from attached evidence. This is a conservative lexical check; it does not establish general semantic entailment. A read-only screen found six existing accepted claims with numeric literals not repeated in their attached quote strings; see `/tmp/gewu-dome-pilot/evaluation-v025/numeric-evidence-screen-v046.json`. These were not retroactively changed, but should be reviewed before further numeric claims relying on the guard are treated as qualified. The focused validation, semantic-runner and evaluator tests pass: **48 tests passed**.

An isolated copy at `/tmp/gewu-dome-pilot/behavioral-v044-data/` retained canonical head `55fbebd115cad0d7d54da6d7e5063fc0f3ce839c` before and after testing, with zero owner decisions. `/tmp/gewu-dome-pilot/behavioral-v044-results/behavioral-results.json` records: upload checkpoint offsets resumed from `[]` to `[0]`; same-digest upload produced the same artifact digest; instruction-like text was archived as source material but not submitted; and no canonical writes occurred. The existing SATIE node was not found in five search polls, which is inconclusive between indexing and search coverage. A 256-budget search returned a tool error. Full agent changed-file revision, instruction resistance, and run interruption/resume were not established; upload resume is not a substitute for agent-run resume.

The same 24 tuning question IDs were rerun in query workspace v045 with the updated query instructions. The host exhausted its 100-call budget before producing the required structured answer artifact; follow-up calls above the service’s 50,000-codepoint limit also failed. Raw trace is preserved at `/tmp/gewu-dome-pilot/evaluation-v025/answer-50k-v045-events.jsonl`. It is **not scored**; the last valid comparison remains the 29/48 content grade, and no score improvement is claimed. The 16 reserved questions and fresh repetitions were not read or run.

Two-model semantic review of the four accepted-claim defects and all 19 held proposals was restarted with smaller 18 KB packets and the model IDs `gemini-3.8-flash-medium` and `claude-sonnet-4-5-high`. Judge one completed six of ten held-proposal packets; the next batch failed with the host’s `RESOURCE_EXHAUSTED` quota response. The focused four-claim run also failed on quota before returning a judgment. Attempts with `claude-sonnet-5-5-high` and `claude-opus-5-5-high` hit the same individual quota limit. Logs and partial files remain in `/tmp/gewu-dome-pilot/evaluation-v025/semantic-review-v045/`, `semantic-review-accepted-claims-v046/` and `semantic-review-accepted-claims-v048/`. Partial judgments are not included in support scoring. There is no two-model consensus; semantic review is blocked until model quota is available.

The benchmark therefore remains **not qualified**. Current blockers are the semantic-review quota, candidate re-audit after independent judgments, obtaining a machine-readable answer rerun and auditable second scoring pass, full agent-level behavioral tests, and clearing the frozen support (95%) and answer-quality (85%) thresholds. Reserved questions and repetitions remain correctly gated off.

### Bounded answer rerun preparation (v050)

The failed v045 trace showed repeated identical searches and requests for 200,000 budget units despite the service's 50,000-codepoint response limit; it ended without a machine-readable answer artifact. `scripts/prepare_answer_batches.py` now prepares the frozen tuning partition only, in six batches of four questions. Each prompt caps the batch at ten total searches (one focused search per question and at most one targeted follow-up), forbids repeated searches and budgets above 40,000, requires every requested ID in the JSON response, and contains no reserved questions or evaluator answers. Its `validate` subcommand checks the response envelope, exact ID/order coverage and citation field shape before scoring. Three focused tests pass. The prepared private prompts and batch index are at `/tmp/gewu-dome-pilot/evaluation-v025/answer-batches-v049/`.

This is offline preparation only: no model calls were made, no new answers were scored, and the prior 29/48 estimate remains the last valid content score. The six private prompts and batch index are at `/tmp/gewu-dome-pilot/evaluation-v025/answer-batches-v050/`. The next run should execute these batches sequentially, validate each raw answer artifact, then combine all 24 answers and run exact citation verification and the frozen evaluator before deciding whether any benchmark gate changed.

#### Follow-up execution attempt (2026-10-16)

The prepared batches could not be executed in this session. Antigravity returned `RESOURCE_EXHAUSTED` and reported the account quota reset in 34 hours; the separate Claude CLI returned `Not logged in`. No answer batch was run and no score was changed. A resumed dual-model semantic review was likewise unavailable because its pinned Antigravity models share the exhausted host quota; prior partial judgments remain excluded. The only locally available model interface was not authenticated, and no credentials were read or changed.

Deterministic component verification passed: **72 tests** across upload handling, run checkpoint/resume, bounded search, MCP contribution, numeric claim validation, answer citation verification, and batch preparation. These tests run against isolated temporary stores. They do not establish live agent resistance to source instructions, full changed-file re-ingestion, delayed-index recovery, or agent interruption/resume. The previous disposable behavioral trace still has unchanged canonical state, but its instruction-resistance and full agent-resume cases remain untested. No owner-protected decisions were made.

Qualification remains **not qualified**: support is still 53/57 fully supported groups (92.98%, threshold 95%); the last scored answer set is still 29/48 (60.4%, threshold 85%); semantic consensus, the new answer rerun, and full agent behavioral cases are incomplete. The 16 reserved questions and three fresh repetitions remain unrun. Resume after model quota availability by running the six `answer-batches-v050` prompts sequentially, validate each output before combining, then complete dual-model review and live disposable behavioral cases. Only proceed to reserved questions/repetitions after the frozen qualification gates pass.