GEWU/docs/acceptance/four-file-failure-analysis.md

Four-file validation: failure analysis

Date: 16 October 2026
Inputs: frozen reference-v1.json, assertion-support-judgments.json, answer-scores.json, original answers-50k.json, and the separately repaired automated-bundle-v040/answers-50k-verified.json.
Scope: diagnostic only. No canonical data, owner decisions, raw answers, frozen rubric, or reserved questions were changed.

Read this before acting

The saved support result is 53/57 fully supported (92.98%), with four partial-support claim groups. It does not list any fully unsupported group. The 57 groups comprise 50 main assertions and seven limitations, as recorded in the judgment artifact. These grades were made by one assistant evaluator, without human review or a second independent judge. The machine evidence-binding audit passed all structural quote/revision/locator checks; that audit does not establish that the cited passage entails the assertion.

The answer score for the 50k run is 29/48 content points before citation penalties (60.4%). The original artifact also had 15 invalid quote spans among 16 source citations, which reduced its recorded post-penalty score to 21/48. In bundle v040 those quote strings were restored from the cited accepted evidence in a separate verified copy; the raw answer remains unchanged. This is a structural repair only. The saved 29/48 content grade was not rerun against the verified copy, so it remains the last available content estimate, not a validated final score. A new grade should follow any candidate changes and should be independently checked.

Claim-support failures

Node / claim	Finding	Why the attached evidence falls short	Recommended candidate repair
cmmm2015-research-control / cmmm-research-control-0	Partial support	Claim combines control of the surrounding environment, multiple sensory modalities, separate manipulation of inputs, a vection study, and faster event processing. Its attached excerpt clearly supports the first two; the separately manipulated inputs, Keshavarz vection study, and faster processing passage occur later and are not included in this claim’s evidence span.	Split this into atomic claims. Keep the currently supported clauses with their quote; attach separate exact page spans to the manipulation/vection and processing claims. Do not infer these clauses from a neighboring claim’s evidence.
cmmm2015-research-control / cmmm-research-control-1	Partial support	Claim combines Kozlov/Johansen helping behavior, Glotzbach avoidance, and Renaud gaze behavior. The attached later passage includes the latter descriptions but not the earlier study setup and full helping-behavior outcome.	Split by study; attach each study’s full result passage and preserve the node’s explicit limit that these are VR research applications, not dome-engagement evidence.
transcript1-attention / attention-0	Partial support	The transcript supports the attributed positions, but the attached quote ends before the remainder of Lucas’s argument. The saved grader flags partial support; the exact missing clause should be checked against the extraction before editing.	Narrow the claim to what the current excerpt contains or extend its evidence span to include the complete exchange. Keep the attribution and avoid presenting the transcript’s sickness explanation as an experimentally validated universal threshold.
satie-performance-conditions / satie-performance-conditions-2	Partial support / overstatement	Source says the authors were unable to perform the 128-speaker test with the current SuperCollider version. It does not say the version prevented the test.	Replace “the current version … prevented” with “the authors report they were unable to perform the 128-speaker test using the then-current version”; preserve the stated version comparison as a separate clause.

The more specific pre-repair operator notes identify missing page-2 evidence for both CMMM claims and the unsupported causal implication in the SATIE wording. The attention claim’s saved judgment is partial support, but the operator note does not explain that particular claim. Resolve it from the full extraction before changing it; do not treat the guessed missing clause as established.

Missing answer points

The table is the frozen primary evaluator’s breakdown for the 50k answer set. 0/2 means no expected content point was credited; 1/2 means partial coverage. It is not a human-verified grade. Many answers explicitly report BUDGET_TRUNCATED; those are evidence that bounded retrieval stopped before coverage was complete, not evidence that the source lacks the answer.

Question	Score	Missing expected content / issue	Candidate improvement
Q01	0/2	Export’s attributed “immersive fallacy”: envelopment is not equivalent to engagement.	Retrieve the export’s critical-framework passage; answer as an attributed framework, not an independently proven law.
Q02	1/2	Funding gaps, distribution/licensing routes, and weak artist-to-venue pathways. Current answer covers deployment/time friction.	Add the export’s sector-barriers passage and distinguish funding/distribution barriers from production logistics.
Q05	0/2	Review’s distinction: immersion is technical capability; presence is psychological response.	Retrieve the review’s page-3 definition passage.
Q06	0/2	Higher visual fidelity/immersion does not necessarily create presence or realistic response.	Retrieve the review’s presence discussion and state the qualification directly.
Q09	0/2	Transcript’s attributed contrast: collective dome viewing versus isolated headset VR.	Retrieve the explicit comparison in the transcript; preserve speaker attribution.
Q10	0/2	Vortex Concerts people, Morrison Planetarium, and circa 1957.	Retrieve the specific historical segment; include names/place/date with a transcript citation.
Q13	1/2	Correctly distinguishes loudspeakers, clusters, and live sound objects, but the saved grade says the 8k response omitted physical counts/clusters.	Use the source-specific distinction (157 speakers, 31 output clusters; table counts rendered objects) and exact citations. Regrade both run variants consistently.
Q17	1/2	The answer gives venue-specific decoding, coordinates, and Ambisonics order, but does not state preservation/transfer costs identified by the rubric.	Retrieve the export’s transfer/preservation passage; avoid claiming a quantified cost unless the source provides it.
Q18	0/2	Export distinguishes planetaria, artistic fulldome, corporate immersion, and CAVE-adjacent installations.	Retrieve the taxonomy passage and answer each category explicitly.
Q20	1/2	Correctly rejects a universal sickness result but omits the review’s conflicting findings and conditions.	Retrieve the contrasting sickness findings and summarize their different conditions; do not invent a universal rate.
Q21	1/2	Content is graded complete, but original citation quote did not exactly match its source span.	Use v040’s repaired quote as a structural correction, then revalidate entailment and rescore. Treat the reported latency as a speaker claim.
Q23	1/2	Content is graded complete, but two original quote spans were invalid.	Keep the repaired exact quotes in the verified output; rerun structural and semantic checks.
Q24	1/2	Content is graded complete, but original citation quote was invalid.	Keep the repaired exact quote; rerun checks.
Q25	0/2	Cross-source synthesis needs both the immersion/presence qualification and the export’s immersive-fallacy framework.	Retrieve both passages; answer that neither source establishes a guarantee from sensory envelopment to engagement.
Q26	1/2	Avoids claiming universal dome preference but lacks the transcript’s communal/headset contrast and review’s individual variation in presence.	Retrieve both source passages and distinguish attributed rhetoric from user-level research findings.
Q28	1/2	Correctly says SATIE rendered-object counts cannot substitute for venue speaker inventory; expected point also includes venue-specific transfer costs. Original citations were invalid.	Keep quantities separate, add the transfer-cost evidence, and use repaired exact quotations from v040.
Q29	1/2	Gives SATIE and transcript examples, but original three citation spans were invalid.	Use repaired citations and retain the limitation: case studies demonstrate reported interaction, not comparative audience efficacy.
Q31	1/2	Gives export deployment constraints but misses the review’s apparatus size/cost/complexity discussion.	Retrieve CMMM resource-limit passage and compare it with export funding/distribution constraints without inferring project ROI.
Q32	1/2	Avoids unsupported override, but lacks both the transcript’s collective framing and review’s individual variation in presence.	Retrieve both passages and state that collective framing does not establish uniform individual response.
Q34	1/2	Correctly refuses a hardware guarantee; original citation quote was invalid.	Keep the conditional benchmark answer, use its exact repaired citation, and revalidate.

This breakdown accounts for 19 lost points relative to a perfect 48-point score. It must not be interpreted as 19 independent answer-generation defects: Q21/Q23/Q24/Q28/Q29/Q34 include structural citation penalties in the old recorded score, while several zero/partial answers are explicit bounded-retrieval gaps.

Improvement order

1. Repair the four partial claims in a disposable candidate dataset by splitting compound claims, adding claim-owned exact excerpts, and narrowing the SATIE causal wording. Re-run deterministic bindings and obtain a second independent semantic assessment. Keep changes out of canonical knowledge pending existing policy/owner review.
2. Fix retrieval coverage before answer prompting. The missed Q01/Q05/Q06/Q09/Q10/Q18/Q25/Q26/Q31/Q32 content is mainly absent from the bounded answer context. Add question-to-source coverage checks and allow targeted continuation when a relevant search reports BUDGET_TRUNCATED; do not answer from a truncation marker as if search were exhaustive.
3. Use structured answer completeness checks. Compare each draft against the frozen expected-point checklist before finalizing. Require a source citation for each factual point, preserve attribution and caveats, and keep cross-file answers tied to both required sources.
4. Retain output citation validation. The quote restoration in bundle v040 fixed all 15 malformed source quote strings in the separate verified copy. Make exact quote validation a pre-score hard gate in the runner so invalid spans cannot depress or contaminate content-quality analysis.
5. Regrade with an auditable method. Score the corrected answer artifact against the unchanged reference, save per-point decisions and cited evidence IDs, and use an independent second evaluator. Report agreement and adjudicate disagreements as unresolved rather than silently awarding points.
6. Only then rerun qualification gates. Recalculate support and answer quality on the frozen benchmark, leaving reserved questions untouched until tuning thresholds pass.

16 October candidate trace and semantic recheck

The four original support groups have now been traced against their extraction text. The two CMMM2015 groups were compound claims whose passages were split across the PDF’s two-column extraction; the source supports the claims when separated and given claim-owned spans. The SATIE quote supports the careful wording that the authors could not perform the 128-speaker test with their then-current SuperCollider version; it does not establish why they could not. The transcript does contain the 2006 date and the later Black Holes identification, but the original quote began after the date.

An isolated candidate packet split the two CMMM claims into seven atomic claims and added the missing transcript context. The deterministic evidence-binding audit passed 8/8 candidate claims. Two independent Antigravity CLI judges (Gemini 3.8 Flash Medium and Gemini 4 Argon Medium) then marked 8/8 supported at confidence ≥0.90. The earlier dual review independently marked the SATIE claim supported at 0.98 confidence from both judges. Together, these results resolve the four traced support defects at the candidate/advisory level. They do not change the frozen canonical score of 53/57, accept any correction proposal, or authorize an owner-protected decision. Candidate packet, binding audit, and judge traces are under /tmp/gewu-dome-pilot/evaluation-v025/claim-repair-candidates-v052.json, claim-repair-structural-review-v052.json, and semantic-review-claim-repair-v052/.

Answer and live-agent retest status

Antigravity CLI 1.3.6 successfully ran the pinned two-judge semantic review in sandboxed plan mode. This confirms the model endpoint was available; an earlier quota message from a different provider attempt was stale. The same CLI did not complete the four-question GEWU answer batch within its print timeout, and a one-question retry ended with a provider API EOF. The GEWU MCP tool was not available in the clean CLI configuration. A Codex CLI Luna MCP attempt was also blocked by its approval_policy=never read-only setup, so it did not become a comparable answer run. A direct Luna fallback produced four explicit evidence gaps and one citation-verified Q13 answer; these smoke results are not a 24-question score.

The frozen 24-question answer rerun therefore remains incomplete. Its last comparable result is still 29/48 content points (60.4%) after separate exact-quote repair, below the 85% target; the original support audit remains 53/57 (92.98%), below 95%. The 16 reserved questions and three fresh repetitions remain correctly gated. The existing behavioral harness verifies upload-chunk resume, same-digest re-ingest, archive handling for injected instructions, and canonical-head preservation, but it did not test agent resistance or agent interruption/resume. Its changed-file derivative case, successful delayed indexing, and useful constrained-budget response are also not demonstrated end to end. These gaps must be closed with a working read-only/isolated GEWU MCP connection before qualification.

Important score limitation

The two quality measures are not fully independent: the support judgment and answer score were produced by the same primary assistant evaluator, and neither received human review or a second blind evaluation. The support report is a useful failure locator, not a definitive certification. The v040 citation repair is verified structurally, but semantic answer scoring after repair has not yet been performed. These limitations must be addressed before claiming a pass.