44 lines
5.9 KiB
Markdown
44 lines
5.9 KiB
Markdown
# Operations and recovery protocol
|
|
|
|
Status: proposed v0.1, 12 October 2026. Applies to local and VPS installations of the [architecture](../architecture/ARCHITECTURE.md).
|
|
|
|
## Installation identity and health
|
|
|
|
Each installation has a stable service ID, environment label and one authoritative knowledge ref. Diagnostics report software/schema versions, canonical head, indexed revision, index lag, pending queue count/age, parser availability, model/index profile and configured limits. Return only the caller's permitted identity/capability details. Public liveness must not reveal private project names or configuration.
|
|
|
|
Readiness requires complete startup recovery and available canonical/control stores. An unavailable index may permit explicitly degraded reads but prevents reporting full retrieval readiness. A writer-lock conflict prevents a second instance from accepting writes. Deployment scripts must fail visibly rather than start a competing writer.
|
|
|
|
Use structured, bounded logs with request, run, proposal and safe credential identifiers. Redact before persistence. Never log source bodies, full tool payloads, passwords or authorization headers by default. Record safe error codes and private stack traces with retention limits. Durable acceptance/security audit records have separate retention and backup requirements from disposable diagnostics.
|
|
|
|
## Backups
|
|
|
|
Back up a consistent set: knowledge Git repository, artifact/extraction tree, staging/run/receipt tree, private control database, schema versions and required configuration. Keep recovery credentials/encryption keys in a separate protected recovery location. The derived index is optional backup material. A Git remote alone is not an artifact or account backup.
|
|
|
|
For the first implementation, use a short write-quiescence window: stop new acceptances/uploads/control mutations, drain active publications, checkpoint durable file state, take a supported SQLite backup of the control database, snapshot/copy the matching filesystem set, then resume writes. Record the authoritative Git head and inventory hashes. Do not copy a live SQLite main file while ignoring its WAL.
|
|
|
|
Encrypt off-host backups and restrict credentials. Proposed starting schedule is daily plus before migration, but owner-selected retention, recovery-point objective and recovery-time objective are prerequisites to team launch. Do not label these values agreed or declare recovery adequate without a measured restore.
|
|
|
|
Restore into a new isolated directory/host. Verify artifact/extraction hashes, canonical manifests, identity/grants and proposal states, reconcile receipts, rebuild the index and exercise cited reads and a disposable acceptance. Prevent the restored clone from writing as a second authority. After a real rollback, invalidate sessions and reissue/reconcile credentials and revocations before networking is enabled; restoring an older control database must not revive revoked access.
|
|
|
|
## Schema and model migrations
|
|
|
|
Version public contracts, canonical records and index schemas separately. A migration declares supported source/target versions, backup prerequisites, dry-run findings, transformed records and rollback limits. Use the same validators as ordinary writes. Canonical migrations create attributable service-maintenance commits; they cannot fabricate knowledge or bypass protected content decisions.
|
|
|
|
Test migration on a restored copy first. Old readers must reject unsupported schemas explicitly. Rebuild derived indexes into new generations. Keep the previous generation until the new one passes integrity/retrieval checks and old connections drain. An embedding model update cannot mix vector dimensions or silently erase citations to old source extractions.
|
|
|
|
Rollback of knowledge uses a corrective/superseding proposal or documented maintenance recovery, preserving history. Never solve an application bug by silently rewriting accepted Git history. Software rollback must remain compatible with the resulting schema, or restore the complete consistent snapshot under a stopped service.
|
|
|
|
## VPS transfer and release
|
|
|
|
Freeze local writes, record the final head, transfer and verify all authoritative stores, restore the service under a restricted account, then run its local health/recovery checks. Test HTTPS, cookie/key boundaries and MCP transfer from an actual client. Switch client endpoint configuration after the VPS is ready. The old local service stays read-only/offline until deliberately made authoritative again through a controlled handback.
|
|
|
|
The web and MCP routes share domain logic but retain different capabilities. Built UI assets may be served by the Python app behind Caddy. Private data mounts are never exposed as a static directory. Parser processes have restricted mounts; the public service runs without host-admin privileges. Record container/image digests and exact lockfiles used for the release.
|
|
|
|
Release evidence separates local checks, pushed revision, deployed revision, authenticated runtime behavior and restore verification. No deployment command or backup schedule has been executed as part of this design exercise.
|
|
|
|
## Capacity and incident handling
|
|
|
|
Before team release, name benchmark hardware, model, corpus, archive sizes, concurrent users/agents, queue limits and query budget distribution. Measure latency percentiles, top-five source recall, cold/warm index rebuild, parse time, memory and disk growth. PRD targets remain search under 300 ms at 10,000 project nodes, 100,000 total nodes and rebuild under ten minutes; define the measurement setup before running it. Saturation returns a retryable bounded outcome rather than unlimited queuing.
|
|
|
|
For an incident, inspect persisted production history and the actual affected endpoint/state before proposing causes. Separate observed facts from likely causes and untested hypotheses. Preserve the failed record, isolate a reproducer, repair the boundary and add a regression fixture when justified. Never infer deployed state from a local checkout or hide invalid evidence behind a successful request status.
|