66 lines
3.9 KiB
Markdown
66 lines
3.9 KiB
Markdown
# Hybrid retrieval dependency probe
|
|
|
|
Status: implementation and compatibility evidence. This record does not meet the
|
|
PRD's labeled top-five recall target or close FR-A2.
|
|
|
|
## Runtime
|
|
|
|
- Linux x86_64, Python 3.14.7, SQLite 3.53.4, CPU inference.
|
|
- Disposable environment: `sentence-transformers==6.1.0`, `torch==2.14.1+cpu`,
|
|
and `sqlite-vec==0.1.9`. These packages were not installed into the project
|
|
runtime.
|
|
- `sqlite-vec` extension loaded into an in-memory SQLite database and reported
|
|
`v0.1.9`; a `vec0` table accepted normalized float32 vectors and returned the
|
|
nearest row first (distance 0.00612 versus 0.88957 for the second row).
|
|
- Model: `intfloat/multilingual-e5-small`, repository revision
|
|
`614241f622f53c4eeff9890bdc4f31cfecc418b3`; local packaged tree SHA-256
|
|
`b96279dfc10e6c6c05782feead2f0155d7e16975d3ebc440cda0c46b9a038094`.
|
|
- Output was 384-dimensional and normalized to norm 1.0. Both query and passage
|
|
calls used the required `query: ` and `passage: ` prefixes.
|
|
- English query/passage cosine: 0.84662. Chinese query/passage cosine: 0.94244.
|
|
English query/Chinese passage cosine: 0.79640. This small hand-written probe is
|
|
not a relevance benchmark and makes no recall claim.
|
|
- End-to-end composition used the real local model with the Git, FTS, vector and
|
|
application search adapters. It returned `retrieval_mode=hybrid` with four
|
|
results. The vector generation and canonical index both reported revision
|
|
`734c31c64a22803134a76bba30fec25c8f322c8f`; model ID, artifact digest, content
|
|
profile and dimensions matched, and both indexes used the same SQLite file.
|
|
|
|
## Code delivered
|
|
|
|
`src/gewu/application/ports.py` defines the replaceable embedding port.
|
|
`src/gewu/adapters/embeddings.py` loads only an already-installed local model,
|
|
checks its tree digest, disables remote code and network loading, supplies E5
|
|
prefixes, and validates row count, dimensions, finite values, and normalization.
|
|
`src/gewu/adapters/sqlite_vectors.py` stores scoped vectors in immutable
|
|
generations, records revision/model/artifact/content-profile/dimension metadata,
|
|
and atomically switches the active generation using compare-and-swap. Canonical
|
|
FTS and vectors publish in the same SQLite transaction. Prior generations remain
|
|
available for rollback, and active-generation deletion is rejected.
|
|
The `retrieval` optional dependency set is pinned and included in `uv.lock`;
|
|
`sqlite-vec` is also in the dev extra so CI exercises its native adapter. Unit
|
|
tests use an injected encoder to verify model-adapter contracts without
|
|
downloading weights. Vector-index tests cover project isolation, activation,
|
|
rollback, and invalid-vector rejection (`tests/adapters/test_sqlite_vectors.py`).
|
|
The application tests cover fusion, stable pages, cursor invalidation on model
|
|
change, generation replacement, scope/filter enforcement, labeled degradation,
|
|
and bounded one-hop graph expansion. They also verify that an injected failure
|
|
after vector activation rolls back both vector and FTS publication
|
|
(`tests/application/test_hybrid_search.py`).
|
|
|
|
## Still required before hybrid acceptance
|
|
|
|
- Expand citation/scope fixtures against the PRD's labeled corpus; current
|
|
tests verify graph-origin citation propagation, filter enforcement and that
|
|
cross-project neighbors do not escape scope.
|
|
- Qualify old-citation stability across real source/parser revisions and measure
|
|
cold re-embedding and generation publication against target hardware.
|
|
- Run the PRD's labeled top-five recall set and language/corpus fixtures.
|
|
- Package model assets for offline installation and probe the native vector
|
|
runtime on Windows and macOS; qualify cold rebuild time on target hardware.
|
|
|
|
The [model card](https://huggingface.co/intfloat/multilingual-e5-small/blob/614241f622f53c4eeff9890bdc4f31cfecc418b3/README.md)
|
|
documents the candidate model. The [sqlite-vec project](https://github.com/asg017/sqlite-vec)
|
|
identifies its current pre-v1 compatibility risk; using it in the probe does not
|
|
qualify it for production storage.
|