Offline Hybrid Retrieval

Answers Grounded in Real Documentation · Shipped · workstation, home

Sparse BM25 keyword matching merged with dense semantic embeddings via Reciprocal Rank Fusion — no cloud search needed.

Halbert’s knowledge retrieval combines two complementary search strategies. BM25 sparse retrieval excels at exact keyword matches (specific flag names, error codes). Dense embedding retrieval finds semantically similar content even when the exact words differ. Reciprocal Rank Fusion merges both result sets, and optional cross-encoder reranking provides final precision.

Queries are dispatched concurrently to BM25 and dense vector indices. Results are merged using RRF scoring. The top candidates are optionally reranked by a cross-encoder for maximum precision before being injected into the agent’s prompt context.

Limits and invariants

Strict licensing compliance: indices only ingest documentation with verified open licenses.

Where this lives

• halbert_core/halbert_core/rag/pipeline.py
• halbert_core/halbert_core/rag/retriever.py
• halbert_core/halbert_core/rag/embeddings.py
• halbert_core/halbert_core/integrations/sourceprep_retrieval_backend.py