I got shut out by my own document search system last February. Not a cloud outage or a bad deploy — I typed a legal clause number I knew was in the contract repository, and the system returned zero results. The query was exact. The document existed. And a keyword search that had indexed every word in the corpus couldn’t find it because the clause used “indemnification” and I’d typed “hold harmless.”
That’s the moment this RAG system started.
The central problem with retrieval-augmented generation — RAG, the pattern where you feed an AI model relevant documents alongside a user’s question so it can answer from real sources instead of hallucinating — isn’t about model quality. It’s about retrieval quality. If the documents you pull back are wrong, the best model in the world will still give you a convincing wrong answer. And enterprise documents are uniquely hostile to retrieval: legal clauses, product codes, regulatory citations, and nested section hierarchies all break the vector-similarity approach that works fine for “how do I reset my password.”
The fix isn’t one strategy. It’s three, fused.
The thing I got wrong first
My first pass treated every document as flat text. The unstructured library pulled paragraphs out of PDFs and I shoved them into a vector database — pgvector, a Postgres extension that stores embeddings (numerical fingerprints of meaning) and lets you search by similarity rather than exact keyword match.
It worked great on the demo. It failed on real documents.
A 90-page services agreement has structure that carries meaning. Section 4.2(a) nested under Article IV modifies the definition in Section 1.1, and if you flatten that hierarchy into semantically-similar chunks, you lose the modification chain. A system that returns the most “relevant” paragraph by embedding distance will happily return Section 1.1’s definition without Section 4.2(a)‘s override, because both paragraphs talk about the same thing.
So I added layout awareness to the ingestion pipeline: coordinate extraction, font-weight analysis, heading-level detection, and visual-emphasis scoring. The processor now tracks where each chunk sits in the document’s structural tree, not just what it says.
Making GPU crunch actually reliable
The processing pipeline hits thousands of pages. Doing that on CPU means you walk away and come back tomorrow. Doing it on GPU means you walk away and come back in ten minutes — unless the GPU runs out of memory on page 847 of 900 and you re-run from scratch.
I burned a Saturday on that exact failure mode.
The fix was three things running together: smart batching that watches VRAM pressure and shrinks batch size before it OOMs, automatic GPU selection that picks the card with the most free memory rather than a hard-coded device ID, and a checkpoint system that writes progress markers to disk so a crash on page 847 means re-running from page 800, not page 1.
# Example of our GPU-accelerated document processor
class DocumentProcessor:
def process_file(self, file_path: str, timeout_seconds: int) -> Tuple[List[Document], Dict]:
with timeout(timeout_seconds):
# GPU-accelerated processing
pdf_elements = self._extract_pdf_elements(file_path)
documents = self._process_elements(pdf_elements)
return documents, self._get_processing_stats()The multi-GPU support came later, after we realized a single A100 was bottlenecked on PDF parsing while the second card sat idle. Now both cards chew through separate files in parallel.
Why hybrid retrieval actually matters
why hybrid retrieval works — and how to wire it give me the detail
The core insight: dense vector search (embedding similarity) and sparse keyword search (BM25) fail in opposite directions. Vectors miss exact product codes, legal clause numbers, and rare proper nouns — the very tokens that matter most in enterprise docs. BM25 misses paraphrase and synonym matches. Combining them with Reciprocal Rank Fusion (RRF) exploits both without requiring a tuned weighting scheme, because RRF is rank-based and robust to score-scale differences.
The real stack behind each strategy:
| Strategy | Tooling |
|---|---|
| Dense vector | pgvector (Postgres extension) + Azure text-embedding-ada-002 or text-embedding-3-large |
| Sparse keyword | BM25 via rank_bm25 or Postgres tsvector / ts_rank — no separate index needed |
| Layout / structure | unstructured library for PDF element extraction (coordinates, font weight, heading level) |
| Eval loop | ragas — measures Answer Relevance, Faithfulness, Context Recall against a golden Q&A set |
Concrete try-it snippet — RRF fusion in pure Python, no extra dependencies:
from collections import defaultdict
def reciprocal_rank_fusion(*ranked_lists, k=60):
"""Fuse N ranked result lists. k=60 is the standard constant."""
scores = defaultdict(float)
for ranked in ranked_lists:
for rank, doc_id in enumerate(ranked, start=1):
scores[doc_id] += 1.0 / (k + rank)
return sorted(scores, key=scores.__getitem__, reverse=True)
# Usage: fuse vector hits with BM25 hits
fused = reciprocal_rank_fusion(vector_hits, bm25_hits, metadata_hits)Drop this into your retrieval layer and run ragas on your eval set before and after — faithfulness typically improves because the keyword leg keeps grounding citations in exact source text.
Without the hybrid approach, you pick your failure mode: exact-match misses (vector-only) or paraphrase misses (keyword-only). With RRF, you get both strengths without hand-tuning weights that drift every time the document corpus changes. The metadata layer — document type, date, author, department — adds a third signal that helps when neither the vector nor the keyword path is confident.
Don’t trust the output. Verify it.
Even with perfect retrieval, LLMs fabricate. Not maliciously — statistically. If 90% of the training data uses one phrasing, the model will “complete” your answer that way whether or not the source document says it.
The quality pipeline runs three checks on every response:
Answer relevance — does the response actually address what was asked, or did the model get distracted by a high-scoring but irrelevant chunk? This is a graded check with a confidence score, not a binary gate.
Hallucination detection — every factual claim gets verified against the retrieved source documents. Unsupported statements get flagged. The system doesn’t delete them; it annotates them so the user knows which sentences are grounded and which aren’t.
Source validation — citations get checked for accuracy, source-to-source relationships get tracked, and the whole chain lands in an audit trail so compliance can reconstruct exactly which documents fed which answers.
def validate_response(self, response: str, sources: List[Document]) -> bool:
return all([
self.fact_checker.verify(response, sources),
self.hallucination_detector.check(response),
self.relevance_scorer.evaluate(response) > 0.8
])The 0.8 threshold isn’t magic — it’s where we stopped seeing false passes in the eval set. Below that, the system surfaces uncertainty rather than pretending confidence.
Smart chunking that respects structure
Chunking — splitting documents into searchable pieces — is the least glamorous part of RAG and the place most systems quietly fail. Split too small and you lose context. Split too large and you dilute search precision.
The chunker here preserves three things: hierarchy (Section 4.2(a) stays nested under Section 4.2), relationships (cross-references between sections survive the split), and layout (a paragraph in a warning callout box gets different treatment from body text).
def smart_chunk(self, document: Document) -> List[Chunk]:
return self.chunk_analyzer.split(
document,
preserve_hierarchy=True,
maintain_relationships=True,
respect_layout=True
)Where this goes next
Three things on the near-term roadmap:
Multi-modal processing. A surprising amount of enterprise knowledge lives in charts, tables, and diagrams — not text. Extracting meaning from a revenue waterfall chart or an org-chart image requires vision models wired into the same retrieval pipeline, not a separate image-search silo.
Usage analytics. Who’s searching for what, what queries return zero results, which documents get retrieved but never cited — this is the feedback loop that tells you whether your retrieval quality is actually improving or just passing evals.
Cross-document relationships. Right now the system understands hierarchy within a document. It doesn’t understand that Section 4.2(a) of the Master Services Agreement modifies Schedule B of the Statement of Work signed six months later. Dynamic knowledge graphs that span documents are the next retrieval-quality jump.
This system runs in production now. It’s not perfect — no retrieval system is — but it fails in predictable, auditable ways rather than silently returning confident wrong answers. For organizations sitting on millions of pages of contracts, compliance documents, and technical specifications, that’s the difference between an AI tool you demo and one you actually use.
The code is open-source at github.com/RooseveltAdvisors/enterprise-rag.