RAG — Retrieval-Augmented Generation — has become the standard in 2025 and 2026 when it comes to making company-specific knowledge usable for LLM applications. The idea is compelling: documents are indexed, relevant passages are retrieved on a user query, and the LLM gets them along with the query as context. Sounds simple. In practice, several months of work lie between a working prototype and a production system.
The three levers that really matter
First: chunk size. Chunks that are too small lose context; chunks that are too large dilute relevance. For PDF-based technical literature, 400–600 tokens per chunk with 80–120 tokens of overlap have proven effective. For structured content (wikis, knowledge-base articles), the natural section boundary works better than fixed token sizes.
Second: metadata. A chunk without metadata is a chunk without context. At a minimum, title, author, date, category, and language should be stored alongside it — often a URL back to the original as well. In retrieval, this is used not only for filtering but also to provide the LLM with valuable positional information.
Third: reranking. Switching from pure vector retrieval to hybrid search (BM25 + dense embeddings) followed by cross-encoder reranking has consistently raised hit quality by 30–40% in our projects. It costs around 200 ms more per request — almost always the best 200 ms investment in the entire system.
What is often underestimated
Curating the knowledge base. A system that worked well yesterday will be mediocre in six months — simply because new documents are added and old ones become outdated. Without a clear maintenance process, a system that confidently gives wrong answers will gradually emerge.
The clean answer is an editorial workflow: documents have a status (draft/published/archived), a review date, and an owner. The indexer skips archived and flags documents whose review date has passed.
Results from a recent project
In a project with 4,500 technical documents (a mix of PDF, DOCX, Markdown), we measured the following values:
- Precision@5: 0.73 (before reranking) → 0.89 (after cross-encoder rerank)
- Median latency: 380 ms (retrieval) + 1.2 s (LLM response) = 1.6 s end-to-end
- User adoption (after 4 weeks): 71% of users return weekly
The adoption number is more interesting than the retrieval metrics. A system with 0.89 precision that no one uses is worthless. A system with 0.73 that helps in everyday work creates real impact.
Outlook
The next leap will depend less on the model and more on orchestration: when does a search even need to happen? When is a fast decision without RAG enough? When should the system ask a clarifying question instead of hallucinating? The answers to these questions will determine whether a RAG system becomes a real assistant.