How to Build a RAG System Over Company Documents

⚡What you are building and why the demo lies to you

A RAG system over company documents answers employee questions from your own corpus: policies, contracts, runbooks, wikis, past proposals, support macros. The pipeline is conceptually simple — parse documents into chunks, embed the chunks into a vector index, retrieve the relevant ones for each question, and have a language model compose an answer that cites them. Every part of that sentence hides a decision that determines whether the system is trusted or abandoned.

The weekend demo lies to you because it runs on fifty clean PDFs and five friendly questions. Production runs on fifty thousand files in eleven formats, fifteen years of accumulated contradiction, and questions from people who do not know what the document was called. The gap between the two is not the model — it is corpus hygiene, parsing quality, retrieval tuning, and permissions. That gap is what this guide is about.

One scoping note before the architecture: RAG answers questions about documents. If you need the system to take actions, you are building an agent with retrieval as one of its tools — a related but larger scope. Keep the first system read-only. A trustworthy answer engine earns the right to become an action engine later.

🗂️Step one: audit the corpus before touching a model

The corpus audit is the least technical step and the one that predicts success best. Inventory what you actually have: where documents live (SharePoint, Google Drive, Confluence, network shares, email attachments), what formats they are in, how many are duplicates or near-duplicates, which are superseded but still sitting next to their replacements, and who owns each collection. In most companies this audit surfaces the uncomfortable finding that nobody fully knows.

Pay special attention to contradictions and staleness, because the retriever cannot tell a 2019 vacation policy from a 2026 one unless the metadata says so. Effective dates, document owners, and version identifiers are not nice-to-haves — they are retrieval features. A policy library without effective dates will eventually answer an HR question with last decade in the voice of this quarter.

Decide what is out of scope for version one and write it down. A RAG system over the top three collections your people actually search — usually HR policies, product documentation, and one operational wiki — beats a system over everything that answers nothing well. Scope the corpus like you would scope a product, because it is one.

✂️Parsing and chunking: where answer quality is decided

Parsing is the most underinvested layer in most RAG builds. A PDF parser that treats a two-column contract as one stream of text produces chunks that interleave unrelated clauses, and no embedding model can recover from that. Use a layout-aware parser — tools like Unstructured, Docling, or LlamaParse are the common choices as of writing; evaluate them on your own worst documents, not their demo files. Tables deserve special treatment: either serialize them row-aware or extract them into a structured store and route tabular questions to SQL instead of semantic search.

Chunking is a retrieval decision disguised as a text-processing step. Fixed-size windows (say, 512 tokens with overlap) are the default because they are easy, and they are wrong for structured documents because they slice through sections mid-thought. Chunk by the document structure: headings, sections, clauses, list items. Keep chunks self-contained — a chunk that says "as described above, the limit does not apply" is useless without the chunk above it, so prepend section titles and document context to each chunk before embedding.

Size matters less than coherence, but as a working range, most corporate corpora retrieve well with chunks between 200 and 800 tokens, with parent-child retrieval for long sections: embed small child chunks for precise matching, return the parent section for context. Whatever you choose, make chunking a versioned, re-runnable pipeline, because you will change your mind after the eval data arrives — and re-chunking a corpus should be an afternoon, not a project.

🧬Choosing an embedding model (and living with the choice)

The embedding model converts chunks and questions into vectors, and its quality bounds everything downstream. The practical decision is hosted API versus self-hosted open model. Hosted models — from OpenAI, Cohere, Voyage and others — are the default for a reason: strong quality, zero operations, per-token pricing that is cheap at corporate-corpus scale. Self-hosted open models make sense when data residency forbids sending text to an external API or when corpus size makes per-token economics hurt. Benchmarks shift quarterly, so as of writing — verify current options on public retrieval benchmarks and, more importantly, on a test set built from your own documents before committing.

Two operational facts matter more than the model leaderboard. First, embeddings are not interchangeable: changing models means re-embedding the entire corpus, so treat the choice as a one-way door with a toll. Second, multilingual corpora need multilingual models — if your documents mix English with French or Hindi, a monolingual-optimized model will silently degrade half your retrieval.

Dimensionality and index size are rarely the constraint people worry about. A million chunks at typical dimensions is comfortably within a managed vector index or a single Postgres instance with pgvector. Do not let infrastructure anxiety drive the model choice; let retrieval quality on your eval set drive it.

🎯Retrieval tuning: hybrid search, reranking, and metadata filters

Pure vector search fails on the queries corporate users actually type: part numbers, error codes, exact policy names, acronyms that appear in three departments meaning three things. The fix is hybrid retrieval — combine vector similarity with keyword search (BM25 via Postgres full-text, Elasticsearch, or your vector store hybrid feature) and merge the result lists. This single change fixes the majority of "it found the wrong document" complaints.

The second layer is reranking: retrieve a generous candidate set (thirty to fifty chunks), then pass them through a cross-encoder or reranking API that scores each against the actual question, and keep the top few. Rerankers add latency and cost per query, and they are worth it — the precision gain is usually the largest single improvement after fixing chunking.

The third layer is metadata filtering, and it does double duty. Effective-date filters keep superseded documents out. Department and document-type filters let a question about "expense limits" search finance policy instead of the engineering travel wiki. And permission filters — the next section — keep the whole system legal. Tune the pipeline in this order: fix chunking first, add hybrid second, add reranking third, and measure each step on your golden questions so you know which lever did what.

Pinecone vs pgvector vs Qdrant: vector database shootout→

🔐Permission-aware retrieval: the non-negotiable layer

An internal RAG system concentrates every document in the company behind one search box, including the ones most employees were never supposed to read. If retrieval ignores access control, you have built a confidentiality incident with a friendly interface. This layer is not optional and it is not a phase-two item.

The correct pattern is filter-at-query-time: every chunk carries the ACL metadata of its source document, the user identity resolves to their allowed collections at query time, and the index filters before ranking, not after. Post-hoc filtering — retrieve first, then drop what the user cannot see — is a common shortcut and a broken one, because it leaks through answer composition and citation snippets.

The hard part is identity plumbing, not vector math. Permissions live in SharePoint groups, Google Drive sharing, Confluence spaces, and HR systems, and they change daily. You need a sync pipeline that keeps chunk ACLs fresh and a decision about staleness: how stale can a permission be before you would rather not answer? For most companies the honest answer is "same day," which shapes the sync architecture. Start with the collections that have simple, collection-level permissions; documents with per-user sharing lists are where implementations go to die.

✅Evals: golden questions are the only way to know it works

RAG systems fail silently. There is no exception, no stack trace — the answers just get worse after a chunking change, and nobody notices until trust is gone. The only defense is an eval harness: a fixed set of golden questions with known correct answers and known correct source documents, run automatically on every pipeline change.

Build the golden set from reality. Pull fifty to two hundred real questions from search logs, Slack help channels, and the people who currently answer "where is X" all day. For each, record the question, the document that should answer it, and the key facts the answer must contain. Include adversarial cases: questions with no answer in the corpus (the system must decline), questions that match a superseded document (it must use the current one), and questions outside the user permission scope (it must refuse).

Measure three things separately: retrieval recall (did the right chunk come back in the top results), answer faithfulness (does the answer only say things the retrieved chunks support), and citation accuracy (do the citations point at the real sources). Frameworks like RAGAS can automate parts of this, but a human review pass on a sampled slice every week catches what automated judges miss — especially tone, completeness, and the subtle ways answers drift wrong.

🚀The production checklist, costs, and when to hire a team

Before go-live, walk the checklist below honestly. The items look operational because they are — the difference between a RAG pilot and a RAG product is entirely in the plumbing around the pipeline: freshness, permissions sync, monitoring, and the eval harness that catches regressions before users do.

On cost, as honest labelled ranges: a focused pilot — two or three document collections, hybrid retrieval, citations, basic permissions, a golden-question harness — typically runs $50,000 to $100,000 and eight to twelve weeks. A production system with permission-aware retrieval across major sources, OCR for scanned archives, full eval automation, and monitoring typically runs $150,000 to $300,000 over four to six months. Ongoing costs — embedding and generation tokens, vector storage, sync pipelines — are usually modest relative to the build, often hundreds to low thousands of dollars per month at mid-company scale; verify current pricing before committing.

Hire a specialist team when the corpus is large or messy, when permissions are real (HR, legal, M&A material in scope), when OCR and table extraction matter, or when the system will be load-bearing for a team that depends on its answers. At Codazz, our RAG development practice builds exactly this kind of system — corpus audit first, permission-aware from day one, measured against golden questions before and after every change. The first conversation is free and includes an honest read on whether your corpus is ready.

RAG development services at Codazz→RAG system cost in 2026: build vs buy→