RAG is not a custom-trained model. That distinction can save a project.

A company has 40,000 internal documents. Its leadership asks for a “custom-trained AI that understands our business.” The phrase is understandable. People associate knowledge with training. But the organization probably does not need to modify a foundation model to let employees search current manuals and policies. It needs the right data available to a model, with permission boundaries and source evidence.

That is a different engineering problem.

What retrieval actually does

Retrieval-augmented generation (RAG) creates an application path between source information and a generative model. It identifies relevant source material, includes selected context in a model request and presents a response to the user. The model does not automatically absorb the documents into its own weights. The application can update its index as source information changes.

The difficult part is making the retrieved context trustworthy. Source documents may be scanned, duplicated, poorly structured or outdated. A useful pipeline has to parse information, preserve context, index it for search, filter it by permissions and return passages that answer the actual question.

Embedding similarity is only one candidate technique. Hybrid lexical and semantic retrieval, metadata filtering, reranking and structure-aware chunking may work better for certain tasks.

Why fine-tuning is different

Fine-tuning changes model behavior using examples. It can help a model produce a specialized output schema or follow a recurring style or classification task. It does not automatically establish a live connection to changing company knowledge. A policy updated yesterday does not appear in a fine-tuned model unless the relevant system is designed to bring that information forward.

Fine-tuning also requires evaluation. Without a representative dataset and measurable improvement, it can be a costly way to solve the wrong problem.

Where self-hosting fits

Self-hosting addresses deployment control and some licensing or network constraints. It can be useful when an organization has a strict environment, predictable operating volume or particular model requirements. But it introduces infrastructure and operational responsibility. It does not make retrieval quality or access controls automatic.

A privately served model can still answer from the wrong document or reveal restricted information if the application feeds it material it should not have received.

The failure no model benchmark catches

Suppose a knowledge assistant is configured to answer maintenance questions. The index contains two versions of a service procedure. One is current; the other was withdrawn. Both are semantically relevant. A model can produce a fluent answer using the obsolete procedure. The failure is source governance and retrieval, not a lack of language capability.

An enterprise system therefore needs versioning, source authority, citations, freshness and a state for insufficient or contradictory evidence. It may need human escalation for consequential use cases.

A simple diagnostic

Ask these questions in order: Is the information changing? Does the user need references? Are permissions source-specific? Does the task require actions in another system? Is the problem output behavior, knowledge freshness, latency or deployment policy? Which condition can be evaluated with a realistic dataset?

If the information changes and evidence matters, retrieval is likely relevant. If the output format is unusually strict and repeated labeled examples exist, selective fine-tuning may be worth testing. If data location is the constraint, solve deployment. If the workflow requires actions, design the application and tool boundaries.

The case against custom RAG

Commercial enterprise search products and managed knowledge bases can already solve many common needs. Their capabilities, licensing and permission models vary. It is often more prudent to configure or extend an existing platform than build an entire retrieval system. Custom engineering becomes justified when source structure, permissions, UX or workflow integration goes beyond the platform's capabilities.

The practical rule: Start with the job and the information boundary. Choose retrieval, fine-tuning and hosting separately. They are architectural options, not interchangeable promises.

Sources and limitations

[1] Microsoft Learn, Document-level access control in Azure AI Search: https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview (features marked preview require specific review).
[2] AWS Bedrock Knowledge Bases documentation: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base.html.
[3] Model/provider documentation is version-specific; validate chosen stack before quoting behavior.