Private AI doesn't mean training your own model.

A leadership team decides its employees should have an internal assistant. The company cannot simply send confidential documents to any consumer account. Someone suggests building a private large language model. Another proposes fine-tuning. A third points to a Microsoft enterprise product. Soon the discussion is about GPUs and models, even though no one has agreed on the original restriction.

This is a common failure of translation. The business describes the control it wants using the language of the technology it has heard about. An architect's first job is to separate the underlying needs.

01. What exactly must stay private?

Privacy might mean that inputs are not used to train a provider's foundation models. It might mean limited retention, regional processing, access through a private endpoint, source documents remaining inside the client's storage or no external inference at all. The required architecture changes with each meaning.

A commercial enterprise API may satisfy the first requirement under appropriate terms without meeting the last. A model hosted in an approved cloud account can offer additional control, but still depend on licensed infrastructure and managed services. A fully self-hosted open-weight model may address network and inference constraints while creating new obligations for patching, serving, monitoring, evaluation and capacity.

None of these options automatically provides a correct answer or enforces an employee's document permissions. Those are application responsibilities.

02. Data ownership is not model ownership

If a company owns its policies, manuals and transactional data, it can connect that information to an approved model through retrieval. The model can use relevant excerpts as context for a response, without being trained on all of the documents. This pattern is called retrieval-augmented generation (RAG).

A custom RAG application can be genuinely company-specific even if it uses a commercial foundation model. The custom elements may include ingestion, access control, ranking, citation, user experience, tools, integrations, tests and operating procedures. Those elements can be more strategically valuable than the model itself.

The company's data remains subject to its own governance and the chosen service's contractual terms. Nothing in a RAG design should be described as “100% secure” by default.

03. Fine-tuning is not a document search strategy

Fine-tuning adjusts a model's behavior using additional training examples. It may help with format, classification, tone or specialized tasks where labeled examples and clear evaluation exist. It is not generally the best way to give an assistant rapidly changing manuals, policies or product information. Updating that knowledge through retrieval is usually more direct and easier to govern.

Training a new foundation model from scratch is a separate, far larger research and compute undertaking. Most mid-market business problems do not require it.

04. The difficult problem is often permission-aware retrieval

Suppose an HR document is visible to one employee but not another. A search index that contains it must preserve that distinction. The application should not retrieve the content and then ask the model to keep it secret. Authorization belongs before information reaches the model context, with additional enforcement in the application.

Microsoft's Azure AI Search documentation describes document-level access control patterns, including filters and native permissions features. Some capabilities may still be in preview or subject to specific limitations, so the engineer needs to verify the actual version and source synchronization behavior. [1]

Source content also changes. A revoked permission or withdrawn policy needs to propagate through the retrieval layer. Otherwise, a privately hosted system may still reveal material it should no longer use.

05. A practical architecture decision

Start by documenting the data classes, allowed users, hosting/network restrictions, geography, retention, vendor agreements, model access, retrieval freshness, required actions and acceptable operating cost. Evaluate existing enterprise platforms, approved managed models and private hosting against those constraints. Then test one important use case with authorized and unauthorized personas.

The test should include a good answer with verifiable sources; a no-answer case; an inaccessible document; changed permissions; an obsolete source; and a deliberate adversarial instruction embedded in retrieved material. If the system cannot handle those, 'private' does not make it ready.

The case for not building

An organization's Microsoft, cloud or knowledge platform may already satisfy common search and assistant needs. Configuring the existing service can be faster and easier to operate than a custom stack. A bespoke application is justified when the workflow, access model, UX, integration or deployment requires capability the commercial platform cannot provide economically.

The rule: Specify the privacy and control requirement, then choose the simplest architecture that satisfies it. Model ownership is not the starting point.

Sources and limitations

[1] Microsoft Learn, Document-level access control in Azure AI Search, current page (review preview/GA states and sync caveats).
[2] NIST, AI RMF Generative AI Profile, July 2024, https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence
[3] OpenAI, Enterprise privacy and business data terms; specific configuration/contract matters. https://openai.com/enterprise-privacy/
[4] Anthropic, Commercial terms and privacy; verify current contract. https://privacy.claude.com/