# Fine-tuning, RAG or prompting: ask what needs to change. > Does the AI need clearer instructions, better access to company information or a change to how it responds? Those are different problems with different costs and solutions. Canonical URL: https://marmation.com/review/fine-tuning-versus-rag-versus-prompting/ By Vikram Roy · 2026-11-12 An assistant that gives the wrong answer does not always need more training. It may need a clearer instruction, a current document or a different place to run. Identifying the actual defect helps avoid paying for a change that leaves the problem intact. An organization wants an internal assistant to answer policy questions in a standard format. The team proposes fine-tuning because the answer needs to sound consistent. Another proposes a massive context window. Neither has first established which problem is actually difficult. ## Four choices that solve different problems Prompting can structure behavior. Retrieval finds current, relevant source material and supplies it to the model. Adding that material before the answer is written is called retrieval-augmented generation (RAG). Fine-tuning adjusts performance through training examples. Hosting addresses where the model runs to produce a result—a process called inference—and who operates it. Each choice has different data, evaluation, cost and maintenance consequences. ## Find the failure before changing the system Create a small evaluation set that isolates the failure: Is the model failing to follow format? Does it lack current facts? Is retrieval finding irrelevant documents? Is the organization blocking a provider? Change one architectural component at a time and observe the outcome. ## An old policy and a broken format need different fixes If the assistant answers from outdated policy, the issue may be indexing and source authority, not a model deficit. If it consistently produces invalid JSON, a structured data format the application needs to read reliably, despite clear examples and structured-output support, model selection or fine-tuning may be worth testing. If security policy prohibits external inference, deployment must be considered separately. ## When a simple instruction is enough For simple tasks, a carefully engineered prompt and an approved model may be all that is needed. For specialized classification with enough labeled data, selective fine-tuning can outperform elaborate prompting. The point is not to reject any tool; it is to choose deliberately. ## Require a measured reason for custom training Before procuring custom model work, define the measured defect and which layer causes it. Require an evidence-based comparison against simpler options. ### References and context [AWS Bedrock Knowledge Bases](); [NIST AI RMF](). ## Sources - [Knowledge bases for Amazon Bedrock]() — AWS Docs. Managed retrieval and knowledge base infrastructure available, making simple RAG less differentiated. Limit: Features, charges and regional availability depend on the configuration and should be checked for an actual deployment. - [Generative AI Profile for AI RMF]() — NIST. Reference governance and risk design principles for gen AI. Risk management reference for generative AI Limit: This is a risk-management reference, not a certification or evidence that Marmation has been audited or complies with it. ## Related reading - [From working prototype to dependable system](https://marmation.com/review/engineering/) - [Design the action boundary before designing the agent.](https://marmation.com/review/designing-an-agent-with-bounded-actions/) - [Engineering autonomy is a control design problem.](https://marmation.com/review/engineering-autonomy-is-a-control-design/)