Fine-tuning, RAG or prompting: ask what needs to change.

An organization wants an internal assistant to answer policy questions in a standard format. The team proposes fine-tuning because the answer needs to sound consistent. Another proposes a massive context window. Neither has first established which problem is actually difficult.

The decision behind the technology

Prompting can structure behavior. Retrieval provides current relevant external knowledge. Fine-tuning adjusts performance through training examples. Hosting addresses where inference occurs and who operates it. Each choice has different data, evaluation, cost and maintenance consequences.

A better way to frame the work

Create a small evaluation set that isolates the failure: Is the model failing to follow format? Does it lack current facts? Is retrieval finding irrelevant documents? Is the organization blocking a provider? Change one architectural component at a time and observe the outcome.

Put the idea under realistic pressure

If the assistant answers from outdated policy, the issue may be indexing and source authority, not a model deficit. If it consistently produces invalid JSON despite clear examples and structured-output support, model selection or fine-tuning may be worth testing. If security policy prohibits external inference, deployment must be considered separately.

When the recommendation changes

For simple tasks, a carefully engineered prompt and an approved model may be all that is needed. For specialized classification with enough labeled data, selective fine-tuning can outperform elaborate prompting. The point is not to reject any tool; it is to choose deliberately.

What to decide before moving forward

Before procuring custom model work, define the measured defect and which layer causes it. Require an evidence-based comparison against simpler options.

References and context

AWS Bedrock Knowledge Bases; NIST AI RMF.